2024-03-21T13:45:23.374 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-21T13:45:23.374 DEBUG:teuthology.run:Teuthology command: teuthology --name adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi --verbose --archive /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898 --owner scheduled_adking@teuthology --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/orig.config.yaml 2024-03-21T13:45:23.408 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T13:45:23.500 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898 branch: wip-adk-testing-2024-03-20-2326 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: adking@redhat.com first_in_suite: false job_id: '7614898' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-03-20-2326 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN - mons down, quorum - is down \(out of quorum\) - OSDMAP_FLAGS - PG_DEGRADED - OSD_DOWN - POOL_APP_NOT_ENABLED - CACHE_POOL_NO_HIT_SET - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: ba760091cd7bd2b0d23f4825ac856ba66450e988 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: ba760091cd7bd2b0d23f4825ac856ba66450e988 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-adk-testing-2024-03-20-2326 sha1: ba760091cd7bd2b0d23f4825ac856ba66450e988 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4016 sha1: ba760091cd7bd2b0d23f4825ac856ba66450e988 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-03-20-2326 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_ba760091cd7bd2b0d23f4825ac856ba66450e988/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: ba760091cd7bd2b0d23f4825ac856ba66450e988 targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWhCkuReM9xbtZKpVQKs+kZrE6aoEb7ZNCcv5kMELOyG62o4S0DlDfqRDFSemtZec7rJY0D/ZjFMj6ZWPvdD64= smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZuKDaELm0Kia4bvPb7Em1xTwNKRVDWQkiRD09GTgPkuopYjIIVVlLoyVxsUp9NPlJzCxiINkKKvNYRB/Vh6u8= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-21_11:21:25 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-21T13:45:23.500 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_ba760091cd7bd2b0d23f4825ac856ba66450e988/qa; will attempt to use it 2024-03-21T13:45:23.501 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_ba760091cd7bd2b0d23f4825ac856ba66450e988/qa/tasks 2024-03-21T13:45:23.501 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-21T13:45:23.502 INFO:teuthology.task.internal:Checking packages... 2024-03-21T13:45:23.529 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'ba760091cd7bd2b0d23f4825ac856ba66450e988' 2024-03-21T13:45:23.529 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-21T13:45:23.529 INFO:teuthology.packaging:ref: None 2024-03-21T13:45:23.529 INFO:teuthology.packaging:tag: None 2024-03-21T13:45:23.529 INFO:teuthology.packaging:branch: wip-adk-testing-2024-03-20-2326 2024-03-21T13:45:23.529 INFO:teuthology.packaging:sha1: ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:45:23.529 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-03-20-2326 2024-03-21T13:45:23.746 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2309.gba760091 2024-03-21T13:45:23.748 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-21T13:45:23.757 INFO:teuthology.task.internal:no buildpackages task found 2024-03-21T13:45:23.757 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-21T13:45:23.783 INFO:teuthology.task.internal:Saving configuration 2024-03-21T13:45:23.796 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-21T13:45:23.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T13:45:23.833 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 13:39:44.895663', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWhCkuReM9xbtZKpVQKs+kZrE6aoEb7ZNCcv5kMELOyG62o4S0DlDfqRDFSemtZec7rJY0D/ZjFMj6ZWPvdD64='} 2024-03-21T13:45:23.858 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 13:39:44.898204', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZuKDaELm0Kia4bvPb7Em1xTwNKRVDWQkiRD09GTgPkuopYjIIVVlLoyVxsUp9NPlJzCxiINkKKvNYRB/Vh6u8='} 2024-03-21T13:45:23.858 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-21T13:45:23.866 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-21T13:45:23.867 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-21T13:45:23.867 INFO:teuthology.run_tasks:Running task console_log... 2024-03-21T13:45:23.948 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0491dda160>, signals=[15]) 2024-03-21T13:45:23.949 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-21T13:45:23.956 INFO:teuthology.task.internal:Opening connections... 2024-03-21T13:45:23.956 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-03-21T13:45:23.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:45:24.067 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-03-21T13:45:24.068 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:45:24.156 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-21T13:45:24.168 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-03-21T13:45:24.185 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-03-21T13:45:24.185 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-03-21T13:45:24.241 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-03-21T13:45:24.241 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T13:45:24.242 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T13:45:24.243 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-03-21T13:45:24.268 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-03-21T13:45:24.286 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-03-21T13:45:24.286 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-03-21T13:45:24.343 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-03-21T13:45:24.343 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-21T13:45:24.344 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-21T13:45:24.345 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-03-21T13:45:24.368 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-21T13:45:24.378 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-21T13:45:24.386 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-21T13:45:24.387 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-03-21T13:45:24.389 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-03-21T13:45:24.405 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-21T13:45:24.414 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-21T13:45:24.414 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-03-21T13:45:24.446 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-03-21T13:45:24.473 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-21T13:45:24.537 INFO:teuthology.run_tasks:Running task kernel... 2024-03-21T13:45:24.559 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-21T13:45:24.559 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-21T13:45:24.559 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T13:45:24.560 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-21T13:45:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:45:24.576 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-21T13:45:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:45:24.577 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-21T13:45:24.632 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-361.el9.x86_64 2024-03-21T13:45:24.633 INFO:teuthology.task.kernel:Running kernel on smithi118: 5.14.0-361.el9.x86_64 2024-03-21T13:45:24.633 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum install -y kernel 2024-03-21T13:45:24.634 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-03-21T13:45:24.634 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-03-21T13:45:24.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-03-21T13:45:27.302 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 6.5 MB/s | 6.7 MB 00:01 2024-03-21T13:45:27.694 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-21T13:45:28.852 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 5.2 MB/s | 2.0 MB 00:00 2024-03-21T13:45:29.403 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-21T13:45:30.260 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-21T13:45:30.525 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.0 MB 00:01 2024-03-21T13:45:31.994 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-21T13:45:32.635 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-03-21T13:45:40.277 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.1 MB 00:04 2024-03-21T13:45:41.927 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-21T13:45:43.461 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 783 kB/s | 6.1 MB 00:08 2024-03-21T13:45:43.920 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-21T13:45:44.803 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 74 kB/s | 16 kB 00:00 2024-03-21T13:45:45.879 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-21T13:45:49.154 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-21T13:45:50.846 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-21T13:45:51.361 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T13:45:51.361 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T13:45:51.458 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:45:51.466 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:45:51.466 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-03-21T13:45:51.467 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:45:51.468 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 76 M 2024-03-21T13:45:51.468 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 123 M 2024-03-21T13:45:51.468 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:45:51.689 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 290 kB/s | 44 kB 00:00 2024-03-21T13:45:52.911 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-21T13:45:52.911 INFO:teuthology.orchestra.run.smithi118.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-21T13:45:53.001 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:45:53.012 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:45:53.012 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout:Install 4 Packages 2024-03-21T13:45:53.013 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:45:53.014 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 76 M 2024-03-21T13:45:53.014 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 123 M 2024-03-21T13:45:53.014 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:45:53.522 INFO:teuthology.orchestra.run.smithi118.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 195 kB/s | 44 kB 00:00 2024-03-21T13:45:54.022 INFO:teuthology.orchestra.run.smithi118.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-21T13:45:54.547 INFO:teuthology.orchestra.run.smithi118.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-21T13:45:55.589 INFO:teuthology.orchestra.run.smithi118.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-21T13:45:55.592 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:45:55.592 INFO:teuthology.orchestra.run.smithi118.stdout:Total 29 MB/s | 76 MB 00:02 2024-03-21T13:45:56.260 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:45:56.615 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:45:56.615 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:45:57.627 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:45:57.627 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:46:00.114 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:46:00.735 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T13:46:00.830 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T13:46:03.551 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T13:46:03.692 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T13:46:08.429 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T13:46:08.516 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:46:10.070 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 823 kB/s | 15 MB 00:18 2024-03-21T13:46:13.289 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:46:29.251 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 749 kB/s | 27 MB 00:37 2024-03-21T13:46:43.391 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 660 kB/s | 33 MB 00:51 2024-03-21T13:46:43.392 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:46:43.392 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.5 MB/s | 76 MB 00:51 2024-03-21T13:46:43.686 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:46:43.686 INFO:teuthology.orchestra.run.smithi118.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T13:46:43.686 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:46:43.730 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:46:44.080 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:46:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:46:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T13:46:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T13:46:44.116 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T13:46:44.497 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:46:44.497 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:46:44.499 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:46:44.820 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T13:46:44.916 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T13:46:44.916 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T13:46:44.916 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T13:46:44.916 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T13:46:44.916 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T13:46:44.917 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-21T13:46:44.917 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T13:46:44.917 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T13:46:44.995 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T13:46:44.996 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T13:46:44.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-03-21T13:46:45.035 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T13:46:45.036 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:46:45.037 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T13:46:45.038 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T13:46:45.038 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 5.14.0-430.el9.x86_64 2024-03-21T13:46:45.038 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T13:46:45.537 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:46:45.537 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:46:45.658 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-03-21T13:46:47.036 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T13:46:47.069 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-03-21T13:46:47.071 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-03-21T13:46:47.087 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.ZlcUatempL 2024-03-21T13:46:47.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZlcUatempL 2024-03-21T13:46:47.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.ZlcUatempL 2024-03-21T13:46:47.293 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.ZlcUatempL is 6KB 2024-03-21T13:46:47.342 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.ZlcUatempL 2024-03-21T13:46:47.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T13:46:47.421 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T13:46:47.421 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T13:46:47.421 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T13:46:47.421 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T13:46:47.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T13:46:47.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-03-21T13:46:48.145 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:46:48.798 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-21T13:46:48.885 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T13:46:51.774 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T13:46:51.911 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T13:46:57.014 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T13:46:57.045 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:47:02.169 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:47:17.573 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-21T13:47:17.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-21T13:47:17.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:47:32.776 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:47:32.776 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-21T13:47:32.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:47:32.820 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:47:33.207 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:47:33.207 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-21T13:47:33.207 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-21T13:47:33.207 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:47:33.631 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:47:33.983 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T13:47:34.082 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T13:47:34.083 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T13:47:34.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-21T13:47:34.084 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-21T13:47:34.084 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-21T13:47:34.084 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-21T13:47:34.084 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-03-21T13:47:34.084 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-03-21T13:47:34.166 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-21T13:47:34.166 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-21T13:47:34.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-21T13:47:34.204 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:47:34.205 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-03-21T13:47:34.206 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-21T13:47:34.206 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-430.el9.x86_64 2024-03-21T13:47:34.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-21T13:47:34.838 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-03-21T13:47:36.244 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-21T13:47:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-03-21T13:47:36.280 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-03-21T13:47:36.295 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.c1TfhrJEQ8 2024-03-21T13:47:36.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c1TfhrJEQ8 2024-03-21T13:47:36.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.c1TfhrJEQ8 2024-03-21T13:47:36.515 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.c1TfhrJEQ8 is 6KB 2024-03-21T13:47:36.564 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.c1TfhrJEQ8 2024-03-21T13:47:36.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-03-21T13:47:36.656 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-21T13:47:36.657 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-21T13:47:36.657 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-21T13:47:36.657 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-21T13:47:36.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-21T13:47:36.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-03-21T13:47:52.223 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-21T13:48:01.230 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-21T13:48:01.231 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:48:04.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-21T13:48:06.814 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-21T13:48:06.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-21T13:48:06.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:48:16.298 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-21T13:48:16.299 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:48:48.086 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-21T13:48:48.523 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-21T13:48:48.524 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-21T13:48:48.524 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-21T13:48:48.538 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-430.el9.x86_64 2024-03-21T13:48:48.538 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T13:48:48.539 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T13:48:48.539 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T13:48:49.539 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-21T13:48:49.540 DEBUG:teuthology.orchestra.run.smithi118:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T13:48:49.615 INFO:teuthology.orchestra.run.smithi118.stdout:ttyS1 2024-03-21T13:48:49.647 DEBUG:teuthology.parallel:result is None 2024-03-21T13:49:06.819 DEBUG:teuthology.orchestra.remote:timed out 2024-03-21T13:49:15.820 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-21T13:49:15.821 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:49:16.248 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-21T13:49:16.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-21T13:49:16.712 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-21T13:49:16.712 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-03-21T13:49:16.726 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-430.el9.x86_64 2024-03-21T13:49:16.726 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-21T13:49:16.726 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-21T13:49:16.726 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-21T13:49:17.727 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-21T13:49:17.727 DEBUG:teuthology.orchestra.run.smithi003:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-21T13:49:17.813 INFO:teuthology.orchestra.run.smithi003.stdout:ttyS1 2024-03-21T13:49:17.862 DEBUG:teuthology.parallel:result is None 2024-03-21T13:49:17.862 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-21T13:49:17.872 INFO:teuthology.task.internal:Creating test directory... 2024-03-21T13:49:17.873 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T13:49:17.875 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-21T13:49:17.893 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-21T13:49:17.899 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-21T13:49:17.917 INFO:teuthology.task.internal:Creating archive directory... 2024-03-21T13:49:17.917 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T13:49:17.930 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-21T13:49:17.989 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-21T13:49:17.996 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-21T13:49:17.996 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T13:49:17.998 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-21T13:49:18.025 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T13:49:18.033 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T13:49:18.038 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T13:49:18.047 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-21T13:49:18.049 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-21T13:49:18.059 INFO:teuthology.task.internal:Configuring sudo... 2024-03-21T13:49:18.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T13:49:18.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-21T13:49:18.119 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-21T13:49:18.128 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-21T13:49:18.129 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T13:49:18.149 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-21T13:49:18.176 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T13:49:18.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T13:49:18.286 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T13:49:18.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T13:49:18.409 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:49:18.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T13:49:18.478 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T13:49:18.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-21T13:49:18.573 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T13:49:18.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-21T13:49:18.703 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:49:18.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-21T13:49:18.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-03-21T13:49:18.775 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-03-21T13:49:18.821 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:49:18.862 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T13:49:19.218 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-21T13:49:19.227 INFO:teuthology.task.internal:Starting timer... 2024-03-21T13:49:19.227 INFO:teuthology.run_tasks:Running task pcp... 2024-03-21T13:49:19.288 INFO:teuthology.run_tasks:Running task selinux... 2024-03-21T13:49:19.388 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-21T13:49:19.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-03-21T13:49:19.443 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-03-21T13:49:19.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-03-21T13:49:19.507 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-03-21T13:49:19.509 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-21T13:49:19.510 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-03-21T13:49:19.534 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-03-21T13:49:19.534 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-03-21T13:49:19.555 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-03-21T13:49:19.555 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi118.front.sepia.ceph.com': 'permissive'} 2024-03-21T13:49:19.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T13:49:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:49:19.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T13:49:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T13:49:19.634 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-21T13:49:19.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-03-21T13:49:19.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-03-21T13:49:19.723 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-21T13:49:19.732 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-21T13:49:19.737 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-21T13:49:19.867 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-21T13:49:19.883 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-21T13:49:19.885 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi118.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-21T13:53:41.604 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi118.front.sepia.ceph.com')] 2024-03-21T13:53:41.605 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-21T13:53:41.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:53:41.699 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-03-21T13:53:41.781 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-03-21T13:53:41.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-21T13:53:41.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-21T13:53:41.868 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-21T13:53:41.941 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-21T13:53:41.942 INFO:teuthology.run_tasks:Running task clock... 2024-03-21T13:53:41.952 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-21T13:53:41.952 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T13:53:41.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:53:41.955 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-21T13:53:41.955 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T13:53:41.985 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T13:53:42.002 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T13:53:42.012 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-21T13:53:42.029 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-21T13:53:42.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-03-21T13:53:42.046 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-03-21T13:53:42.058 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-03-21T13:53:42.063 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T13:53:42.071 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-03-21T13:53:42.079 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T13:53:42.087 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-21T13:53:42.104 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-21T13:53:42.132 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:53:42.139 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:53:42.139 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-21T13:53:42.139 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.139 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.139 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.139 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.158 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-21T13:53:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T13:53:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-21T13:53:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.165 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T13:53:42.166 INFO:teuthology.run_tasks:Running task install... 2024-03-21T13:53:42.174 DEBUG:teuthology.task.install:project ceph 2024-03-21T13:53:42.174 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988'}} 2024-03-21T13:53:42.175 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988'} 2024-03-21T13:53:42.175 INFO:teuthology.task.install:Using flavor: default 2024-03-21T13:53:42.180 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-21T13:53:42.181 INFO:teuthology.task.install:extra packages: [] 2024-03-21T13:53:42.181 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988', 'tag': None, 'wait_for_package': False} 2024-03-21T13:53:42.181 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:53:42.183 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988', 'tag': None, 'wait_for_package': False} 2024-03-21T13:53:42.184 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:53:42.410 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/ 2024-03-21T13:53:42.410 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2309.gba760091 2024-03-21T13:53:42.424 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/ 2024-03-21T13:53:42.424 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2309.gba760091 2024-03-21T13:53:42.519 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T13:53:42.519 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:53:42.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T13:53:42.532 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-21T13:53:42.532 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:53:42.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-21T13:53:42.548 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-21T13:53:42.548 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba760091cd7bd2b0d23f4825ac856ba66450e988/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T13:53:42.560 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-21T13:53:42.560 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ba760091cd7bd2b0d23f4825ac856ba66450e988/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-21T13:53:42.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T13:53:42.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-21T13:53:42.697 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T13:53:42.710 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-21T13:53:42.763 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-03-21T13:53:42.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-03-21T13:53:42.777 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-03-21T13:53:42.779 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-03-21T13:53:43.005 INFO:teuthology.orchestra.run.smithi003.stdout:74 files removed 2024-03-21T13:53:43.010 INFO:teuthology.orchestra.run.smithi118.stdout:74 files removed 2024-03-21T13:53:43.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw 2024-03-21T13:53:43.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw 2024-03-21T13:53:43.787 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 184 kB/s | 83 kB 00:00 2024-03-21T13:53:43.826 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 170 kB/s | 83 kB 00:00 2024-03-21T13:53:44.202 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-03-21T13:53:44.246 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 28 kB/s | 12 kB 00:00 2024-03-21T13:53:44.672 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-03-21T13:53:44.719 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 4.0 kB/s | 1.9 kB 00:00 2024-03-21T13:53:45.372 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-21T13:53:45.383 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-21T13:53:46.864 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-21T13:53:46.958 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-21T13:53:51.587 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-21T13:53:53.505 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-21T13:53:54.448 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-03-21T13:53:56.758 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-21T13:54:01.808 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-21T13:54:03.343 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-03-21T13:54:05.072 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-21T13:54:05.830 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 717 kB/s | 6.1 MB 00:08 2024-03-21T13:54:07.356 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 47 kB/s | 16 kB 00:00 2024-03-21T13:54:08.266 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-21T13:54:09.986 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-21T13:54:12.380 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.0.0-2309.gba760091.el9 ceph 9.9 M 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.0.0-2309.gba760091.el9 ceph 3.4 M 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.0.0-2309.gba760091.el9 ceph 3.1 M 2024-03-21T13:54:12.400 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.0.0-2309.gba760091.el9 ceph 5.4 M 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.0.0-2309.gba760091.el9 ceph 20 M 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.0.0-2309.gba760091.el9 ceph 25 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.0.0-2309.gba760091.el9 ceph 748 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.0.0-2309.gba760091.el9 ceph 501 k 2024-03-21T13:54:12.401 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.0.0-2309.gba760091.el9 ceph 5.1 M 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: x86_64 2:19.0.0-2309.gba760091.el9 ceph 45 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.0.0-2309.gba760091.el9 ceph 134 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.0.0-2309.gba760091.el9 ceph 162 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.0.0-2309.gba760091.el9 ceph 321 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.0.0-2309.gba760091.el9 ceph 299 k 2024-03-21T13:54:12.402 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.0.0-2309.gba760091.el9 ceph 99 k 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout:Install 30 Packages 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 2 Packages 2024-03-21T13:54:12.403 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:12.404 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 59 M 2024-03-21T13:54:12.404 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:54:13.457 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-21T13:54:13.796 INFO:teuthology.orchestra.run.smithi118.stdout:(1/32): ceph-base-19.0.0-2309.gba760091.el9.x86 5.9 MB/s | 5.4 MB 00:00 2024-03-21T13:54:13.829 INFO:teuthology.orchestra.run.smithi118.stdout:(2/32): ceph-selinux-19.0.0-2309.gba760091.el9. 760 kB/s | 25 kB 00:00 2024-03-21T13:54:13.955 INFO:teuthology.orchestra.run.smithi118.stdout:(3/32): libcephfs2-19.0.0-2309.gba760091.el9.x8 5.8 MB/s | 748 kB 00:00 2024-03-21T13:54:14.055 INFO:teuthology.orchestra.run.smithi118.stdout:(4/32): libradosstriper1-19.0.0-2309.gba760091. 4.9 MB/s | 501 kB 00:00 2024-03-21T13:54:14.172 INFO:teuthology.orchestra.run.smithi118.stdout:(5/32): ceph-radosgw-19.0.0-2309.gba760091.el9. 7.7 MB/s | 9.9 MB 00:01 2024-03-21T13:54:14.230 INFO:teuthology.orchestra.run.smithi118.stdout:(6/32): python3-ceph-argparse-19.0.0-2309.gba76 775 kB/s | 45 kB 00:00 2024-03-21T13:54:14.289 INFO:teuthology.orchestra.run.smithi118.stdout:(7/32): python3-ceph-common-19.0.0-2309.gba7600 2.2 MB/s | 134 kB 00:00 2024-03-21T13:54:14.348 INFO:teuthology.orchestra.run.smithi118.stdout:(8/32): python3-cephfs-19.0.0-2309.gba760091.el 2.7 MB/s | 162 kB 00:00 2024-03-21T13:54:14.423 INFO:teuthology.orchestra.run.smithi118.stdout:(9/32): python3-rados-19.0.0-2309.gba760091.el9 4.2 MB/s | 321 kB 00:00 2024-03-21T13:54:14.490 INFO:teuthology.orchestra.run.smithi118.stdout:(10/32): python3-rbd-19.0.0-2309.gba760091.el9. 4.4 MB/s | 299 kB 00:00 2024-03-21T13:54:14.582 INFO:teuthology.orchestra.run.smithi118.stdout:(11/32): librgw2-19.0.0-2309.gba760091.el9.x86_ 9.8 MB/s | 5.1 MB 00:00 2024-03-21T13:54:14.607 INFO:teuthology.orchestra.run.smithi118.stdout:(12/32): python3-rgw-19.0.0-2309.gba760091.el9. 850 kB/s | 99 kB 00:00 2024-03-21T13:54:14.774 INFO:teuthology.orchestra.run.smithi118.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 551 kB/s | 106 kB 00:00 2024-03-21T13:54:14.824 INFO:teuthology.orchestra.run.smithi118.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 733 kB/s | 159 kB 00:00 2024-03-21T13:54:14.858 INFO:teuthology.orchestra.run.smithi118.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 527 kB/s | 44 kB 00:00 2024-03-21T13:54:15.226 INFO:teuthology.orchestra.run.smithi118.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.6 MB/s | 661 kB 00:00 2024-03-21T13:54:15.326 INFO:teuthology.orchestra.run.smithi118.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 605 kB/s | 282 kB 00:00 2024-03-21T13:54:15.342 INFO:teuthology.orchestra.run.smithi118.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 353 kB/s | 41 kB 00:00 2024-03-21T13:54:15.601 INFO:teuthology.orchestra.run.smithi118.stdout:(19/32): ceph-common-19.0.0-2309.gba760091.el9. 7.5 MB/s | 20 MB 00:02 2024-03-21T13:54:15.965 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:54:15.968 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:54:15.968 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:54:15.968 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:54:15.968 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:54:15.968 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:19.0.0-2309.gba760091.el9 ceph 9.9 M 2024-03-21T13:54:15.968 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:19.0.0-2309.gba760091.el9 ceph 3.4 M 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:19.0.0-2309.gba760091.el9 ceph 3.1 M 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:19.0.0-2309.gba760091.el9 ceph 5.4 M 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:19.0.0-2309.gba760091.el9 ceph 20 M 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:19.0.0-2309.gba760091.el9 ceph 25 k 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:19.0.0-2309.gba760091.el9 ceph 748 k 2024-03-21T13:54:15.969 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:19.0.0-2309.gba760091.el9 ceph 501 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:19.0.0-2309.gba760091.el9 ceph 5.1 M 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: x86_64 2:19.0.0-2309.gba760091.el9 ceph 45 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:19.0.0-2309.gba760091.el9 ceph 134 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:19.0.0-2309.gba760091.el9 ceph 162 k 2024-03-21T13:54:15.970 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:19.0.0-2309.gba760091.el9 ceph 321 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:19.0.0-2309.gba760091.el9 ceph 299 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:19.0.0-2309.gba760091.el9 ceph 99 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:54:15.971 INFO:teuthology.orchestra.run.smithi003.stdout:Install 30 Packages 2024-03-21T13:54:15.972 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 2 Packages 2024-03-21T13:54:15.972 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:15.972 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 59 M 2024-03-21T13:54:15.972 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:54:16.143 INFO:teuthology.orchestra.run.smithi118.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 44 kB/s | 35 kB 00:00 2024-03-21T13:54:16.243 INFO:teuthology.orchestra.run.smithi118.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 69 kB/s | 44 kB 00:00 2024-03-21T13:54:16.327 INFO:teuthology.orchestra.run.smithi118.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-21T13:54:16.385 INFO:teuthology.orchestra.run.smithi118.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-21T13:54:16.436 INFO:teuthology.orchestra.run.smithi118.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-03-21T13:54:16.486 INFO:teuthology.orchestra.run.smithi118.stdout:(25/32): qatlib-23.11.0-1.el9.x86_64.rpm 187 kB/s | 216 kB 00:01 2024-03-21T13:54:16.511 INFO:teuthology.orchestra.run.smithi118.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 908 kB/s | 67 kB 00:00 2024-03-21T13:54:16.620 INFO:teuthology.orchestra.run.smithi118.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-21T13:54:16.645 INFO:teuthology.orchestra.run.smithi118.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-21T13:54:16.704 INFO:teuthology.orchestra.run.smithi118.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-21T13:54:16.821 INFO:teuthology.orchestra.run.smithi118.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.4 MB/s | 838 kB 00:00 2024-03-21T13:54:16.942 INFO:teuthology.orchestra.run.smithi003.stdout:(1/32): ceph-base-19.0.0-2309.gba760091.el9.x86 6.1 MB/s | 5.4 MB 00:00 2024-03-21T13:54:17.009 INFO:teuthology.orchestra.run.smithi003.stdout:(2/32): ceph-selinux-19.0.0-2309.gba760091.el9. 379 kB/s | 25 kB 00:00 2024-03-21T13:54:17.167 INFO:teuthology.orchestra.run.smithi003.stdout:(3/32): libcephfs2-19.0.0-2309.gba760091.el9.x8 4.6 MB/s | 748 kB 00:00 2024-03-21T13:54:17.213 INFO:teuthology.orchestra.run.smithi118.stdout:(31/32): librados2-19.0.0-2309.gba760091.el9.x8 6.0 MB/s | 3.4 MB 00:00 2024-03-21T13:54:17.292 INFO:teuthology.orchestra.run.smithi003.stdout:(4/32): libradosstriper1-19.0.0-2309.gba760091. 3.9 MB/s | 501 kB 00:00 2024-03-21T13:54:17.510 INFO:teuthology.orchestra.run.smithi003.stdout:(5/32): ceph-radosgw-19.0.0-2309.gba760091.el9. 6.8 MB/s | 9.9 MB 00:01 2024-03-21T13:54:17.593 INFO:teuthology.orchestra.run.smithi003.stdout:(6/32): python3-ceph-argparse-19.0.0-2309.gba76 543 kB/s | 45 kB 00:00 2024-03-21T13:54:17.677 INFO:teuthology.orchestra.run.smithi003.stdout:(7/32): python3-ceph-common-19.0.0-2309.gba7600 1.6 MB/s | 134 kB 00:00 2024-03-21T13:54:17.769 INFO:teuthology.orchestra.run.smithi003.stdout:(8/32): python3-cephfs-19.0.0-2309.gba760091.el 1.7 MB/s | 162 kB 00:00 2024-03-21T13:54:17.869 INFO:teuthology.orchestra.run.smithi003.stdout:(9/32): python3-rados-19.0.0-2309.gba760091.el9 3.1 MB/s | 321 kB 00:00 2024-03-21T13:54:17.944 INFO:teuthology.orchestra.run.smithi003.stdout:(10/32): librgw2-19.0.0-2309.gba760091.el9.x86_ 7.9 MB/s | 5.1 MB 00:00 2024-03-21T13:54:17.969 INFO:teuthology.orchestra.run.smithi003.stdout:(11/32): python3-rbd-19.0.0-2309.gba760091.el9. 2.9 MB/s | 299 kB 00:00 2024-03-21T13:54:18.021 INFO:teuthology.orchestra.run.smithi118.stdout:(32/32): librbd1-19.0.0-2309.gba760091.el9.x86_ 2.4 MB/s | 3.1 MB 00:01 2024-03-21T13:54:18.026 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:54:18.026 INFO:teuthology.orchestra.run.smithi118.stdout:Total 10 MB/s | 59 MB 00:05 2024-03-21T13:54:18.028 INFO:teuthology.orchestra.run.smithi003.stdout:(12/32): python3-rgw-19.0.0-2309.gba760091.el9. 1.2 MB/s | 99 kB 00:00 2024-03-21T13:54:18.062 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:54:18.103 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:54:18.103 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:54:18.229 INFO:teuthology.orchestra.run.smithi003.stdout:(13/32): ceph-common-19.0.0-2309.gba760091.el9. 9.5 MB/s | 20 MB 00:02 2024-03-21T13:54:18.270 INFO:teuthology.orchestra.run.smithi003.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 352 kB/s | 106 kB 00:00 2024-03-21T13:54:18.312 INFO:teuthology.orchestra.run.smithi003.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 563 kB/s | 159 kB 00:00 2024-03-21T13:54:18.346 INFO:teuthology.orchestra.run.smithi003.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 377 kB/s | 44 kB 00:00 2024-03-21T13:54:18.421 INFO:teuthology.orchestra.run.smithi003.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-03-21T13:54:18.505 INFO:teuthology.orchestra.run.smithi003.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-03-21T13:54:18.555 INFO:teuthology.orchestra.run.smithi003.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 197 kB/s | 41 kB 00:00 2024-03-21T13:54:18.598 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:54:18.598 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:54:18.639 INFO:teuthology.orchestra.run.smithi003.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 420 kB/s | 35 kB 00:00 2024-03-21T13:54:18.665 INFO:teuthology.orchestra.run.smithi003.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 412 kB/s | 44 kB 00:00 2024-03-21T13:54:18.697 INFO:teuthology.orchestra.run.smithi003.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 786 kB/s | 216 kB 00:00 2024-03-21T13:54:18.964 INFO:teuthology.orchestra.run.smithi003.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-21T13:54:19.173 INFO:teuthology.orchestra.run.smithi003.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 578 kB/s | 309 kB 00:00 2024-03-21T13:54:19.307 INFO:teuthology.orchestra.run.smithi003.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-21T13:54:19.516 INFO:teuthology.orchestra.run.smithi003.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-21T13:54:19.932 INFO:teuthology.orchestra.run.smithi003.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 4.4 MB 00:01 2024-03-21T13:54:19.974 INFO:teuthology.orchestra.run.smithi003.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-21T13:54:20.016 INFO:teuthology.orchestra.run.smithi003.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 383 kB/s | 192 kB 00:00 2024-03-21T13:54:20.155 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:54:20.229 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-21T13:54:20.268 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-21T13:54:20.293 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-21T13:54:20.308 INFO:teuthology.orchestra.run.smithi003.stdout:(30/32): librados2-19.0.0-2309.gba760091.el9.x8 10 MB/s | 3.4 MB 00:00 2024-03-21T13:54:20.324 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:54:20.341 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:54:20.391 INFO:teuthology.orchestra.run.smithi003.stdout:(31/32): librbd1-19.0.0-2309.gba760091.el9.x86_ 8.4 MB/s | 3.1 MB 00:00 2024-03-21T13:54:20.550 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:54:20.606 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:54:20.783 INFO:teuthology.orchestra.run.smithi003.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 1.9 MB/s | 1.6 MB 00:00 2024-03-21T13:54:20.788 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:54:20.788 INFO:teuthology.orchestra.run.smithi003.stdout:Total 12 MB/s | 59 MB 00:04 2024-03-21T13:54:20.823 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:54:20.863 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:54:20.863 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:54:21.078 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-21T13:54:21.195 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.0.0-2309.gba760091.el9.x86_64 7/34 2024-03-21T13:54:21.287 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.0.0-2309.gba760091.el9.x86_64 7/34 2024-03-21T13:54:21.374 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 8/34 2024-03-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:54:21.375 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:54:21.397 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 9/34 2024-03-21T13:54:21.460 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 9/34 2024-03-21T13:54:21.488 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-21T13:54:21.532 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.0.0-2309.gba760091.el9. 11/34 2024-03-21T13:54:21.600 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 12/34 2024-03-21T13:54:21.624 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_6 13/34 2024-03-21T13:54:21.671 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_6 13/34 2024-03-21T13:54:21.698 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-21T13:54:21.806 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-21T13:54:21.841 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-21T13:54:21.882 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-21T13:54:21.914 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-21T13:54:21.943 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-21T13:54:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-21T13:54:22.291 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-21T13:54:22.322 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.0.0-2309.gba760091.el9.x86_64 22/34 2024-03-21T13:54:22.404 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.0.0-2309.gba760091.el9.x86_64 22/34 2024-03-21T13:54:22.435 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 23/34 2024-03-21T13:54:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-21T13:54:22.855 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:54:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-21T13:54:22.947 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-21T13:54:22.979 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-21T13:54:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.0.0-2309.gba760091.el9.x86_64 26/34 2024-03-21T13:54:23.005 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-21T13:54:23.028 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:54:23.045 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-21T13:54:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.0.0-2309.gba760091.el9.x86_64 26/34 2024-03-21T13:54:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 27/34 2024-03-21T13:54:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.0.0-2309.gba760091.el9.x8 28/34 2024-03-21T13:54:23.236 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:54:23.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-21T13:54:23.642 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-21T13:54:23.690 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:19.0.0-2309.gba760091.el9.x86_64 7/34 2024-03-21T13:54:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:19.0.0-2309.gba760091.el9.x86_64 7/34 2024-03-21T13:54:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 8/34 2024-03-21T13:54:23.900 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 9/34 2024-03-21T13:54:23.954 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 9/34 2024-03-21T13:54:23.983 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-21T13:54:24.027 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:19.0.0-2309.gba760091.el9. 11/34 2024-03-21T13:54:24.097 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 12/34 2024-03-21T13:54:24.119 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_6 13/34 2024-03-21T13:54:24.173 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_6 13/34 2024-03-21T13:54:24.201 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-21T13:54:24.320 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-21T13:54:24.360 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-21T13:54:24.401 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-21T13:54:24.442 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-21T13:54:24.474 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-21T13:54:24.504 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-21T13:54:24.834 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-21T13:54:24.867 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:19.0.0-2309.gba760091.el9.x86_64 22/34 2024-03-21T13:54:24.958 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:19.0.0-2309.gba760091.el9.x86_64 22/34 2024-03-21T13:54:24.988 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 23/34 2024-03-21T13:54:25.032 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-21T13:54:25.410 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 29/34 2024-03-21T13:54:25.449 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 29/34 2024-03-21T13:54:25.519 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-21T13:54:25.544 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:19.0.0-2309.gba760091.el9.x86_64 26/34 2024-03-21T13:54:25.599 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:19.0.0-2309.gba760091.el9.x86_64 26/34 2024-03-21T13:54:25.670 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 27/34 2024-03-21T13:54:25.714 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:19.0.0-2309.gba760091.el9.x8 28/34 2024-03-21T13:54:25.994 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 29/34 2024-03-21T13:54:26.028 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 30/34 2024-03-21T13:54:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 30/34 2024-03-21T13:54:26.112 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T13:54:26.112 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T13:54:26.112 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:26.154 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 31/34 2024-03-21T13:54:28.022 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 29/34 2024-03-21T13:54:28.060 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 29/34 2024-03-21T13:54:28.633 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 29/34 2024-03-21T13:54:28.688 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 30/34 2024-03-21T13:54:28.769 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 30/34 2024-03-21T13:54:28.769 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-21T13:54:28.769 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-21T13:54:28.769 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:28.789 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 31/34 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 31/34 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-03-21T13:54:36.731 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-03-21T13:54:36.732 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-03-21T13:54:36.732 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:37.705 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 32/34 2024-03-21T13:54:37.730 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 32/34 2024-03-21T13:54:37.730 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:54:37.730 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T13:54:37.730 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:54:37.730 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:54:37.730 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:37.732 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:54:37.808 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:54:37.808 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 1/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 2/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 3/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 4/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 5/34 2024-03-21T13:54:38.927 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_6 6/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.0.0-2309.gba760091.el9.x86_64 7/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2309.gba760091.el9. 8/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.0.0-2309.gba760091.el9.x8 9/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 10/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 11/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 12/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 13/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-21T13:54:38.928 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.0.0-2309.gba760091.el9.x86_64 31/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-21T13:54:38.929 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.0.0-2309.gba760091.el9.x86_64 33/34 2024-03-21T13:54:39.026 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 31/34 2024-03-21T13:54:39.026 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-03-21T13:54:39.027 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:39.979 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 32/34 2024-03-21T13:54:40.003 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 32/34 2024-03-21T13:54:40.003 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:54:40.003 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-21T13:54:40.004 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:54:40.004 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-21T13:54:40.004 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:40.005 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:54:40.089 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-21T13:54:40.090 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:54:40.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:54:40.095 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:40.095 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-03-21T13:54:40.095 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.095 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.095 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.096 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T13:54:40.097 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T13:54:40.098 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T13:54:40.098 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T13:54:40.098 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T13:54:40.098 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:40.098 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:54:40.469 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-test 2024-03-21T13:54:41.036 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:32 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:54:41.184 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:54:41.184 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:54:41.184 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:54:41.184 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.0.0-2309.gba760091.el9 ceph 48 M 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.0.0-2309.gba760091.el9 ceph 163 k 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-03-21T13:54:41.185 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:41.186 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 49 M 2024-03-21T13:54:41.186 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 206 M 2024-03-21T13:54:41.186 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 1/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 2/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 3/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 4/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 5/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_6 6/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:19.0.0-2309.gba760091.el9.x86_64 7/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2309.gba760091.el9. 8/34 2024-03-21T13:54:41.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:19.0.0-2309.gba760091.el9.x8 9/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 10/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 11/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 12/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 13/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-21T13:54:41.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:19.0.0-2309.gba760091.el9.x86_64 31/34 2024-03-21T13:54:41.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-21T13:54:41.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:19.0.0-2309.gba760091.el9.x86_64 33/34 2024-03-21T13:54:41.638 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-21T13:54:41.694 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.3 MB/s | 300 kB 00:00 2024-03-21T13:54:41.721 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): libcephsqlite-19.0.0-2309.gba760091.el9. 656 kB/s | 163 kB 00:00 2024-03-21T13:54:41.747 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-21T13:54:42.006 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 660 kB/s | 187 kB 00:00 2024-03-21T13:54:42.134 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-21T13:54:42.134 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:42.134 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-21T13:54:42.135 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.136 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:42.137 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:54:42.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-test 2024-03-21T13:54:43.033 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:30 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:54:43.196 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:54:43.196 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:19.0.0-2309.gba760091.el9 ceph 48 M 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:19.0.0-2309.gba760091.el9 ceph 163 k 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:54:43.197 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:54:43.198 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-21T13:54:43.198 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:43.198 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 49 M 2024-03-21T13:54:43.198 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 206 M 2024-03-21T13:54:43.198 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:54:43.588 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 945 kB/s | 217 kB 00:00 2024-03-21T13:54:43.655 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-21T13:54:43.688 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-21T13:54:43.755 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): libcephsqlite-19.0.0-2309.gba760091.el9. 411 kB/s | 163 kB 00:00 2024-03-21T13:54:43.839 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-21T13:54:44.839 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): ceph-test-19.0.0-2309.gba760091.el9.x86_ 14 MB/s | 48 MB 00:03 2024-03-21T13:54:44.842 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:54:44.842 INFO:teuthology.orchestra.run.smithi118.stdout:Total 13 MB/s | 49 MB 00:03 2024-03-21T13:54:44.857 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:54:44.877 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:54:44.877 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:54:45.262 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:54:45.262 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:54:45.843 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:54:45.924 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-21T13:54:45.967 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-21T13:54:46.000 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:54:46.039 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-21T13:54:46.067 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 5/6 2024-03-21T13:54:47.214 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): ceph-test-19.0.0-2309.gba760091.el9.x86_ 12 MB/s | 48 MB 00:03 2024-03-21T13:54:47.217 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:54:47.217 INFO:teuthology.orchestra.run.smithi003.stdout:Total 12 MB/s | 49 MB 00:04 2024-03-21T13:54:47.231 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:54:47.254 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:54:47.254 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:54:47.657 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:54:47.658 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:54:48.251 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:54:48.316 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-21T13:54:48.352 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-21T13:54:48.393 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:54:48.439 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-21T13:54:48.466 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 5/6 2024-03-21T13:54:50.595 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 5/6 2024-03-21T13:54:50.666 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 6/6 2024-03-21T13:54:53.228 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 5/6 2024-03-21T13:54:53.309 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 6/6 2024-03-21T13:54:53.533 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 6/6 2024-03-21T13:54:53.534 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 1/6 2024-03-21T13:54:53.534 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 2/6 2024-03-21T13:54:53.534 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:54:53.534 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-21T13:54:53.534 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-16.el9.x86_64 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:53.903 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T13:54:53.904 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T13:54:53.904 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T13:54:53.904 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:53.904 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:54:54.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph 2024-03-21T13:54:54.781 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:00:45 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:54:54.937 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:54:54.939 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-03-21T13:54:54.939 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:54:54.939 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-03-21T13:54:54.939 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:54:54.939 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.0.0-2309.gba760091.el9 ceph 6.6 k 2024-03-21T13:54:54.939 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.0.0-2309.gba760091.el9 ceph 2.3 M 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.0.0-2309.gba760091.el9 ceph 1.5 M 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 249 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.0.0-2309.gba760091.el9 ceph 4.7 M 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.0.0-2309.gba760091.el9 ceph 17 M 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T13:54:54.940 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T13:54:54.941 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T13:54:54.942 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout:========================================================================================= 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout:Install 36 Packages 2024-03-21T13:54:54.943 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:54:54.944 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 30 M 2024-03-21T13:54:54.944 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 107 M 2024-03-21T13:54:54.944 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:54:55.618 INFO:teuthology.orchestra.run.smithi118.stdout:(1/36): ceph-19.0.0-2309.gba760091.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-21T13:54:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 6/6 2024-03-21T13:54:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 1/6 2024-03-21T13:54:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 2/6 2024-03-21T13:54:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-21T13:54:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-21T13:54:55.787 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-21T13:54:55.985 INFO:teuthology.orchestra.run.smithi118.stdout:(2/36): ceph-mon-19.0.0-2309.gba760091.el9.x86_ 13 MB/s | 4.7 MB 00:00 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-16.el9.x86_64 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:54:56.165 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-21T13:54:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-21T13:54:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-21T13:54:56.166 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:56.166 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:54:56.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph 2024-03-21T13:54:56.835 INFO:teuthology.orchestra.run.smithi118.stdout:(3/36): ceph-osd-19.0.0-2309.gba760091.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-21T13:54:56.886 INFO:teuthology.orchestra.run.smithi118.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2309.gba76 4.9 MB/s | 249 kB 00:00 2024-03-21T13:54:57.060 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:44 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:54:57.153 INFO:teuthology.orchestra.run.smithi118.stdout:(5/36): ceph-mgr-19.0.0-2309.gba760091.el9.x86_ 896 kB/s | 1.5 MB 00:01 2024-03-21T13:54:57.178 INFO:teuthology.orchestra.run.smithi118.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 640 kB/s | 187 kB 00:00 2024-03-21T13:54:57.218 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:54:57.221 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-21T13:54:57.221 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:54:57.221 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:19.0.0-2309.gba760091.el9 ceph 6.6 k 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:19.0.0-2309.gba760091.el9 ceph 2.3 M 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:19.0.0-2309.gba760091.el9 ceph 1.5 M 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 249 k 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:19.0.0-2309.gba760091.el9 ceph 4.7 M 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:19.0.0-2309.gba760091.el9 ceph 17 M 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-21T13:54:57.222 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-21T13:54:57.223 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout:========================================================================================= 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout:Install 36 Packages 2024-03-21T13:54:57.224 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:54:57.225 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 30 M 2024-03-21T13:54:57.225 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 107 M 2024-03-21T13:54:57.225 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:54:57.245 INFO:teuthology.orchestra.run.smithi118.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 144 kB 00:00 2024-03-21T13:54:57.262 INFO:teuthology.orchestra.run.smithi118.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 376 kB/s | 32 kB 00:00 2024-03-21T13:54:57.304 INFO:teuthology.orchestra.run.smithi118.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 806 kB/s | 47 kB 00:00 2024-03-21T13:54:57.321 INFO:teuthology.orchestra.run.smithi118.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 660 kB/s | 39 kB 00:00 2024-03-21T13:54:57.388 INFO:teuthology.orchestra.run.smithi118.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-03-21T13:54:57.530 INFO:teuthology.orchestra.run.smithi118.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.5 MB/s | 1.1 MB 00:00 2024-03-21T13:54:57.555 INFO:teuthology.orchestra.run.smithi118.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 618 kB/s | 103 kB 00:00 2024-03-21T13:54:57.647 INFO:teuthology.orchestra.run.smithi118.stdout:(14/36): ceph-mds-19.0.0-2309.gba760091.el9.x86 1.0 MB/s | 2.3 MB 00:02 2024-03-21T13:54:57.673 INFO:teuthology.orchestra.run.smithi118.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 873 kB/s | 124 kB 00:00 2024-03-21T13:54:57.698 INFO:teuthology.orchestra.run.smithi118.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.3 MB/s | 187 kB 00:00 2024-03-21T13:54:57.840 INFO:teuthology.orchestra.run.smithi118.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 652 kB/s | 125 kB 00:00 2024-03-21T13:54:57.907 INFO:teuthology.orchestra.run.smithi118.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 107 kB/s | 22 kB 00:00 2024-03-21T13:54:58.049 INFO:teuthology.orchestra.run.smithi003.stdout:(1/36): ceph-19.0.0-2309.gba760091.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-21T13:54:58.332 INFO:teuthology.orchestra.run.smithi118.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 93 kB/s | 43 kB 00:00 2024-03-21T13:54:58.357 INFO:teuthology.orchestra.run.smithi118.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 229 kB/s | 151 kB 00:00 2024-03-21T13:54:58.432 INFO:teuthology.orchestra.run.smithi003.stdout:(2/36): ceph-mon-19.0.0-2309.gba760091.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-03-21T13:54:58.499 INFO:teuthology.orchestra.run.smithi118.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2024-03-21T13:54:58.533 INFO:teuthology.orchestra.run.smithi118.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 274 kB/s | 172 kB 00:00 2024-03-21T13:54:58.608 INFO:teuthology.orchestra.run.smithi118.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-03-21T13:54:58.641 INFO:teuthology.orchestra.run.smithi118.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-21T13:54:58.733 INFO:teuthology.orchestra.run.smithi118.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 893 kB/s | 358 kB 00:00 2024-03-21T13:54:58.759 INFO:teuthology.orchestra.run.smithi118.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 129 kB/s | 19 kB 00:00 2024-03-21T13:54:58.784 INFO:teuthology.orchestra.run.smithi118.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 138 kB/s | 20 kB 00:00 2024-03-21T13:54:58.859 INFO:teuthology.orchestra.run.smithi118.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 369 kB/s | 46 kB 00:00 2024-03-21T13:54:58.934 INFO:teuthology.orchestra.run.smithi118.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 525 kB/s | 79 kB 00:00 2024-03-21T13:54:58.968 INFO:teuthology.orchestra.run.smithi118.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-21T13:54:58.993 INFO:teuthology.orchestra.run.smithi118.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-21T13:54:59.058 INFO:teuthology.orchestra.run.smithi003.stdout:(3/36): ceph-mds-19.0.0-2309.gba760091.el9.x86_ 2.0 MB/s | 2.3 MB 00:01 2024-03-21T13:54:59.077 INFO:teuthology.orchestra.run.smithi118.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 632 kB/s | 90 kB 00:00 2024-03-21T13:54:59.091 INFO:teuthology.orchestra.run.smithi003.stdout:(4/36): ceph-mgr-19.0.0-2309.gba760091.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-21T13:54:59.102 INFO:teuthology.orchestra.run.smithi118.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 267 kB/s | 36 kB 00:00 2024-03-21T13:54:59.136 INFO:teuthology.orchestra.run.smithi118.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-21T13:54:59.245 INFO:teuthology.orchestra.run.smithi118.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 141 kB/s | 20 kB 00:00 2024-03-21T13:54:59.292 INFO:teuthology.orchestra.run.smithi003.stdout:(5/36): ceph-osd-19.0.0-2309.gba760091.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-21T13:54:59.317 INFO:teuthology.orchestra.run.smithi003.stdout:(6/36): ceph-mgr-modules-core-19.0.0-2309.gba76 963 kB/s | 249 kB 00:00 2024-03-21T13:54:59.370 INFO:teuthology.orchestra.run.smithi118.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-03-21T13:54:59.375 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:54:59.375 INFO:teuthology.orchestra.run.smithi118.stdout:Total 6.8 MB/s | 30 MB 00:04 2024-03-21T13:54:59.384 INFO:teuthology.orchestra.run.smithi003.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 640 kB/s | 187 kB 00:00 2024-03-21T13:54:59.398 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:54:59.409 INFO:teuthology.orchestra.run.smithi003.stdout:(8/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-21T13:54:59.434 INFO:teuthology.orchestra.run.smithi003.stdout:(9/36): python3-markupsafe-1.1.1-12.el9.x86_64. 270 kB/s | 32 kB 00:00 2024-03-21T13:54:59.438 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:54:59.438 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:54:59.459 INFO:teuthology.orchestra.run.smithi003.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-21T13:54:59.484 INFO:teuthology.orchestra.run.smithi003.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-21T13:54:59.510 INFO:teuthology.orchestra.run.smithi003.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-21T13:54:59.543 INFO:teuthology.orchestra.run.smithi003.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-21T13:54:59.568 INFO:teuthology.orchestra.run.smithi003.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.2 MB/s | 124 kB 00:00 2024-03-21T13:54:59.594 INFO:teuthology.orchestra.run.smithi003.stdout:(15/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-21T13:54:59.619 INFO:teuthology.orchestra.run.smithi003.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-21T13:54:59.678 INFO:teuthology.orchestra.run.smithi003.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-03-21T13:54:59.694 INFO:teuthology.orchestra.run.smithi003.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-21T13:54:59.863 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:54:59.864 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:54:59.978 INFO:teuthology.orchestra.run.smithi003.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 145 kB/s | 43 kB 00:00 2024-03-21T13:55:00.003 INFO:teuthology.orchestra.run.smithi003.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-21T13:55:00.070 INFO:teuthology.orchestra.run.smithi003.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 458 kB/s | 172 kB 00:00 2024-03-21T13:55:00.087 INFO:teuthology.orchestra.run.smithi003.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-21T13:55:00.163 INFO:teuthology.orchestra.run.smithi003.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-21T13:55:00.179 INFO:teuthology.orchestra.run.smithi003.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-03-21T13:55:00.230 INFO:teuthology.orchestra.run.smithi003.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-21T13:55:00.255 INFO:teuthology.orchestra.run.smithi003.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-21T13:55:00.272 INFO:teuthology.orchestra.run.smithi003.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-21T13:55:00.322 INFO:teuthology.orchestra.run.smithi003.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-21T13:55:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 784 kB/s | 79 kB 00:00 2024-03-21T13:55:00.406 INFO:teuthology.orchestra.run.smithi003.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 196 kB/s | 16 kB 00:00 2024-03-21T13:55:00.448 INFO:teuthology.orchestra.run.smithi003.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-03-21T13:55:00.473 INFO:teuthology.orchestra.run.smithi003.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 769 kB/s | 90 kB 00:00 2024-03-21T13:55:00.490 INFO:teuthology.orchestra.run.smithi003.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 426 kB/s | 36 kB 00:00 2024-03-21T13:55:00.549 INFO:teuthology.orchestra.run.smithi003.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-21T13:55:00.590 INFO:teuthology.orchestra.run.smithi003.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-21T13:55:00.649 INFO:teuthology.orchestra.run.smithi003.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-21T13:55:00.652 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:00.653 INFO:teuthology.orchestra.run.smithi003.stdout:Total 8.7 MB/s | 30 MB 00:03 2024-03-21T13:55:00.677 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:00.722 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:00.722 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:01.142 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:01.142 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:01.203 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:01.306 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-21T13:55:01.344 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-21T13:55:01.375 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-21T13:55:01.438 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-21T13:55:01.491 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-21T13:55:01.523 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-21T13:55:01.616 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-21T13:55:01.744 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-21T13:55:01.785 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-21T13:55:01.856 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-21T13:55:01.910 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-21T13:55:01.949 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-21T13:55:01.998 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-21T13:55:02.053 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-21T13:55:02.283 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-21T13:55:02.333 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-21T13:55:02.368 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-21T13:55:02.401 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-21T13:55:02.432 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-21T13:55:02.508 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-21T13:55:02.550 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:55:02.626 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-21T13:55:02.641 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:02.666 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-21T13:55:02.701 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-21T13:55:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-21T13:55:02.735 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-21T13:55:02.751 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-21T13:55:02.779 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-21T13:55:02.825 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-21T13:55:02.854 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-21T13:55:02.883 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-21T13:55:02.904 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-21T13:55:02.936 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-21T13:55:03.004 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-21T13:55:03.033 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-21T13:55:03.110 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-21T13:55:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-21T13:55:03.164 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-21T13:55:03.239 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-21T13:55:03.311 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:55:03.416 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-21T13:55:03.453 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-21T13:55:03.492 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2309.gba760091.el9. 31/36 2024-03-21T13:55:03.512 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-21T13:55:03.522 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 32/36 2024-03-21T13:55:03.547 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 32/36 2024-03-21T13:55:03.547 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:03.547 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T13:55:03.547 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:55:03.547 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:55:03.548 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:03.576 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-21T13:55:03.788 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-21T13:55:03.840 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-21T13:55:03.879 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-21T13:55:03.914 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-21T13:55:03.951 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-21T13:55:04.031 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-21T13:55:04.195 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:55:04.228 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-21T13:55:04.269 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-21T13:55:04.312 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-21T13:55:04.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-21T13:55:04.442 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-21T13:55:04.477 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-21T13:55:04.552 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-21T13:55:04.702 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-21T13:55:04.834 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:55:04.989 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 33/36 2024-03-21T13:55:05.016 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 33/36 2024-03-21T13:55:05.016 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:05.016 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T13:55:05.016 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:55:05.016 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:55:05.016 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:05.137 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2309.gba760091.el9. 31/36 2024-03-21T13:55:05.165 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 32/36 2024-03-21T13:55:05.190 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 32/36 2024-03-21T13:55:05.190 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:05.190 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-21T13:55:05.190 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:55:05.190 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-21T13:55:05.190 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:05.515 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 34/36 2024-03-21T13:55:05.539 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 34/36 2024-03-21T13:55:05.539 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:05.539 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T13:55:05.539 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:55:05.539 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:55:05.539 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:05.766 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 35/36 2024-03-21T13:55:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 35/36 2024-03-21T13:55:05.789 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:05.789 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T13:55:05.789 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:55:05.789 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:55:05.789 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:05.848 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.0.0-2309.gba760091.el9.x86_64 36/36 2024-03-21T13:55:06.699 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 33/36 2024-03-21T13:55:06.725 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 33/36 2024-03-21T13:55:06.725 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:06.725 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-21T13:55:06.725 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:55:06.725 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-21T13:55:06.726 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:07.186 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 34/36 2024-03-21T13:55:07.206 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 34/36 2024-03-21T13:55:07.207 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:07.207 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-21T13:55:07.207 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:55:07.207 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-21T13:55:07.207 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:07.433 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 35/36 2024-03-21T13:55:07.457 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 35/36 2024-03-21T13:55:07.457 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:07.457 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-21T13:55:07.457 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:55:07.457 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-21T13:55:07.457 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:07.526 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:19.0.0-2309.gba760091.el9.x86_64 36/36 2024-03-21T13:55:07.628 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-2:19.0.0-2309.gba760091.el9.x86_64 36/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.0.0-2309.gba760091.el9.x86_64 1/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 2/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 3/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 4/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 5/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2309.gba760091.el9. 6/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-21T13:55:07.629 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:55:07.630 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-21T13:55:07.631 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-21T13:55:08.716 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T13:55:08.717 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T13:55:08.718 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:08.719 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:09.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-base 2024-03-21T13:55:09.215 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-2:19.0.0-2309.gba760091.el9.x86_64 36/36 2024-03-21T13:55:09.215 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:19.0.0-2309.gba760091.el9.x86_64 1/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 2/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 3/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 4/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 5/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2309.gba760091.el9. 6/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-21T13:55:09.216 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-21T13:55:09.217 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-21T13:55:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-21T13:55:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-21T13:55:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-21T13:55:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-21T13:55:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-21T13:55:09.218 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-21T13:55:09.661 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:00 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:09.758 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:09.811 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:09.812 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:55:09.812 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:09.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install cephadm 2024-03-21T13:55:10.147 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-21T13:55:10.147 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:10.147 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:10.147 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:10.147 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: lua-5.4.4-4.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-21T13:55:10.148 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-21T13:55:10.149 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:10.150 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:10.437 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:01 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:10.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-base 2024-03-21T13:55:10.586 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 762 k 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 762 k 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 768 k 2024-03-21T13:55:10.587 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:11.039 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:58 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:11.072 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm-19.0.0-2309.gba760091.el9.noarch.rpm 1.5 MB/s | 762 kB 00:00 2024-03-21T13:55:11.072 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:11.072 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.5 MB/s | 762 kB 00:00 2024-03-21T13:55:11.073 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:11.078 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:11.078 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:11.083 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:11.084 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:11.138 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-base-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:11.156 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:11.192 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:11.193 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:55:11.193 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:11.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install cephadm 2024-03-21T13:55:11.546 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:11.634 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:11.817 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:00:58 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:11.969 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 762 k 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:11.970 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:55:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 762 k 2024-03-21T13:55:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 768 k 2024-03-21T13:55:11.971 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:12.025 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm-19.0.0-2309.gba760091.el9.noarch.rpm 2.7 MB/s | 762 kB 00:00 2024-03-21T13:55:12.247 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout:Total 2.7 MB/s | 762 kB 00:00 2024-03-21T13:55:12.248 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:12.253 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:12.322 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:12.322 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:12.322 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:12.322 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:12.322 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:12.322 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:12.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-immutable-object-cache 2024-03-21T13:55:12.774 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:12.862 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:13.087 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:04 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:13.237 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:13.237 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:13.237 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-21T13:55:13.237 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:13.237 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:13.237 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2309.gba760091.el9 ceph 145 k 2024-03-21T13:55:13.238 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:13.238 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:13.238 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:13.238 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:55:13.238 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:13.238 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 145 k 2024-03-21T13:55:13.239 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 436 k 2024-03-21T13:55:13.239 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:13.262 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:13.462 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-immutable-object-cache-19.0.0-2309.gba7600 651 kB/s | 145 kB 00:00 2024-03-21T13:55:13.462 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:13.463 INFO:teuthology.orchestra.run.smithi118.stdout:Total 647 kB/s | 145 kB 00:00 2024-03-21T13:55:13.463 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:13.471 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:13.471 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:13.522 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:13.522 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/1 2024-03-21T13:55:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:13.534 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:13.534 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:13.534 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:13.663 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:13.738 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2309.gba760091. 1/1 2024-03-21T13:55:13.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-immutable-object-cache 2024-03-21T13:55:13.762 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2309.gba760091. 1/1 2024-03-21T13:55:13.762 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:13.763 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T13:55:13.763 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:14.332 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:01 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:14.488 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2309.gba760091.el9 ceph 145 k 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:14.489 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:14.490 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:55:14.490 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:14.490 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 145 k 2024-03-21T13:55:14.490 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 436 k 2024-03-21T13:55:14.490 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:14.711 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2309.gba760091. 1/1 2024-03-21T13:55:14.711 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:14.712 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:14.712 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:14.712 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:14.712 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:14.719 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-immutable-object-cache-19.0.0-2309.gba7600 634 kB/s | 145 kB 00:00 2024-03-21T13:55:14.719 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:14.720 INFO:teuthology.orchestra.run.smithi003.stdout:Total 630 kB/s | 145 kB 00:00 2024-03-21T13:55:14.720 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:14.728 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:14.728 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:14.781 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:14.782 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:14.921 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:14.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr 2024-03-21T13:55:14.979 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2309.gba760091. 1/1 2024-03-21T13:55:15.002 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2309.gba760091. 1/1 2024-03-21T13:55:15.002 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:15.003 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-21T13:55:15.003 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:15.522 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:06 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:15.618 INFO:teuthology.orchestra.run.smithi118.stdout:Package ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:15.671 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:15.671 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:55:15.671 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:15.765 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-dashboard 2024-03-21T13:55:16.048 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2309.gba760091. 1/1 2024-03-21T13:55:16.048 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:16.048 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:16.048 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:16.048 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:16.048 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:16.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr 2024-03-21T13:55:16.325 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:07 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:16.474 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:16.475 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-03-21T13:55:16.475 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:16.475 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-03-21T13:55:16.475 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:16.475 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 3.5 M 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 34 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 16 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:16.476 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:16.477 INFO:teuthology.orchestra.run.smithi118.stdout:====================================================================================== 2024-03-21T13:55:16.477 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-03-21T13:55:16.477 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:16.477 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 8.6 M 2024-03-21T13:55:16.477 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 103 M 2024-03-21T13:55:16.477 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:16.817 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2309.gba 165 kB/s | 34 kB 00:00 2024-03-21T13:55:16.842 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2309.gba7 71 kB/s | 16 kB 00:00 2024-03-21T13:55:16.856 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:03 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:16.955 INFO:teuthology.orchestra.run.smithi003.stdout:Package ceph-mgr-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:16.992 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 238 kB 00:00 2024-03-21T13:55:17.009 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:17.010 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:55:17.010 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:17.034 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-21T13:55:17.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-dashboard 2024-03-21T13:55:17.234 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2309.gba76009 5.7 MB/s | 3.5 MB 00:00 2024-03-21T13:55:17.406 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-03-21T13:55:17.435 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 97 kB/s | 19 kB 00:00 2024-03-21T13:55:17.460 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-21T13:55:17.527 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 460 kB/s | 31 kB 00:00 2024-03-21T13:55:17.594 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-21T13:55:17.641 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:04 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:17.644 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 691 kB/s | 144 kB 00:00 2024-03-21T13:55:17.798 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:17.799 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-21T13:55:17.799 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:17.799 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-21T13:55:17.799 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 3.5 M 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 34 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 16 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-21T13:55:17.800 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout:====================================================================================== 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 8.6 M 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 103 M 2024-03-21T13:55:17.801 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:17.903 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.1 MB/s | 2.0 MB 00:00 2024-03-21T13:55:17.907 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:17.907 INFO:teuthology.orchestra.run.smithi118.stdout:Total 6.0 MB/s | 8.6 MB 00:01 2024-03-21T13:55:17.925 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:17.942 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:17.942 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:18.149 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:18.150 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:18.272 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2309.gba7 88 kB/s | 16 kB 00:00 2024-03-21T13:55:18.289 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2309.gba 167 kB/s | 34 kB 00:00 2024-03-21T13:55:18.472 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-21T13:55:18.523 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.0 MB/s | 1.0 MB 00:00 2024-03-21T13:55:18.639 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): ceph-mgr-dashboard-19.0.0-2309.gba76009 6.4 MB/s | 3.5 MB 00:00 2024-03-21T13:55:18.681 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:18.765 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 155 kB/s | 19 kB 00:00 2024-03-21T13:55:18.779 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-21T13:55:18.822 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-21T13:55:18.832 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-21T13:55:18.870 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-21T13:55:18.882 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.1 MB/s | 863 kB 00:00 2024-03-21T13:55:18.899 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-21T13:55:18.907 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-03-21T13:55:18.974 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-21T13:55:18.995 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-21T13:55:19.016 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 230 kB/s | 31 kB 00:00 2024-03-21T13:55:19.108 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.9 MB/s | 2.0 MB 00:00 2024-03-21T13:55:19.112 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:19.112 INFO:teuthology.orchestra.run.smithi003.stdout:Total 6.6 MB/s | 8.6 MB 00:01 2024-03-21T13:55:19.117 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-21T13:55:19.130 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:19.148 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:19.148 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:19.203 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-21T13:55:19.296 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-21T13:55:19.324 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:55:19.336 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:19.336 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:19.357 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2309.gba760091.el9 10/12 2024-03-21T13:55:19.977 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:20.059 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-21T13:55:20.103 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-21T13:55:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-21T13:55:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-21T13:55:20.278 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-21T13:55:20.404 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2309.gba760091.el 11/12 2024-03-21T13:55:20.409 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-21T13:55:20.449 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noa 12/12 2024-03-21T13:55:20.503 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-21T13:55:20.550 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-21T13:55:20.579 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:55:20.611 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2309.gba760091.el9 10/12 2024-03-21T13:55:21.711 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2309.gba760091.el 11/12 2024-03-21T13:55:21.753 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noa 12/12 2024-03-21T13:55:22.094 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noa 12/12 2024-03-21T13:55:22.094 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2309.gba760091.el 1/12 2024-03-21T13:55:22.094 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noa 2/12 2024-03-21T13:55:22.094 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2309.gba760091.el9 3/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-21T13:55:22.095 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-21T13:55:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-21T13:55:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:22.455 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T13:55:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:22.456 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:22.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-21T13:55:23.131 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noa 12/12 2024-03-21T13:55:23.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2309.gba760091.el 1/12 2024-03-21T13:55:23.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noa 2/12 2024-03-21T13:55:23.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2309.gba760091.el9 3/12 2024-03-21T13:55:23.131 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-21T13:55:23.132 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-21T13:55:23.318 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:14 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:23.478 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:23.479 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-21T13:55:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:23.479 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-21T13:55:23.479 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 7.4 M 2024-03-21T13:55:23.479 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:23.480 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-21T13:55:23.481 INFO:teuthology.orchestra.run.smithi118.stdout:Install 12 Packages 2024-03-21T13:55:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:23.481 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 37 M 2024-03-21T13:55:23.481 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 209 M 2024-03-21T13:55:23.481 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-21T13:55:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:23.484 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-21T13:55:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:23.484 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-21T13:55:23.485 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-21T13:55:23.486 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:23.486 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:23.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-21T13:55:23.971 INFO:teuthology.orchestra.run.smithi118.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 172 kB/s | 32 kB 00:00 2024-03-21T13:55:24.004 INFO:teuthology.orchestra.run.smithi118.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2024-03-21T13:55:24.038 INFO:teuthology.orchestra.run.smithi118.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-21T13:55:24.088 INFO:teuthology.orchestra.run.smithi118.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-03-21T13:55:24.320 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:11 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:24.422 INFO:teuthology.orchestra.run.smithi118.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-03-21T13:55:24.481 INFO:teuthology.orchestra.run.smithi118.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-21T13:55:24.484 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 7.4 M 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-21T13:55:24.485 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout:Install 12 Packages 2024-03-21T13:55:24.486 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:24.487 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 37 M 2024-03-21T13:55:24.487 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 209 M 2024-03-21T13:55:24.487 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:24.497 INFO:teuthology.orchestra.run.smithi118.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 217 kB 00:00 2024-03-21T13:55:24.765 INFO:teuthology.orchestra.run.smithi118.stdout:(8/12): ceph-mgr-diskprediction-local-19.0.0-23 7.6 MB/s | 7.4 MB 00:00 2024-03-21T13:55:25.049 INFO:teuthology.orchestra.run.smithi003.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 220 kB/s | 32 kB 00:00 2024-03-21T13:55:25.074 INFO:teuthology.orchestra.run.smithi003.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 727 kB/s | 17 kB 00:00 2024-03-21T13:55:25.107 INFO:teuthology.orchestra.run.smithi003.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-21T13:55:25.166 INFO:teuthology.orchestra.run.smithi003.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-21T13:55:25.384 INFO:teuthology.orchestra.run.smithi003.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2024-03-21T13:55:25.415 INFO:teuthology.orchestra.run.smithi118.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 17 MB/s | 16 MB 00:00 2024-03-21T13:55:25.448 INFO:teuthology.orchestra.run.smithi118.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 845 kB/s | 803 kB 00:00 2024-03-21T13:55:25.450 INFO:teuthology.orchestra.run.smithi003.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-21T13:55:25.475 INFO:teuthology.orchestra.run.smithi003.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-21T13:55:25.532 INFO:teuthology.orchestra.run.smithi118.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 255 kB/s | 195 kB 00:00 2024-03-21T13:55:25.818 INFO:teuthology.orchestra.run.smithi003.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-21T13:55:25.901 INFO:teuthology.orchestra.run.smithi003.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-23 7.4 MB/s | 7.4 MB 00:00 2024-03-21T13:55:25.957 INFO:teuthology.orchestra.run.smithi118.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 453 kB/s | 245 kB 00:00 2024-03-21T13:55:25.959 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:25.960 INFO:teuthology.orchestra.run.smithi118.stdout:Total 15 MB/s | 37 MB 00:02 2024-03-21T13:55:26.005 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:26.031 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:26.405 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:26.406 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:26.726 INFO:teuthology.orchestra.run.smithi003.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 215 kB/s | 195 kB 00:00 2024-03-21T13:55:26.869 INFO:teuthology.orchestra.run.smithi003.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 254 kB/s | 245 kB 00:00 2024-03-21T13:55:27.132 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:27.196 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-21T13:55:27.228 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-21T13:55:27.321 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:55:27.356 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-21T13:55:27.360 INFO:teuthology.orchestra.run.smithi003.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 427 kB/s | 803 kB 00:01 2024-03-21T13:55:27.363 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:27.363 INFO:teuthology.orchestra.run.smithi003.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-21T13:55:27.407 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:27.433 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:27.433 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:27.532 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:55:27.564 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:55:27.649 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-21T13:55:27.817 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:27.818 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:28.291 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-21T13:55:28.418 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-21T13:55:28.582 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:28.650 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-21T13:55:28.681 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-21T13:55:28.782 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:55:28.826 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-21T13:55:29.012 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:55:29.041 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:55:29.130 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-21T13:55:29.693 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-21T13:55:29.818 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-21T13:55:29.925 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-21T13:55:30.803 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-21T13:55:30.840 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760 12/12 2024-03-21T13:55:31.246 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-21T13:55:32.337 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-21T13:55:32.364 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760 12/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760 12/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760 1/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:55:33.980 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:55:33.981 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-21T13:55:33.981 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-21T13:55:33.981 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-21T13:55:33.981 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-21T13:55:33.981 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-21T13:55:34.402 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T13:55:34.403 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T13:55:34.404 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:34.404 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:34.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-rook 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760 12/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760 1/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-21T13:55:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-21T13:55:34.842 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-21T13:55:34.842 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-21T13:55:34.842 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-21T13:55:35.348 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-21T13:55:35.349 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-21T13:55:35.350 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-21T13:55:35.350 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:35.350 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:35.365 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:26 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:35.517 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:35.517 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-21T13:55:35.517 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:35.517 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 49 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T13:55:35.518 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T13:55:35.519 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T13:55:35.519 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:35.519 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:35.519 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-21T13:55:35.519 INFO:teuthology.orchestra.run.smithi118.stdout:Install 13 Packages 2024-03-21T13:55:35.519 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:35.520 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 2.0 M 2024-03-21T13:55:35.520 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 26 M 2024-03-21T13:55:35.520 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:35.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-rook 2024-03-21T13:55:35.783 INFO:teuthology.orchestra.run.smithi118.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 131 kB/s | 24 kB 00:00 2024-03-21T13:55:35.808 INFO:teuthology.orchestra.run.smithi118.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 86 kB/s | 18 kB 00:00 2024-03-21T13:55:35.967 INFO:teuthology.orchestra.run.smithi118.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 955 kB/s | 174 kB 00:00 2024-03-21T13:55:36.026 INFO:teuthology.orchestra.run.smithi118.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 609 kB/s | 132 kB 00:00 2024-03-21T13:55:36.059 INFO:teuthology.orchestra.run.smithi118.stdout:(5/13): ceph-mgr-rook-19.0.0-2309.gba760091.el9 107 kB/s | 49 kB 00:00 2024-03-21T13:55:36.093 INFO:teuthology.orchestra.run.smithi118.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.6 MB/s | 211 kB 00:00 2024-03-21T13:55:36.118 INFO:teuthology.orchestra.run.smithi118.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 463 kB/s | 43 kB 00:00 2024-03-21T13:55:36.298 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:23 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:36.454 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 49 k 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-21T13:55:36.455 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:36.456 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-21T13:55:36.457 INFO:teuthology.orchestra.run.smithi003.stdout:Install 13 Packages 2024-03-21T13:55:36.457 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:36.457 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 2.0 M 2024-03-21T13:55:36.457 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 26 M 2024-03-21T13:55:36.457 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:36.552 INFO:teuthology.orchestra.run.smithi118.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-03-21T13:55:36.585 INFO:teuthology.orchestra.run.smithi118.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 30 kB/s | 15 kB 00:00 2024-03-21T13:55:36.839 INFO:teuthology.orchestra.run.smithi003.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 151 kB/s | 24 kB 00:00 2024-03-21T13:55:36.864 INFO:teuthology.orchestra.run.smithi003.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 97 kB/s | 18 kB 00:00 2024-03-21T13:55:36.877 INFO:teuthology.orchestra.run.smithi118.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 267 kB/s | 203 kB 00:00 2024-03-21T13:55:36.889 INFO:teuthology.orchestra.run.smithi003.stdout:(3/13): ceph-mgr-rook-19.0.0-2309.gba760091.el9 233 kB/s | 49 kB 00:00 2024-03-21T13:55:36.902 INFO:teuthology.orchestra.run.smithi118.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 186 kB/s | 59 kB 00:00 2024-03-21T13:55:36.973 INFO:teuthology.orchestra.run.smithi003.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 174 kB 00:00 2024-03-21T13:55:36.998 INFO:teuthology.orchestra.run.smithi003.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 990 kB/s | 132 kB 00:00 2024-03-21T13:55:37.023 INFO:teuthology.orchestra.run.smithi003.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 211 kB 00:00 2024-03-21T13:55:37.040 INFO:teuthology.orchestra.run.smithi003.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 639 kB/s | 43 kB 00:00 2024-03-21T13:55:37.111 INFO:teuthology.orchestra.run.smithi118.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 384 kB/s | 90 kB 00:00 2024-03-21T13:55:37.161 INFO:teuthology.orchestra.run.smithi118.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-03-21T13:55:37.166 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:37.166 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-21T13:55:37.166 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:37.181 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:37.181 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:37.274 INFO:teuthology.orchestra.run.smithi003.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 117 kB/s | 32 kB 00:00 2024-03-21T13:55:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:37.275 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:37.299 INFO:teuthology.orchestra.run.smithi003.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-21T13:55:37.433 INFO:teuthology.orchestra.run.smithi003.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 517 kB/s | 203 kB 00:00 2024-03-21T13:55:37.466 INFO:teuthology.orchestra.run.smithi003.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-21T13:55:37.533 INFO:teuthology.orchestra.run.smithi003.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 896 kB/s | 90 kB 00:00 2024-03-21T13:55:37.708 INFO:teuthology.orchestra.run.smithi003.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-21T13:55:37.710 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:37.714 INFO:teuthology.orchestra.run.smithi003.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-21T13:55:37.714 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:37.726 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:37.726 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:37.803 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:37.824 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:37.825 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:37.993 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-21T13:55:38.063 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-21T13:55:38.118 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-21T13:55:38.156 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-21T13:55:38.197 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-21T13:55:38.290 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-21T13:55:38.338 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:38.393 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-21T13:55:38.440 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-21T13:55:38.512 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-21T13:55:38.569 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-21T13:55:38.633 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-21T13:55:38.669 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-21T13:55:38.711 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-21T13:55:38.807 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-21T13:55:38.907 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-21T13:55:38.962 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-21T13:55:39.156 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-21T13:55:39.221 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-21T13:55:39.255 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-21T13:55:39.303 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-21T13:55:39.325 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 13/13 2024-03-21T13:55:39.725 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-21T13:55:39.767 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-21T13:55:39.825 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-21T13:55:39.902 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-21T13:55:39.928 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 13/13 2024-03-21T13:55:40.171 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 13/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 1/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-21T13:55:40.172 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T13:55:40.545 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:40.546 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:40.681 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 13/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 1/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-21T13:55:40.682 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-21T13:55:40.683 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-21T13:55:40.683 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-21T13:55:40.683 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-21T13:55:40.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-mgr-cephadm 2024-03-21T13:55:41.038 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-21T13:55:41.038 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:41.038 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:41.038 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:41.038 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:41.039 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:41.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-mgr-cephadm 2024-03-21T13:55:41.365 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:32 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:41.515 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:41.515 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 141 k 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T13:55:41.516 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T13:55:41.517 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T13:55:41.517 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:41.517 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:41.517 INFO:teuthology.orchestra.run.smithi118.stdout:======================================================================================== 2024-03-21T13:55:41.517 INFO:teuthology.orchestra.run.smithi118.stdout:Install 6 Packages 2024-03-21T13:55:41.517 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:41.518 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 6.8 M 2024-03-21T13:55:41.518 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 33 M 2024-03-21T13:55:41.518 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:41.865 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:28 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:41.882 INFO:teuthology.orchestra.run.smithi118.stdout:(1/6): ceph-mgr-cephadm-19.0.0-2309.gba760091.e 571 kB/s | 141 kB 00:00 2024-03-21T13:55:42.022 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:42.026 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-21T13:55:42.026 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:42.026 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-21T13:55:42.026 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 141 k 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:======================================================================================== 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:Install 6 Packages 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 6.8 M 2024-03-21T13:55:42.027 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 33 M 2024-03-21T13:55:42.028 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:42.066 INFO:teuthology.orchestra.run.smithi118.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-21T13:55:42.217 INFO:teuthology.orchestra.run.smithi118.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-21T13:55:42.242 INFO:teuthology.orchestra.run.smithi118.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 331 kB/s | 58 kB 00:00 2024-03-21T13:55:42.267 INFO:teuthology.orchestra.run.smithi118.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-03-21T13:55:42.645 INFO:teuthology.orchestra.run.smithi003.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-21T13:55:42.709 INFO:teuthology.orchestra.run.smithi118.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 231 kB/s | 248 kB 00:01 2024-03-21T13:55:42.711 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:42.712 INFO:teuthology.orchestra.run.smithi118.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-21T13:55:42.725 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:42.736 INFO:teuthology.orchestra.run.smithi003.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-21T13:55:42.737 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:42.737 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:42.778 INFO:teuthology.orchestra.run.smithi003.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2309.gba760091.e 428 kB/s | 141 kB 00:00 2024-03-21T13:55:42.827 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:42.829 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:43.127 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:43.187 INFO:teuthology.orchestra.run.smithi003.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 135 kB/s | 58 kB 00:00 2024-03-21T13:55:43.212 INFO:teuthology.orchestra.run.smithi003.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 118 kB/s | 51 kB 00:00 2024-03-21T13:55:43.300 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-21T13:55:43.376 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-21T13:55:43.404 INFO:teuthology.orchestra.run.smithi003.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 724 kB/s | 548 kB 00:00 2024-03-21T13:55:43.409 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:43.409 INFO:teuthology.orchestra.run.smithi003.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-21T13:55:43.424 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:43.436 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:43.436 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:43.532 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:43.532 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:43.833 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:43.893 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-21T13:55:43.982 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-21T13:55:44.016 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-21T13:55:44.045 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-21T13:55:44.065 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-21T13:55:44.075 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 6/6 2024-03-21T13:55:44.617 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-21T13:55:44.701 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-21T13:55:44.763 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-21T13:55:44.792 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 6/6 2024-03-21T13:55:44.866 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 6/6 2024-03-21T13:55:44.866 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/6 2024-03-21T13:55:44.866 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-21T13:55:44.866 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-21T13:55:44.866 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-21T13:55:44.866 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-21T13:55:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-21T13:55:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:45.184 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T13:55:45.184 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T13:55:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T13:55:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T13:55:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T13:55:45.185 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:45.185 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:45.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-fuse 2024-03-21T13:55:45.546 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 6/6 2024-03-21T13:55:45.547 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 1/6 2024-03-21T13:55:45.547 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-21T13:55:45.547 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-21T13:55:45.547 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-21T13:55:45.547 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-21T13:55:45.843 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:45.844 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:46.031 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:37 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:46.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-fuse 2024-03-21T13:55:46.182 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:46.182 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.0.0-2309.gba760091.el9 ceph 890 k 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-03-21T13:55:46.183 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:46.184 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 969 k 2024-03-21T13:55:46.184 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 2.8 M 2024-03-21T13:55:46.184 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:46.396 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 372 kB/s | 79 kB 00:00 2024-03-21T13:55:46.636 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:33 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:46.792 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:19.0.0-2309.gba760091.el9 ceph 890 k 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:46.793 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-21T13:55:46.794 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:46.794 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 969 k 2024-03-21T13:55:46.794 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 2.8 M 2024-03-21T13:55:46.794 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:46.963 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 466 kB/s | 79 kB 00:00 2024-03-21T13:55:46.971 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-fuse-19.0.0-2309.gba760091.el9.x86_ 1.1 MB/s | 890 kB 00:00 2024-03-21T13:55:46.972 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:46.972 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.2 MB/s | 969 kB 00:00 2024-03-21T13:55:46.973 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:46.992 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:46.992 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:47.065 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:47.065 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:47.105 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-fuse-19.0.0-2309.gba760091.el9.x86_ 2.8 MB/s | 890 kB 00:00 2024-03-21T13:55:47.106 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:47.107 INFO:teuthology.orchestra.run.smithi003.stdout:Total 3.0 MB/s | 969 kB 00:00 2024-03-21T13:55:47.107 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:47.128 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:47.128 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:47.203 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:47.203 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:47.250 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-21T13:55:47.462 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 2/2 2024-03-21T13:55:47.500 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:47.636 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-21T13:55:47.708 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 2/2 2024-03-21T13:55:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 2/2 2024-03-21T13:55:48.171 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/2 2024-03-21T13:55:48.471 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-21T13:55:48.471 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:48.471 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:48.472 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-21T13:55:48.472 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:48.472 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:48.534 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 2/2 2024-03-21T13:55:48.534 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/2 2024-03-21T13:55:48.749 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-volume 2024-03-21T13:55:48.820 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-21T13:55:48.820 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:48.820 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:48.820 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:19.0.0-2309.gba760091.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-21T13:55:48.820 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:48.820 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:49.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-volume 2024-03-21T13:55:49.324 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:40 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:49.482 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:49.483 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:49.483 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-21T13:55:49.483 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:49.484 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:49.484 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 264 k 2024-03-21T13:55:49.484 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-21T13:55:49.484 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T13:55:49.484 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:49.484 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:49.485 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:49.485 INFO:teuthology.orchestra.run.smithi118.stdout:Install 2 Packages 2024-03-21T13:55:49.485 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:49.485 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 332 k 2024-03-21T13:55:49.485 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 1.5 M 2024-03-21T13:55:49.485 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:49.627 INFO:teuthology.orchestra.run.smithi118.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 480 kB/s | 69 kB 00:00 2024-03-21T13:55:49.665 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:36 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:49.760 INFO:teuthology.orchestra.run.smithi118.stdout:(2/2): ceph-volume-19.0.0-2309.gba760091.el9.no 953 kB/s | 264 kB 00:00 2024-03-21T13:55:49.761 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:49.762 INFO:teuthology.orchestra.run.smithi118.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-21T13:55:49.762 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:49.770 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:49.770 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:49.806 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:49.807 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:49.822 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:49.823 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:49.825 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-03-21T13:55:49.825 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:49.825 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:49.825 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:19.0.0-2309.gba760091.el9 ceph-noarch 264 k 2024-03-21T13:55:49.825 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-03-21T13:55:49.825 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout:Install 2 Packages 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 332 k 2024-03-21T13:55:49.826 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 1.5 M 2024-03-21T13:55:49.827 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:49.978 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:49.984 INFO:teuthology.orchestra.run.smithi003.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 430 kB/s | 69 kB 00:00 2024-03-21T13:55:50.150 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-21T13:55:50.196 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 2/2 2024-03-21T13:55:50.212 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 2/2 2024-03-21T13:55:50.212 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:50.213 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T13:55:50.213 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:50.235 INFO:teuthology.orchestra.run.smithi003.stdout:(2/2): ceph-volume-19.0.0-2309.gba760091.el9.no 643 kB/s | 264 kB 00:00 2024-03-21T13:55:50.236 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:50.236 INFO:teuthology.orchestra.run.smithi003.stdout:Total 806 kB/s | 332 kB 00:00 2024-03-21T13:55:50.236 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:50.245 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:50.245 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:50.284 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:50.284 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:50.434 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-21T13:55:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 2/2 2024-03-21T13:55:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 2/2 2024-03-21T13:55:50.661 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:55:50.661 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-21T13:55:50.661 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:50.935 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 1/2 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:51.277 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:51.380 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 1/2 2024-03-21T13:55:51.643 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados-devel 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:19.0.0-2309.gba760091.el9.noarch 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:51.723 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:52.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados-devel 2024-03-21T13:55:52.219 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:43 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:52.369 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:52.370 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:52.370 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-21T13:55:52.370 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:52.370 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:52.370 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.0.0-2309.gba760091.el9 ceph 127 k 2024-03-21T13:55:52.370 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 127 k 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 456 k 2024-03-21T13:55:52.371 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:52.618 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:39 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:52.726 INFO:teuthology.orchestra.run.smithi118.stdout:librados-devel-19.0.0-2309.gba760091.el9.x86_64 358 kB/s | 127 kB 00:00 2024-03-21T13:55:52.726 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:52.726 INFO:teuthology.orchestra.run.smithi118.stdout:Total 356 kB/s | 127 kB 00:00 2024-03-21T13:55:52.727 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:52.732 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:52.732 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:52.775 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:19.0.0-2309.gba760091.el9 ceph 127 k 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:52.776 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 127 k 2024-03-21T13:55:52.777 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 456 k 2024-03-21T13:55:52.777 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:52.788 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:52.788 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:52.948 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:53.065 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:53.164 INFO:teuthology.orchestra.run.smithi003.stdout:librados-devel-19.0.0-2309.gba760091.el9.x86_64 327 kB/s | 127 kB 00:00 2024-03-21T13:55:53.165 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:53.165 INFO:teuthology.orchestra.run.smithi003.stdout:Total 326 kB/s | 127 kB 00:00 2024-03-21T13:55:53.165 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:53.171 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:53.171 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:53.230 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:53.231 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:53.380 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:53.490 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:53.524 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:53.802 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:53.802 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:53.803 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:53.803 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:53.942 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:54.063 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs2 2024-03-21T13:55:54.241 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:54.242 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:54.242 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:54.242 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:54.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs2 2024-03-21T13:55:54.655 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:54.756 INFO:teuthology.orchestra.run.smithi118.stdout:Package libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:54.811 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:54.811 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:55:54.812 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:54.887 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install libcephfs-devel 2024-03-21T13:55:55.021 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:55.126 INFO:teuthology.orchestra.run.smithi003.stdout:Package libcephfs2-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:55.183 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:55.184 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:55:55.184 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:55.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install libcephfs-devel 2024-03-21T13:55:55.448 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:46 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:55.598 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.0.0-2309.gba760091.el9 ceph 31 k 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:55:55.599 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:55.600 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 31 k 2024-03-21T13:55:55.600 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 140 k 2024-03-21T13:55:55.600 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:55:55.837 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:42 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:55.922 INFO:teuthology.orchestra.run.smithi118.stdout:libcephfs-devel-19.0.0-2309.gba760091.el9.x86_6 97 kB/s | 31 kB 00:00 2024-03-21T13:55:55.922 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:55.923 INFO:teuthology.orchestra.run.smithi118.stdout:Total 97 kB/s | 31 kB 00:00 2024-03-21T13:55:55.923 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:55:55.926 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:55:55.926 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:55:55.951 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:55:55.951 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:55:55.995 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:55.995 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:55.995 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:19.0.0-2309.gba760091.el9 ceph 31 k 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 31 k 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 140 k 2024-03-21T13:55:55.996 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:55:56.076 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:55:56.178 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:56.359 INFO:teuthology.orchestra.run.smithi003.stdout:libcephfs-devel-19.0.0-2309.gba760091.el9.x86_6 86 kB/s | 31 kB 00:00 2024-03-21T13:55:56.359 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:55:56.360 INFO:teuthology.orchestra.run.smithi003.stdout:Total 86 kB/s | 31 kB 00:00 2024-03-21T13:55:56.360 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:55:56.364 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:55:56.364 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:55:56.393 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:55:56.394 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:55:56.513 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:55:56.568 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:56.848 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:56.849 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:56.849 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:55:56.849 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:56.849 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:55:56.849 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:57.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librados2 2024-03-21T13:55:57.016 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:57.317 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:55:57.317 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:57.317 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:55:57.317 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:55:57.317 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:55:57.318 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:57.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librados2 2024-03-21T13:55:57.560 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:57.657 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:57.709 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:57.709 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:55:57.710 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:57.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install librbd1 2024-03-21T13:55:58.035 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:58.136 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:58.190 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:58.191 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:55:58.191 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:58.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install librbd1 2024-03-21T13:55:58.319 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:49 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:58.416 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:58.467 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:58.468 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:55:58.468 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:58.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rados 2024-03-21T13:55:58.810 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:45 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:58.910 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:58.965 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:58.966 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:55:58.966 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:59.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rados 2024-03-21T13:55:59.094 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:50 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:59.191 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:59.243 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:55:59.244 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:55:59.244 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:55:59.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rgw 2024-03-21T13:55:59.594 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:46 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:55:59.699 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rados-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:55:59.751 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:55:59.752 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:55:59.752 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:55:59.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rgw 2024-03-21T13:55:59.858 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:50 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:55:59.953 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:56:00.004 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:56:00.005 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:56:00.005 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:56:00.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-cephfs 2024-03-21T13:56:00.375 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:47 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:56:00.477 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rgw-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:56:00.531 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:56:00.532 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:56:00.532 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:56:00.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-cephfs 2024-03-21T13:56:00.616 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:51 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:56:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:56:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:56:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:56:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:56:00.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install python3-rbd 2024-03-21T13:56:01.153 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:56:01.254 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-cephfs-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:56:01.309 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:56:01.310 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:56:01.310 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:56:01.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install python3-rbd 2024-03-21T13:56:01.396 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:52 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:56:01.493 INFO:teuthology.orchestra.run.smithi118.stdout:Package python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:56:01.544 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:56:01.545 INFO:teuthology.orchestra.run.smithi118.stdout:Nothing to do. 2024-03-21T13:56:01.545 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:56:01.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-fuse 2024-03-21T13:56:01.935 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:48 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:56:02.035 INFO:teuthology.orchestra.run.smithi003.stdout:Package python3-rbd-2:19.0.0-2309.gba760091.el9.x86_64 is already installed. 2024-03-21T13:56:02.090 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:56:02.091 INFO:teuthology.orchestra.run.smithi003.stdout:Nothing to do. 2024-03-21T13:56:02.091 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:56:02.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-fuse 2024-03-21T13:56:02.165 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:53 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:56:02.314 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:56:02.315 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:02.315 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-21T13:56:02.315 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:02.315 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:56:02.315 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.0.0-2309.gba760091.el9 ceph 85 k 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 85 k 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 231 k 2024-03-21T13:56:02.316 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:56:02.541 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-fuse-19.0.0-2309.gba760091.el9.x86_64.rpm 379 kB/s | 85 kB 00:00 2024-03-21T13:56:02.542 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:02.542 INFO:teuthology.orchestra.run.smithi118.stdout:Total 376 kB/s | 85 kB 00:00 2024-03-21T13:56:02.542 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:56:02.549 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:56:02.549 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:56:02.604 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:56:02.604 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:56:02.712 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:49 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:56:02.745 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:56:02.858 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:02.868 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:56:02.868 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:19.0.0-2309.gba760091.el9 ceph 85 k 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 85 k 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 231 k 2024-03-21T13:56:02.869 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:56:03.087 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-fuse-19.0.0-2309.gba760091.el9.x86_64.rpm 393 kB/s | 85 kB 00:00 2024-03-21T13:56:03.088 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:03.088 INFO:teuthology.orchestra.run.smithi003.stdout:Total 390 kB/s | 85 kB 00:00 2024-03-21T13:56:03.088 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:56:03.095 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:56:03.096 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:56:03.153 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:56:03.154 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:56:03.267 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:03.294 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:56:03.404 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:03.568 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:03.568 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:03.568 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:56:03.568 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:56:03.568 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:03.568 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:56:03.778 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:03.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-mirror 2024-03-21T13:56:04.075 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:04.075 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:04.075 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:56:04.075 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:56:04.075 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:04.075 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:56:04.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-mirror 2024-03-21T13:56:04.389 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:55 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:56:04.544 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:56:04.544 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repo Size 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.0.0-2309.gba760091.el9 ceph 3.1 M 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 3.1 M 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 13 M 2024-03-21T13:56:04.545 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:56:04.880 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:51 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:56:04.929 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-mirror-19.0.0-2309.gba760091.el9.x86_64.rpm 8.1 MB/s | 3.1 MB 00:00 2024-03-21T13:56:04.929 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:04.930 INFO:teuthology.orchestra.run.smithi118.stdout:Total 8.0 MB/s | 3.1 MB 00:00 2024-03-21T13:56:04.930 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:56:04.939 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:56:04.939 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:56:05.012 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:56:05.013 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:56:05.035 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repo Size 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:19.0.0-2309.gba760091.el9 ceph 3.1 M 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:05.036 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:56:05.037 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:05.037 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:56:05.037 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:05.037 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 3.1 M 2024-03-21T13:56:05.037 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 13 M 2024-03-21T13:56:05.037 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:56:05.430 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-mirror-19.0.0-2309.gba760091.el9.x86_64.rpm 7.9 MB/s | 3.1 MB 00:00 2024-03-21T13:56:05.431 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:05.431 INFO:teuthology.orchestra.run.smithi003.stdout:Total 7.8 MB/s | 3.1 MB 00:00 2024-03-21T13:56:05.431 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:56:05.435 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:56:05.441 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:56:05.441 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:56:05.518 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:56:05.518 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:56:05.521 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:05.546 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:05.546 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:56:05.546 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T13:56:05.546 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:56:05.547 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:56:05.547 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:05.962 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:56:06.023 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:06.047 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:06.047 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-21T13:56:06.047 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-21T13:56:06.047 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:56:06.048 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-21T13:56:06.048 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:06.771 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:06.771 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:06.771 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:56:06.771 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:56:06.771 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:06.771 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:56:07.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install rbd-nbd 2024-03-21T13:56:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:07.183 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:56:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:56:07.183 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:07.183 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:56:07.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install rbd-nbd 2024-03-21T13:56:07.620 INFO:teuthology.orchestra.run.smithi118.stdout:Last metadata expiration check: 0:01:58 ago on Thu 21 Mar 2024 01:54:09 PM UTC. 2024-03-21T13:56:07.769 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-21T13:56:07.769 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout: Package Architecture Version Repository Size 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.0.0-2309.gba760091.el9 ceph 170 k 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:Install 1 Package 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 170 k 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:Installed size: 490 k 2024-03-21T13:56:07.770 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-21T13:56:08.002 INFO:teuthology.orchestra.run.smithi003.stdout:Last metadata expiration check: 0:01:55 ago on Thu 21 Mar 2024 01:54:13 PM UTC. 2024-03-21T13:56:08.016 INFO:teuthology.orchestra.run.smithi118.stdout:rbd-nbd-19.0.0-2309.gba760091.el9.x86_64.rpm 693 kB/s | 170 kB 00:00 2024-03-21T13:56:08.017 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:08.017 INFO:teuthology.orchestra.run.smithi118.stdout:Total 689 kB/s | 170 kB 00:00 2024-03-21T13:56:08.017 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-21T13:56:08.026 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-21T13:56:08.026 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-21T13:56:08.081 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-21T13:56:08.081 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-21T13:56:08.157 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-03-21T13:56:08.158 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:08.158 INFO:teuthology.orchestra.run.smithi003.stdout: Package Architecture Version Repository Size 2024-03-21T13:56:08.158 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:19.0.0-2309.gba760091.el9 ceph 170 k 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:Install 1 Package 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 170 k 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 490 k 2024-03-21T13:56:08.159 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-03-21T13:56:08.227 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-21T13:56:08.357 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:08.509 INFO:teuthology.orchestra.run.smithi003.stdout:rbd-nbd-19.0.0-2309.gba760091.el9.x86_64.rpm 488 kB/s | 170 kB 00:00 2024-03-21T13:56:08.509 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:08.509 INFO:teuthology.orchestra.run.smithi003.stdout:Total 486 kB/s | 170 kB 00:00 2024-03-21T13:56:08.510 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-03-21T13:56:08.518 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-03-21T13:56:08.519 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-03-21T13:56:08.578 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-03-21T13:56:08.578 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-03-21T13:56:08.731 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-03-21T13:56:08.768 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:08.828 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:09.021 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:09.022 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-21T13:56:09.022 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:56:09.022 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:56:09.022 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-21T13:56:09.277 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:09.310 DEBUG:teuthology.parallel:result is None 2024-03-21T13:56:09.557 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 1/1 2024-03-21T13:56:09.557 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:09.558 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-03-21T13:56:09.558 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:19.0.0-2309.gba760091.el9.x86_64 2024-03-21T13:56:09.558 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:09.558 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-03-21T13:56:09.787 DEBUG:teuthology.parallel:result is None 2024-03-21T13:56:09.788 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:56:10.006 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T13:56:10.026 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-2309.gba760091.el9 2024-03-21T13:56:10.027 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2309.gba760091.el9 2024-03-21T13:56:10.027 INFO:teuthology.task.install:The correct ceph version 19.0.0-2309.gba760091 is installed. 2024-03-21T13:56:10.029 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:56:10.169 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-21T13:56:10.191 INFO:teuthology.orchestra.run.smithi118.stdout:19.0.0-2309.gba760091.el9 2024-03-21T13:56:10.192 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2309.gba760091.el9 2024-03-21T13:56:10.192 INFO:teuthology.task.install:The correct ceph version 19.0.0-2309.gba760091 is installed. 2024-03-21T13:56:10.194 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-21T13:56:10.194 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:56:10.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T13:56:10.222 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:56:10.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-21T13:56:10.261 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-21T13:56:10.261 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:56:10.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T13:56:10.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T13:56:10.358 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:56:10.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-03-21T13:56:10.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-21T13:56:10.454 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-21T13:56:10.454 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:56:10.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T13:56:10.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T13:56:10.547 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:56:10.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-21T13:56:10.575 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-21T13:56:10.642 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-21T13:56:10.642 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:56:10.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T13:56:10.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T13:56:10.736 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:56:10.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-03-21T13:56:10.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-21T13:56:10.830 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-21T13:56:10.936 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'OSDMAP_FLAGS', 'PG_DEGRADED', 'OSD_DOWN', 'POOL_APP_NOT_ENABLED', 'CACHE_POOL_NO_HIT_SET', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988', 'cephadm_mode': 'root'} 2024-03-21T13:56:10.937 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:56:10.937 INFO:tasks.cephadm:Cluster fsid is c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:10.937 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-21T13:56:10.938 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.c': '[v2:172.21.15.3:3301,v1:172.21.15.3:6790]', 'mon.b': '172.21.15.118'} 2024-03-21T13:56:10.938 INFO:tasks.cephadm:First mon is mon.a on smithi003 2024-03-21T13:56:10.938 INFO:tasks.cephadm:First mgr is y 2024-03-21T13:56:10.938 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-21T13:56:10.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2024-03-21T13:56:10.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo hostname $(hostname -s) 2024-03-21T13:56:10.992 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-21T13:56:10.992 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:56:11.138 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77824/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2309-gba760091', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2309.gba760091'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-21 05:01:49.741818', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-03-20-2326', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-21T13:56:11.280 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-adk-testing-2024-03-20-2326, sha1 ba760091cd7bd2b0d23f4825ac856ba66450e988 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:56:11.282 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/x86_64/flavors/default/cephadm 2024-03-21T13:56:11.282 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T13:56:11.897 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-r--r--. 1 ubuntu ubuntu 781442 Mar 21 13:56 /home/ubuntu/cephtest/cephadm 2024-03-21T13:56:11.897 DEBUG:teuthology.orchestra.run.smithi118:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-20-2326/ba760091cd7bd2b0d23f4825ac856ba66450e988/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-21T13:56:12.333 INFO:teuthology.orchestra.run.smithi118.stdout:-rw-r--r--. 1 ubuntu ubuntu 781442 Mar 21 13:56 /home/ubuntu/cephtest/cephadm 2024-03-21T13:56:12.334 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T13:56:12.353 DEBUG:teuthology.orchestra.run.smithi118:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-21T13:56:12.384 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 on all hosts... 2024-03-21T13:56:12.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 pull 2024-03-21T13:56:12.395 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 pull 2024-03-21T13:56:13.303 INFO:teuthology.orchestra.run.smithi118.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988... 2024-03-21T13:56:13.310 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988... 2024-03-21T13:56:41.245 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T13:56:41.246 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_version": "ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev)", 2024-03-21T13:56:41.246 INFO:teuthology.orchestra.run.smithi118.stdout: "image_id": "07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c", 2024-03-21T13:56:41.246 INFO:teuthology.orchestra.run.smithi118.stdout: "repo_digests": [ 2024-03-21T13:56:41.246 INFO:teuthology.orchestra.run.smithi118.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7" 2024-03-21T13:56:41.246 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T13:56:41.246 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T13:56:45.685 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T13:56:45.686 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev)", 2024-03-21T13:56:45.686 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c", 2024-03-21T13:56:45.686 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2024-03-21T13:56:45.686 INFO:teuthology.orchestra.run.smithi003.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7" 2024-03-21T13:56:45.686 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T13:56:45.686 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T13:56:45.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2024-03-21T13:56:45.748 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph 2024-03-21T13:56:45.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2024-03-21T13:56:45.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /etc/ceph 2024-03-21T13:56:45.854 INFO:tasks.cephadm:Writing seed config... 2024-03-21T13:56:45.855 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-21T13:56:45.856 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-21T13:56:45.857 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-21T13:56:45.858 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:56:45.858 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-21T13:56:45.873 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c60e082a-e78a-11ee-95cd-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-21T13:56:45.873 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a.service 2024-03-21T13:56:45.915 DEBUG:teuthology.orchestra.run.smithi003:mgr.y> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y.service 2024-03-21T13:56:45.958 INFO:tasks.cephadm:Bootstrapping... 2024-03-21T13:56:45.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 -v bootstrap --fsid c60e082a-e78a-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:56:46.153 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-03-21T13:56:46.153 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988', '-v', 'bootstrap', '--fsid', 'c60e082a-e78a-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2024-03-21T13:56:46.153 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-21T13:56:46.153 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2024-03-21T13:56:46.178 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T13:56:46.178 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2024-03-21T13:56:46.178 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2024-03-21T13:56:46.185 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T13:56:46.186 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T13:56:46.192 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T13:56:46.192 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-21T13:56:46.198 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-03-21T13:56:46.204 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-03-21T13:56:46.204 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-03-21T13:56:46.204 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2024-03-21T13:56:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-21T13:56:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-21T13:56:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2024-03-21T13:56:46.227 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2024-03-21T13:56:46.233 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-21T13:56:46.233 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-21T13:56:46.239 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-21T13:56:46.239 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-21T13:56:46.245 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-03-21T13:56:46.251 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-03-21T13:56:46.251 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-03-21T13:56:46.251 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2024-03-21T13:56:46.251 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:46.251 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 140576617943728 on /run/cephadm/c60e082a-e78a-11ee-95cd-87774f69a715.lock 2024-03-21T13:56:46.251 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 140576617943728 acquired on /run/cephadm/c60e082a-e78a-11ee-95cd-87774f69a715.lock 2024-03-21T13:56:46.252 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2024-03-21T13:56:46.252 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2024-03-21T13:56:46.252 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-03-21T13:56:46.255 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2024-03-21T13:56:46.255 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2024-03-21T13:56:46.258 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-21T13:56:46.258 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-21T13:56:46.258 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-21T13:56:46.260 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-21T13:56:46.260 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-21T13:56:46.260 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T13:56:46.260 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-21T13:56:46.261 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-03-21T13:56:46.261 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-21T13:56:46.261 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-03-21T13:56:46.261 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-03-21T13:56:46.261 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-21T13:56:46.262 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-21T13:56:46.262 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988... 2024-03-21T13:56:46.747 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c 2024-03-21T13:56:46.747 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988... 2024-03-21T13:56:46.747 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2024-03-21T13:56:46.748 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:4e10ee87d001645254c426ed06600df0d72b4c104a9e51cecddb44e1f9ef0c69 2024-03-21T13:56:46.748 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-21T13:56:46.748 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c 2024-03-21T13:56:46.748 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-21T13:56:47.702 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev) 2024-03-21T13:56:47.702 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev) 2024-03-21T13:56:47.702 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2024-03-21T13:56:48.516 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2024-03-21T13:56:48.516 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2024-03-21T13:56:49.388 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQCgPPxlNaI3MRAAZSH6Anf5wacDRU/VRrOfeQ== 2024-03-21T13:56:50.319 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQChPPxliaR1KxAAxLRGHtrJ8z1ldGjl7F1VpQ== 2024-03-21T13:56:51.299 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQCiPPxlPwWGJxAAl8ktJzFSDV3isdGtOqIK+A== 2024-03-21T13:56:51.300 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:56:52.156 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2024-03-21T13:56:53.114 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2024-03-21T13:56:53.310 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-21T13:56:53.446 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-21T13:56:53.635 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c60e082a-e78a-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-c60e082a-e78a-11ee-95cd-87774f69a715.target. 2024-03-21T13:56:53.635 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c60e082a-e78a-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-c60e082a-e78a-11ee-95cd-87774f69a715.target. 2024-03-21T13:56:53.857 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a 2024-03-21T13:56:53.857 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-21T13:56:54.016 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c60e082a-e78a-11ee-95cd-87774f69a715.target.wants/ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c60e082a-e78a-11ee-95cd-87774f69a715@.service. 2024-03-21T13:56:54.560 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:56:54.561 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-21T13:56:54.567 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:56:54.567 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-21T13:56:54.567 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-21T13:56:54.567 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T13:56:54.568 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2024-03-21T13:56:54.568 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2024-03-21T13:56:55.611 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:55 smithi003 ceph-mon[29203]: from='client.? 172.21.15.3:0/1820888552' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-21T13:56:55.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2024-03-21T13:56:55.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:55.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-21T13:56:55.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:56:55.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.5503s) 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2024-03-21T13:56:55.616 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2024-03-21T13:56:56.621 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T13:56:56.622 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T13:56:56.623 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T13:56:56.623 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2024-03-21T13:56:56.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:56 smithi003 ceph-mon[29203]: from='client.? 172.21.15.3:0/4200214874' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T13:56:56.743 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:56 smithi003 ceph-mon[29203]: from='client.? 172.21.15.3:0/4200214874' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-21T13:56:57.660 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2024-03-21T13:56:57.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:57 smithi003 ceph-mon[29203]: from='client.? 172.21.15.3:0/4211624582' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:56:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:57 smithi003 systemd[1]: Stopping Ceph mon.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T13:56:58.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:57 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a[29199]: 2024-03-21T13:56:57.926+0000 7fee1c543700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T13:56:58.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:57 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a[29199]: 2024-03-21T13:56:57.926+0000 7fee1c543700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-21T13:56:58.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 podman[29448]: 2024-03-21 13:56:58.091030739 +0000 UTC m=+0.265139859 container died d30e22c6e61b8ac47eff5de24b8aa225811b5e1749ca1d26a291614c489da713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-21T13:56:58.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 podman[29448]: 2024-03-21 13:56:58.377116621 +0000 UTC m=+0.551225740 container cleanup d30e22c6e61b8ac47eff5de24b8aa225811b5e1749ca1d26a291614c489da713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-21T13:56:58.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 bash[29448]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a 2024-03-21T13:56:58.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 podman[29462]: 2024-03-21 13:56:58.611337481 +0000 UTC m=+0.518858474 container remove d30e22c6e61b8ac47eff5de24b8aa225811b5e1749ca1d26a291614c489da713 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-21T13:56:59.088 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-21T13:56:59.088 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 systemd[1]: Stopped Ceph mon.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T13:56:59.088 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 systemd[1]: Starting Ceph mon.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T13:56:59.088 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 podman[29557]: 2024-03-21 13:56:58.872034392 +0000 UTC m=+0.017581805 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:56:59.088 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:58 smithi003 podman[29557]: 2024-03-21 13:56:58.970395421 +0000 UTC m=+0.115942813 container create 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-21T13:56:59.309 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-21T13:56:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 podman[29557]: 2024-03-21 13:56:59.222505194 +0000 UTC m=+0.368052585 container init 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:56:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 podman[29557]: 2024-03-21 13:56:59.22536255 +0000 UTC m=+0.370909932 container start 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:56:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T13:56:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev), process ceph-mon, pid 2 2024-03-21T13:56:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: pidfile_write: ignore empty --pid-file 2024-03-21T13:56:59.432 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: load: jerasure load: lrc 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: RocksDB version: 7.9.2 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Git sha 0 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Compile date 2024-03-21 04:02:31 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: DB SUMMARY 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: DB Session ID: MKJIIG9000JRMGO4NPKQ 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: CURRENT file: CURRENT 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 92009 ; 2024-03-21T13:56:59.433 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.error_if_exists: 0 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.create_if_missing: 0 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.paranoid_checks: 1 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.env: 0x556aaddd5b20 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.info_log: 0x556aaf4bb920 2024-03-21T13:56:59.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.statistics: (nil) 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.use_fsync: 0 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_log_file_size: 0 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.allow_fallocate: 1 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T13:56:59.436 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.use_direct_reads: 0 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.db_log_dir: 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.wal_dir: 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T13:56:59.437 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.write_buffer_manager: 0x556aaf48c1e0 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T13:56:59.438 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.unordered_write: 0 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.row_cache: None 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.wal_filter: None 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T13:56:59.439 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.two_write_queues: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.wal_compression: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.atomic_flush: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.log_readahead_size: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T13:56:59.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_background_jobs: 2 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_background_compactions: -1 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_subcompactions: 1 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T13:56:59.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_open_files: -1 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_background_flushes: -1 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Compression algorithms supported: 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kZSTD supported: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kXpressCompression supported: 0 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kZlibCompression supported: 1 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kSnappyCompression supported: 1 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kLZ4Compression supported: 1 2024-03-21T13:56:59.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: kBZip2Compression supported: 0 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.merge_operator: 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_filter: None 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556aaf4bba20) 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-03-21T13:56:59.443 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x556aaf4bf090 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-03-21T13:56:59.444 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2024-03-21T13:56:59.445 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression: NoCompression 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.num_levels: 7 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T13:56:59.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T13:56:59.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T13:56:59.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T13:56:59.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.table_properties_collectors: 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.inplace_update_support: 0 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.bloom_locality: 0 2024-03-21T13:56:59.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.max_successive_merges: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.ttl: 2592000 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.enable_blob_files: false 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.min_blob_size: 0 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T13:56:59.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 61d28daf-e676-4797-abac-f91b104ca9d3 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029419254442, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029419255776, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 85970, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11802, "raw_average_key_size": 48, "raw_value_size": 79505, "raw_average_value_size": 329, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711029419, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "61d28daf-e676-4797-abac-f91b104ca9d3", "db_session_id": "MKJIIG9000JRMGO4NPKQ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029419255885, "job": 1, "event": "recovery_finished"} 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T13:56:59.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556aaf5ae000 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: DB pointer 0x556aaf598000 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-21T13:56:59.453 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 87.66 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 87.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 107.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 107.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-21T13:56:59.454 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 13.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 13.80 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x556aaf4bf090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T13:56:59.455 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???) e1 preinit fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).mds e1 new map 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).mds e1 print_map 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-21T13:56:59.456 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 bash[29557]: 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 2024-03-21T13:56:59.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 systemd[1]: Started Ceph mon.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T13:56:59.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-21T13:56:59.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:56:59.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: fsmap 2024-03-21T13:56:59.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: osdmap e1: 0 total, 0 up, 0 in 2024-03-21T13:56:59.457 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:56:59 smithi003 ceph-mon[29576]: mgrmap e1: no daemons active 2024-03-21T13:57:00.474 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-21T13:57:00.496 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:57:00.496 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2024-03-21T13:57:00.496 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-21T13:57:00.496 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-21T13:57:00.748 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y 2024-03-21T13:57:00.748 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-21T13:57:00.895 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c60e082a-e78a-11ee-95cd-87774f69a715.target.wants/ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c60e082a-e78a-11ee-95cd-87774f69a715@.service. 2024-03-21T13:57:01.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/816916031' entity='client.admin' 2024-03-21T13:57:01.115 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:00 smithi003 systemd[1]: Starting Ceph mgr.y for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T13:57:01.383 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 podman[29792]: 2024-03-21 13:57:01.114820342 +0000 UTC m=+0.101047626 container create a24079a1e6eb3aa9bf34027805da31f6b05e8eda3dc338c4719e93e63113d4c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T13:57:01.383 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 podman[29792]: 2024-03-21 13:57:01.031415922 +0000 UTC m=+0.017643207 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 2024-03-21T13:57:01.383 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 podman[29792]: 2024-03-21 13:57:01.32539499 +0000 UTC m=+0.311622280 container init a24079a1e6eb3aa9bf34027805da31f6b05e8eda3dc338c4719e93e63113d4c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-03-21T13:57:01.383 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 podman[29792]: 2024-03-21 13:57:01.327833912 +0000 UTC m=+0.314061201 container start a24079a1e6eb3aa9bf34027805da31f6b05e8eda3dc338c4719e93e63113d4c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-21T13:57:01.383 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 bash[29792]: a24079a1e6eb3aa9bf34027805da31f6b05e8eda3dc338c4719e93e63113d4c8 2024-03-21T13:57:01.396 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:57:01.396 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-21T13:57:01.404 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:57:01.404 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-21T13:57:01.404 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-21T13:57:01.404 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-03-21T13:57:01.414 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:57:01.414 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-21T13:57:01.422 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:57:01.422 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-21T13:57:01.423 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-21T13:57:01.423 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-21T13:57:01.423 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2024-03-21T13:57:01.423 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2024-03-21T13:57:01.712 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 systemd[1]: Started Ceph mgr.y for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T13:57:02.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:01 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/269044402' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:57:02.002 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:01 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:01.903+0000 7f9500908200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:57:02.002 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:02.001+0000 7f9500908200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:57:02.517 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:02.523 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:57:02.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:02.525 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:57:02.526 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:02.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:56:54.530585+0000", 2024-03-21T13:57:02.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:02.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:02.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:57:02.527 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:02.527 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2024-03-21T13:57:02.769 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:02.674+0000 7f9500908200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:57:02.769 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:02.768+0000 7f9500908200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:57:03.099 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:02.912+0000 7f9500908200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:57:03.099 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.011+0000 7f9500908200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:57:03.099 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.099+0000 7f9500908200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:57:03.681 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.374+0000 7f9500908200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:57:03.681 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.490+0000 7f9500908200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:57:03.681 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.580+0000 7f9500908200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:57:03.681 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.680+0000 7f9500908200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:57:04.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:03.859+0000 7f9500908200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:57:05.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:04 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:04.877+0000 7f9500908200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:57:05.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:05 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4136771205' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:57:05.363 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:05 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:05.014+0000 7f9500908200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:57:05.576 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:05.576 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-21T13:57:05.577 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:57:05.578 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.582 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:57:05.583 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:05.583 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:57:05.583 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:57:05.583 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:56:54.530585+0000", 2024-03-21T13:57:05.584 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:05.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:05.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:57:05.585 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:05.585 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (2/15)... 2024-03-21T13:57:05.651 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:05 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:05.362+0000 7f9500908200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:57:05.652 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:05 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:05.454+0000 7f9500908200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:57:06.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:05 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:05.656+0000 7f9500908200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:57:06.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:05 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:05.844+0000 7f9500908200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:57:06.400 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:06.136+0000 7f9500908200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:57:06.400 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:06.399+0000 7f9500908200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:57:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:06.642+0000 7f9500908200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:57:07.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:07 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:07.472+0000 7f9500908200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:57:08.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3917398547' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:57:08.395 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:08 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:08.067+0000 7f9500908200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:57:08.395 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:08 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:08.169+0000 7f9500908200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:57:08.395 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:08 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:08.394+0000 7f9500908200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.610 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:57:08.611 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.612 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.613 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:56:54.530585+0000", 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:08.614 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:08.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:57:08.615 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:08.615 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (3/15)... 2024-03-21T13:57:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: Activating manager daemon y 2024-03-21T13:57:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: mgrmap e2: y(active, starting, since 0.00285801s) 2024-03-21T13:57:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: Manager daemon y is now available 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' 2024-03-21T13:57:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:09 smithi003 ceph-mon[29576]: from='mgr.14100 172.21.15.3:0/3206851569' entity='mgr.y' 2024-03-21T13:57:10.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:10 smithi003 ceph-mon[29576]: mgrmap e3: y(active, since 1.00682s) 2024-03-21T13:57:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:12 smithi003 ceph-mon[29576]: mgrmap e4: y(active, since 3s) 2024-03-21T13:57:14.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1298256054' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-21T13:57:20.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:20.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:20.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T13:57:20.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-03-21T13:57:20.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-21T13:57:20.313 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-21T13:57:20.314 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-21T13:57:20.315 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-21T13:57:20.316 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:20.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-21T13:57:20.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-21T13:57:20.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-21T13:57:20.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T13:57:20.317 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-03-21T13:56:54.530585+0000", 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-21T13:57:20.318 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:20.319 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2024-03-21T13:57:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3000437163' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-21T13:57:21.506 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-21T13:57:21.507 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2024-03-21T13:57:22.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:22 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1697639316' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-21T13:57:22.558 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:22 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ignoring --setuser ceph since I am not root 2024-03-21T13:57:22.559 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:22 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ignoring --setgroup ceph since I am not root 2024-03-21T13:57:23.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:22 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:22.833+0000 7fae1072a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:57:23.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:22 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:22.929+0000 7fae1072a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:57:23.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1697639316' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-21T13:57:23.664 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:23 smithi003 ceph-mon[29576]: mgrmap e5: y(active, since 13s) 2024-03-21T13:57:23.947 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:23.606+0000 7fae1072a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:57:23.947 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:23.702+0000 7fae1072a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:57:23.947 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:23.847+0000 7fae1072a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:57:23.947 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:23.946+0000 7fae1072a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:57:24.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:24.033+0000 7fae1072a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:57:24.614 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3035994026' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T13:57:24.614 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:24.311+0000 7fae1072a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:57:24.614 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:24.427+0000 7fae1072a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:57:24.614 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:24.515+0000 7fae1072a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:57:24.615 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:24.613+0000 7fae1072a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:57:24.973 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:24 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:24.793+0000 7fae1072a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:24.977 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-03-21T13:57:24.978 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2024-03-21T13:57:26.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:25 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:25.795+0000 7fae1072a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:57:26.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:25 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:25.929+0000 7fae1072a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:57:26.567 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:26.269+0000 7fae1072a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:57:26.567 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:26.362+0000 7fae1072a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:57:26.568 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:26.567+0000 7fae1072a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:57:27.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:26.757+0000 7fae1072a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:57:27.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:27 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:27.054+0000 7fae1072a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:57:27.317 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:27 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:27.317+0000 7fae1072a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:57:27.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:27 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:27.563+0000 7fae1072a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:57:28.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:28 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:28.397+0000 7fae1072a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:57:29.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:28 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:28.988+0000 7fae1072a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:57:29.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:29.090+0000 7fae1072a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:57:29.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:29.314+0000 7fae1072a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:57:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: Active manager daemon y restarted 2024-03-21T13:57:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: Activating manager daemon y 2024-03-21T13:57:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: osdmap e2: 0 total, 0 up, 0 in 2024-03-21T13:57:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: mgrmap e6: y(active, starting, since 0.00295269s) 2024-03-21T13:57:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: Manager daemon y is now available 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:57:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:29 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:57:30.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:30 smithi003 ceph-mon[29576]: Found migration_current of "None". Setting to last migration. 2024-03-21T13:57:30.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:30 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:57:30.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:30 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:57:30.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:30 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:30.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:30 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:30.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:30 smithi003 ceph-mon[29576]: mgrmap e7: y(active, since 1.00704s) 2024-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2024-03-21T13:57:30.842 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2024-03-21T13:57:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T13:57:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-21T13:57:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:30] ENGINE Bus STARTING 2024-03-21T13:57:31.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:30] ENGINE Serving on https://172.21.15.3:7150 2024-03-21T13:57:31.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:30] ENGINE Serving on http://172.21.15.3:8765 2024-03-21T13:57:31.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:30] ENGINE Bus STARTED 2024-03-21T13:57:31.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:31 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:57:32.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:32 smithi003 ceph-mon[29576]: mgrmap e8: y(active, since 2s) 2024-03-21T13:57:32.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:32 smithi003 ceph-mon[29576]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:32.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:32 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:32.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:32 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:57:33.212 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-21T13:57:33.212 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2024-03-21T13:57:33.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:33 smithi003 ceph-mon[29576]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:34.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: Generating public/private rsa key pair. 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: Your identification has been saved in /tmp/tmpoqtywr0p/key. 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: Your public key has been saved in /tmp/tmpoqtywr0p/key.pub. 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: The key fingerprint is: 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: SHA256:fPZO0TZJYEl4E6A5NVZom30W84dPPFVTLxqkgdfJSHU ceph-c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: The key's randomart image is: 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: +---[RSA 3072]----+ 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | oB@O+E.=| 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | .=**Boo +| 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | +o.=..o*o| 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | . .o .=++=| 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | S o oo=oo| 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | o . o ..| 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | o | 2024-03-21T13:57:34.255 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | o | 2024-03-21T13:57:34.256 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: | . | 2024-03-21T13:57:34.256 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:34 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: +----[SHA256]-----+ 2024-03-21T13:57:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:35 smithi003 ceph-mon[29576]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:35 smithi003 ceph-mon[29576]: Generating ssh key... 2024-03-21T13:57:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:35 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:35 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:35.850 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxDfVoIjmRVfTLc0xQgB/e6b6NpKXzUJ8aywjdNkIC8ioXwi33C8OA0kAxJ3G/KUBUSSmoD3EEP+C5J3C0BYaUlIbapaWQtYQO+2rd57+dGLhzGU45GwwbPeL+XkWj8SfciX7vUxxwW2uaVlBz6zaSaV2NY5j+ENXyqM/QX9XYo7ffRe+OtMkmBmfW+s2BWw22xwVSp432EiGx580DgkCRmLSwQib0m72hr1XYii1l6GqfM3YaVJC788FFubsF5I8KMY5LRJKEEm+w8ZvIJ9i7fk32i08sGO6Co+kS1QjrBifqNpLp7J7h1DY3t42IeMfrzc+YeTTTNyI1ByPzoEcfykwZFyg4E98ZtHBHoU2lpLz6DMHrX5l1n8++dZpuehNQTMWxori1AOIix6YZqN90lN1XPRXIJnsCKg+lj4WQ6yz+9ocnoM/n2McybCxu6vH7rLje1rwLoTB5vtHs5Anzc5SdLlj5G7UX1W9HVF1DhHOahslsiO2iTb8iHokvSU= ceph-c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:57:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-21T13:57:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2024-03-21T13:57:35.851 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2024-03-21T13:57:36.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:36 smithi003 ceph-mon[29576]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:37.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:37 smithi003 ceph-mon[29576]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:37.679 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:37 smithi003 ceph-mon[29576]: Deploying cephadm binary to smithi003 2024-03-21T13:57:41.524 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2024-03-21T13:57:41.524 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2024-03-21T13:57:42.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:41 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:42.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:41 smithi003 ceph-mon[29576]: Added host smithi003 2024-03-21T13:57:42.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:41 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:57:43.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:43 smithi003 ceph-mon[29576]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:43.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:43 smithi003 ceph-mon[29576]: Saving service mon spec with placement count:5 2024-03-21T13:57:43.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:43 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:43.690 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-21T13:57:43.690 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2024-03-21T13:57:44.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:44 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:44.651 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:44 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:44.937 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-21T13:57:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:45 smithi003 ceph-mon[29576]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:45 smithi003 ceph-mon[29576]: Saving service mgr spec with placement count:2 2024-03-21T13:57:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:45 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:46.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:46 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4161333545' entity='client.admin' 2024-03-21T13:57:47.811 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2024-03-21T13:57:48.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:48 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:48.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4118424294' entity='client.admin' 2024-03-21T13:57:49.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:49 smithi003 ceph-mon[29576]: from='mgr.14120 172.21.15.3:0/3192430607' entity='mgr.y' 2024-03-21T13:57:49.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/395745902' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-21T13:57:49.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:49 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ignoring --setuser ceph since I am not root 2024-03-21T13:57:49.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:49 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ignoring --setgroup ceph since I am not root 2024-03-21T13:57:50.498 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:50 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:50.128+0000 7f6453e92200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:57:50.498 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:50 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:50.223+0000 7f6453e92200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:57:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/395745902' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-21T13:57:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:50 smithi003 ceph-mon[29576]: mgrmap e9: y(active, since 20s) 2024-03-21T13:57:51.141 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:50 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:50.898+0000 7f6453e92200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:57:51.141 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:50 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:50.996+0000 7f6453e92200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:57:51.141 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.140+0000 7f6453e92200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:57:51.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.240+0000 7f6453e92200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:57:51.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.326+0000 7f6453e92200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:57:51.796 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1463288599' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-21T13:57:51.796 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.595+0000 7f6453e92200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:57:51.796 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.708+0000 7f6453e92200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:57:51.796 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.795+0000 7f6453e92200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:57:51.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-03-21T13:57:51.892 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2024-03-21T13:57:52.073 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:51.894+0000 7f6453e92200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:57:52.350 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:52 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:52.077+0000 7f6453e92200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:57:53.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:53.065+0000 7f6453e92200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:57:53.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:53.198+0000 7f6453e92200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:57:53.832 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:53.537+0000 7f6453e92200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:57:53.832 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:53.629+0000 7f6453e92200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:57:53.832 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:53.831+0000 7f6453e92200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:57:54.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:54.021+0000 7f6453e92200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:57:54.576 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:54.314+0000 7f6453e92200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:57:54.577 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:54.576+0000 7f6453e92200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:57:55.006 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:54.823+0000 7f6453e92200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:57:56.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:55 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:55.657+0000 7f6453e92200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:57:56.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:56.238+0000 7f6453e92200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:57:56.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:56.339+0000 7f6453e92200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:57:57.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:57:56.563+0000 7f6453e92200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:57:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: Active manager daemon y restarted 2024-03-21T13:57:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: Activating manager daemon y 2024-03-21T13:57:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: osdmap e3: 0 total, 0 up, 0 in 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: mgrmap e10: y(active, starting, since 0.00337974s) 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: Manager daemon y is now available 2024-03-21T13:57:57.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:57:57.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T13:57:57.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T13:57:57.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:57 smithi003 ceph-mon[29576]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-21T13:57:57.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:57 smithi003 ceph-mon[29576]: mgrmap e11: y(active, since 1.00571s) 2024-03-21T13:57:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-03-21T13:57:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-21T13:57:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-21T13:57:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-03-21T13:57:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2024-03-21T13:57:58.149 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2024-03-21T13:57:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:58 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:57] ENGINE Bus STARTING 2024-03-21T13:57:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:58 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:57] ENGINE Serving on http://172.21.15.3:8765 2024-03-21T13:57:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:58 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:57] ENGINE Serving on https://172.21.15.3:7150 2024-03-21T13:57:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:58 smithi003 ceph-mon[29576]: [21/Mar/2024:13:57:57] ENGINE Bus STARTED 2024-03-21T13:57:59.840 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-21T13:57:59.840 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2024-03-21T13:57:59.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:59 smithi003 ceph-mon[29576]: mgrmap e12: y(active, since 2s) 2024-03-21T13:57:59.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:57:59.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:59 smithi003 ceph-mon[29576]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:57:59.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:57:59.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:57:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:01.255 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$P8yHDjABx3xr3GAAR8tUKOgJ56NPMROuygndRDM6jPgxIXur0wdpK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711029480, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-21T13:58:01.255 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2024-03-21T13:58:01.588 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:01.588 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:01 smithi003 ceph-mon[29576]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:01.589 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:02.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/983523227' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-21T13:58:02.597 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2024-03-21T13:58:02.607 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-21T13:58:02.607 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-03-21T13:58:02.614 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-21T13:58:02.614 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-03-21T13:58:02.614 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-03-21T13:58:02.615 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003.front.sepia.ceph.com:8443/ 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout: Password: 25cil3cmke 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config directory 2024-03-21T13:58:02.617 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2024-03-21T13:58:04.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:04 smithi003 ceph-mon[29576]: mgrmap e13: y(active, since 6s) 2024-03-21T13:58:05.564 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-21T13:58:05.564 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-21T13:58:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:05.564 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.564 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:05.565 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2024-03-21T13:58:05.608 INFO:tasks.cephadm:Fetching config... 2024-03-21T13:58:05.609 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:05.609 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-21T13:58:05.624 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-21T13:58:05.624 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:05.625 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-21T13:58:05.680 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-21T13:58:05.680 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:05.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-21T13:58:05.748 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-21T13:58:05.748 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:05.748 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-21T13:58:05.803 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-21T13:58:05.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxDfVoIjmRVfTLc0xQgB/e6b6NpKXzUJ8aywjdNkIC8ioXwi33C8OA0kAxJ3G/KUBUSSmoD3EEP+C5J3C0BYaUlIbapaWQtYQO+2rd57+dGLhzGU45GwwbPeL+XkWj8SfciX7vUxxwW2uaVlBz6zaSaV2NY5j+ENXyqM/QX9XYo7ffRe+OtMkmBmfW+s2BWw22xwVSp432EiGx580DgkCRmLSwQib0m72hr1XYii1l6GqfM3YaVJC788FFubsF5I8KMY5LRJKEEm+w8ZvIJ9i7fk32i08sGO6Co+kS1QjrBifqNpLp7J7h1DY3t42IeMfrzc+YeTTTNyI1ByPzoEcfykwZFyg4E98ZtHBHoU2lpLz6DMHrX5l1n8++dZpuehNQTMWxori1AOIix6YZqN90lN1XPRXIJnsCKg+lj4WQ6yz+9ocnoM/n2McybCxu6vH7rLje1rwLoTB5vtHs5Anzc5SdLlj5G7UX1W9HVF1DhHOahslsiO2iTb8iHokvSU= ceph-c60e082a-e78a-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T13:58:05.881 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxDfVoIjmRVfTLc0xQgB/e6b6NpKXzUJ8aywjdNkIC8ioXwi33C8OA0kAxJ3G/KUBUSSmoD3EEP+C5J3C0BYaUlIbapaWQtYQO+2rd57+dGLhzGU45GwwbPeL+XkWj8SfciX7vUxxwW2uaVlBz6zaSaV2NY5j+ENXyqM/QX9XYo7ffRe+OtMkmBmfW+s2BWw22xwVSp432EiGx580DgkCRmLSwQib0m72hr1XYii1l6GqfM3YaVJC788FFubsF5I8KMY5LRJKEEm+w8ZvIJ9i7fk32i08sGO6Co+kS1QjrBifqNpLp7J7h1DY3t42IeMfrzc+YeTTTNyI1ByPzoEcfykwZFyg4E98ZtHBHoU2lpLz6DMHrX5l1n8++dZpuehNQTMWxori1AOIix6YZqN90lN1XPRXIJnsCKg+lj4WQ6yz+9ocnoM/n2McybCxu6vH7rLje1rwLoTB5vtHs5Anzc5SdLlj5G7UX1W9HVF1DhHOahslsiO2iTb8iHokvSU= ceph-c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:05.896 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxDfVoIjmRVfTLc0xQgB/e6b6NpKXzUJ8aywjdNkIC8ioXwi33C8OA0kAxJ3G/KUBUSSmoD3EEP+C5J3C0BYaUlIbapaWQtYQO+2rd57+dGLhzGU45GwwbPeL+XkWj8SfciX7vUxxwW2uaVlBz6zaSaV2NY5j+ENXyqM/QX9XYo7ffRe+OtMkmBmfW+s2BWw22xwVSp432EiGx580DgkCRmLSwQib0m72hr1XYii1l6GqfM3YaVJC788FFubsF5I8KMY5LRJKEEm+w8ZvIJ9i7fk32i08sGO6Co+kS1QjrBifqNpLp7J7h1DY3t42IeMfrzc+YeTTTNyI1ByPzoEcfykwZFyg4E98ZtHBHoU2lpLz6DMHrX5l1n8++dZpuehNQTMWxori1AOIix6YZqN90lN1XPRXIJnsCKg+lj4WQ6yz+9ocnoM/n2McybCxu6vH7rLje1rwLoTB5vtHs5Anzc5SdLlj5G7UX1W9HVF1DhHOahslsiO2iTb8iHokvSU= ceph-c60e082a-e78a-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-21T13:58:05.942 INFO:teuthology.orchestra.run.smithi118.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvxDfVoIjmRVfTLc0xQgB/e6b6NpKXzUJ8aywjdNkIC8ioXwi33C8OA0kAxJ3G/KUBUSSmoD3EEP+C5J3C0BYaUlIbapaWQtYQO+2rd57+dGLhzGU45GwwbPeL+XkWj8SfciX7vUxxwW2uaVlBz6zaSaV2NY5j+ENXyqM/QX9XYo7ffRe+OtMkmBmfW+s2BWw22xwVSp432EiGx580DgkCRmLSwQib0m72hr1XYii1l6GqfM3YaVJC788FFubsF5I8KMY5LRJKEEm+w8ZvIJ9i7fk32i08sGO6Co+kS1QjrBifqNpLp7J7h1DY3t42IeMfrzc+YeTTTNyI1ByPzoEcfykwZFyg4E98ZtHBHoU2lpLz6DMHrX5l1n8++dZpuehNQTMWxori1AOIix6YZqN90lN1XPRXIJnsCKg+lj4WQ6yz+9ocnoM/n2McybCxu6vH7rLje1rwLoTB5vtHs5Anzc5SdLlj5G7UX1W9HVF1DhHOahslsiO2iTb8iHokvSU= ceph-c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:05.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-21T13:58:06.216 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:06 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1966947195' entity='client.admin' 2024-03-21T13:58:06.354 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:09.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3577291826' entity='client.admin' 2024-03-21T13:58:09.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:09.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:09.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:58:09.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:09.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:09.330 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:09 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:09.362 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-21T13:58:09.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-21T13:58:09.937 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:12.389 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi118 2024-03-21T13:58:12.390 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:58:12.390 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/etc/ceph/ceph.conf 2024-03-21T13:58:12.409 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:58:12.409 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:12.466 INFO:tasks.cephadm:Adding host smithi118 to orchestrator... 2024-03-21T13:58:12.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch host add smithi118 2024-03-21T13:58:12.707 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T13:58:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:58:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:14 smithi003 ceph-mon[29576]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi118", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:15 smithi003 ceph-mon[29576]: Deploying cephadm binary to smithi118 2024-03-21T13:58:17.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:16 smithi003 ceph-mon[29576]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:18.974 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:18 smithi003 ceph-mon[29576]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:18.975 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi118' with addr '172.21.15.118' 2024-03-21T13:58:19.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-21T13:58:19.821 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:20.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:20.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:19 smithi003 ceph-mon[29576]: Added host smithi118 2024-03-21T13:58:20.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:20.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:21.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:21.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:21 smithi003 ceph-mon[29576]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:21.497 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T13:58:21.497 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.118", "hostname": "smithi118", "labels": [], "status": ""}] 2024-03-21T13:58:22.065 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-21T13:58:22.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-21T13:58:22.303 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:22.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:22 smithi003 ceph-mon[29576]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:58:22.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:22 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:23.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:23 smithi003 ceph-mon[29576]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:24.564 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-03-21T13:58:24.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1347494758' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T13:58:24.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:24.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:24.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:24.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:24.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:58:24.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:24.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:25.228 INFO:tasks.cephadm:Adding mon.a on smithi003 2024-03-21T13:58:25.228 INFO:tasks.cephadm:Adding mon.c on smithi003 2024-03-21T13:58:25.228 INFO:tasks.cephadm:Adding mon.b on smithi118 2024-03-21T13:58:25.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b' 2024-03-21T13:58:25.463 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T13:58:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1347494758' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T13:58:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:26.898 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled mon update... 2024-03-21T13:58:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:26 smithi003 ceph-mon[29576]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:26 smithi003 ceph-mon[29576]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:26 smithi003 ceph-mon[29576]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:58:27.480 DEBUG:teuthology.orchestra.run.smithi003:mon.c> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.c.service 2024-03-21T13:58:27.483 DEBUG:teuthology.orchestra.run.smithi118:mon.b> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.b.service 2024-03-21T13:58:27.485 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T13:58:27.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T13:58:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:28.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:28.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:28.990 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:28 smithi003 ceph-mon[29576]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:28 smithi003 ceph-mon[29576]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b;count:3 2024-03-21T13:58:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:28 smithi003 ceph-mon[29576]: Deploying daemon mon.b on smithi118 2024-03-21T13:58:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:29 smithi003 ceph-mon[29576]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:30 smithi003 ceph-mon[29576]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:31.305 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:58:31.305 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","modified":"2024-03-21T13:56:51.623941Z","created":"2024-03-21T13:56:51.623941Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-21T13:58:31.307 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 1 2024-03-21T13:58:31.605 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:31 smithi118 systemd[1]: Starting Ceph mon.b for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T13:58:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:31 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/911605623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 podman[33843]: 2024-03-21 13:58:32.032076252 +0000 UTC m=+0.018956278 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 podman[33843]: 2024-03-21 13:58:32.131713447 +0000 UTC m=+0.118593447 container create ef71a106405ebb82651f494ade417813a017236a394315676da42399342aee89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 podman[33843]: 2024-03-21 13:58:32.32613784 +0000 UTC m=+0.313017864 container init ef71a106405ebb82651f494ade417813a017236a394315676da42399342aee89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 podman[33843]: 2024-03-21 13:58:32.328946176 +0000 UTC m=+0.315826177 container start ef71a106405ebb82651f494ade417813a017236a394315676da42399342aee89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev), process ceph-mon, pid 2 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: pidfile_write: ignore empty --pid-file 2024-03-21T13:58:32.401 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: load: jerasure load: lrc 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: RocksDB version: 7.9.2 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Git sha 0 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Compile date 2024-03-21 04:02:31 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: DB SUMMARY 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: DB Session ID: ZL4DEPOXWAZIELD8YJNI 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: CURRENT file: CURRENT 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.error_if_exists: 0 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.create_if_missing: 0 2024-03-21T13:58:32.402 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.paranoid_checks: 1 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.env: 0x55eb5ef84b20 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.info_log: 0x55eb6135f920 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.statistics: (nil) 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.use_fsync: 0 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_log_file_size: 0 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T13:58:32.403 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.allow_fallocate: 1 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.use_direct_reads: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.db_log_dir: 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.wal_dir: 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T13:58:32.404 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.write_buffer_manager: 0x55eb613301e0 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T13:58:32.405 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.unordered_write: 0 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T13:58:32.407 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.row_cache: None 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.wal_filter: None 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.two_write_queues: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.wal_compression: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.atomic_flush: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.log_readahead_size: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T13:58:32.408 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_background_jobs: 2 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_background_compactions: -1 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_subcompactions: 1 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T13:58:32.409 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_open_files: -1 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_background_flushes: -1 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Compression algorithms supported: 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kZSTD supported: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kXpressCompression supported: 0 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kZlibCompression supported: 1 2024-03-21T13:58:32.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kSnappyCompression supported: 1 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kLZ4Compression supported: 1 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: kBZip2Compression supported: 0 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.merge_operator: 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_filter: None 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T13:58:32.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eb6135fa20) 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: cache_index_and_filter_blocks: 1 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: pin_top_level_index_and_filter: 1 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: index_type: 0 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: data_block_index_type: 0 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: index_shortening: 1 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: checksum: 4 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: no_block_cache: 0 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_cache: 0x55eb61363090 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_cache_name: BinnedLRUCache 2024-03-21T13:58:32.412 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_cache_options: 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: capacity : 536870912 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: num_shard_bits : 4 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: strict_capacity_limit : 0 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: high_pri_pool_ratio: 0.000 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_cache_compressed: (nil) 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: persistent_cache: (nil) 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_size: 4096 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_size_deviation: 10 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_restart_interval: 16 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: index_block_restart_interval: 1 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: metadata_block_size: 4096 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: partition_filters: 0 2024-03-21T13:58:32.413 INFO:journalctl@ceph.mon.b.smithi118.stdout: use_delta_encoding: 1 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: filter_policy: bloomfilter 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: whole_key_filtering: 1 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: verify_compression: 0 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: read_amp_bytes_per_bit: 0 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: format_version: 5 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: enable_index_compression: 1 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: block_align: 0 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: max_auto_readahead_size: 262144 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: prepopulate_block_cache: 0 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: initial_auto_readahead_size: 8192 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T13:58:32.414 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression: NoCompression 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.num_levels: 7 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:58:32.415 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:58:32.416 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T13:58:32.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T13:58:32.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.table_properties_collectors: 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.inplace_update_support: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.bloom_locality: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.max_successive_merges: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T13:58:32.419 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.ttl: 2592000 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.enable_blob_files: false 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.min_blob_size: 0 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T13:58:32.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2d10e918-f9ca-4817-aa10-5b33b99cf9e8 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029512358281, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029512358678, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711029512, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2d10e918-f9ca-4817-aa10-5b33b99cf9e8", "db_session_id": "ZL4DEPOXWAZIELD8YJNI", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029512358758, "job": 1, "event": "recovery_finished"} 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eb61452000 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: DB pointer 0x55eb6143c000 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T13:58:32.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: ** DB Stats ** 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: ** Compaction Stats [default] ** 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:32.422 INFO:journalctl@ceph.mon.b.smithi118.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: ** Compaction Stats [default] ** 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T13:58:32.423 INFO:journalctl@ceph.mon.b.smithi118.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: Block cache BinnedLRUCache@0x55eb61363090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: using public_addr v2:172.21.15.118:0/0 -> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: starting mon.b rank -1 at public addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] at bind addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(???) e0 preinit fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).mds e1 new map 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-21T13:58:32.424 INFO:journalctl@ceph.mon.b.smithi118.stdout: e1 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout: legacy client fscid: -1 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout: 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout: No filesystems configured 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:58:32.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1347494758' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T13:58:32.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1347494758' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:32.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b;count:3 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: Deploying daemon mon.b on smithi118 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/911605623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 ceph-mon[33862]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T13:58:32.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 bash[33843]: ef71a106405ebb82651f494ade417813a017236a394315676da42399342aee89 2024-03-21T13:58:32.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:32 smithi118 systemd[1]: Started Ceph mon.b for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T13:58:32.903 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T13:58:32.904 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T13:58:33.138 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:58:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: mon.a calling monitor election 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: mon.b calling monitor election 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: fsmap 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: mgrmap e13: y(active, since 40s) 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: overall HEALTH_OK 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:37.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:37.894 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:58:37.894 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":2,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","modified":"2024-03-21T13:58:32.380130Z","created":"2024-03-21T13:56:51.623941Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-21T13:58:37.894 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:58:37.894 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: mon.a calling monitor election 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: mon.b calling monitor election 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: fsmap 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:37.895 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: mgrmap e13: y(active, since 40s) 2024-03-21T13:58:37.896 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: overall HEALTH_OK 2024-03-21T13:58:37.896 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:37.896 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:37.896 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:37.896 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 2 2024-03-21T13:58:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:38 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:38 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:38 smithi003 ceph-mon[29576]: Deploying daemon mon.c on smithi003 2024-03-21T13:58:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:38 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/329083272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:38 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:38 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:38 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:38 smithi118 ceph-mon[33862]: Deploying daemon mon.c on smithi003 2024-03-21T13:58:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:38 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/329083272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:38 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:39.440 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-21T13:58:39.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-21T13:58:39.686 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:58:39.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:39 smithi003 ceph-mon[29576]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:39.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 13:58:39 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T13:58:39.379+0000 7f640a038700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-21T13:58:39.797 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:39 smithi118 ceph-mon[33862]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 systemd[1]: Starting Ceph mon.c for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T13:58:40.802 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:40 smithi118 ceph-mon[33862]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.845 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[29576]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 podman[40491]: 2024-03-21 13:58:40.490064718 +0000 UTC m=+0.018287186 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T13:58:40.845 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 podman[40491]: 2024-03-21 13:58:40.604368723 +0000 UTC m=+0.132591232 container create ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 podman[40491]: 2024-03-21 13:58:40.798485054 +0000 UTC m=+0.326707535 container init ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 podman[40491]: 2024-03-21 13:58:40.801357876 +0000 UTC m=+0.329580343 container start ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: set uid:gid to 167:167 (ceph:ceph) 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: ceph version 19.0.0-2309-gba760091 (ba760091cd7bd2b0d23f4825ac856ba66450e988) squid (dev), process ceph-mon, pid 2 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pidfile_write: ignore empty --pid-file 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: load: jerasure load: lrc 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: RocksDB version: 7.9.2 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Git sha 0 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Compile date 2024-03-21 04:02:31 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: DB SUMMARY 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: DB Session ID: X1LX1BMZ68LAPJAT5TXQ 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: CURRENT file: CURRENT 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: IDENTITY file: IDENTITY 2024-03-21T13:58:40.846 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-21T13:58:40.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-21T13:58:40.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.error_if_exists: 0 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.create_if_missing: 0 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.paranoid_checks: 1 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.env: 0x562c181e5b20 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.fs: PosixFileSystem 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.info_log: 0x562c1a439920 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_file_opening_threads: 16 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.statistics: (nil) 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.use_fsync: 0 2024-03-21T13:58:40.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_log_file_size: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.keep_log_file_num: 1000 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.recycle_log_file_num: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.allow_fallocate: 1 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.allow_mmap_reads: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.allow_mmap_writes: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.use_direct_reads: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.create_missing_column_families: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.db_log_dir: 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.wal_dir: 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-21T13:58:40.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.advise_random_on_open: 1 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.db_write_buffer_size: 0 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.write_buffer_manager: 0x562c1a40a1e0 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.rate_limiter: (nil) 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.wal_recovery_mode: 2 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.enable_thread_tracking: 0 2024-03-21T13:58:40.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.enable_pipelined_write: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.unordered_write: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.row_cache: None 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.wal_filter: None 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.allow_ingest_behind: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.two_write_queues: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.manual_wal_flush: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.wal_compression: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.atomic_flush: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-21T13:58:40.851 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.log_readahead_size: 0 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.best_efforts_recovery: 0 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-21T13:58:40.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.allow_data_in_errors: 0 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.db_host_id: __hostname__ 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_background_jobs: 2 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_background_compactions: -1 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_subcompactions: 1 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_total_wal_size: 0 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-21T13:58:40.853 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_open_files: -1 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bytes_per_sync: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_readahead_size: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_background_flushes: -1 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Compression algorithms supported: 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kZSTD supported: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kXpressCompression supported: 0 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kLZ4HCCompression supported: 1 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kZlibCompression supported: 1 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kSnappyCompression supported: 1 2024-03-21T13:58:40.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kLZ4Compression supported: 1 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: kBZip2Compression supported: 0 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.merge_operator: 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_filter: None 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_filter_factory: None 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.sst_partitioner_factory: None 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562c1a439a20) 2024-03-21T13:58:40.855 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_type: 0 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_index_type: 0 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_shortening: 1 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: checksum: 4 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: no_block_cache: 0 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache: 0x562c1a43d090 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_options: 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: capacity : 536870912 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_shard_bits : 4 2024-03-21T13:58:40.856 INFO:journalctl@ceph.mon.c.smithi003.stdout: strict_capacity_limit : 0 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_cache_compressed: (nil) 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: persistent_cache: (nil) 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size: 4096 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_size_deviation: 10 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_restart_interval: 16 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: index_block_restart_interval: 1 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: metadata_block_size: 4096 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: partition_filters: 0 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: use_delta_encoding: 1 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: filter_policy: bloomfilter 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: whole_key_filtering: 1 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: verify_compression: 0 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-03-21T13:58:40.857 INFO:journalctl@ceph.mon.c.smithi003.stdout: format_version: 5 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_index_compression: 1 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout: block_align: 0 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout: max_auto_readahead_size: 262144 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout: prepopulate_block_cache: 0 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout: initial_auto_readahead_size: 8192 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.write_buffer_size: 33554432 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_write_buffer_number: 2 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression: NoCompression 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression: Disabled 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.prefix_extractor: nullptr 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.num_levels: 7 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-21T13:58:40.858 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.level: 32767 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.strategy: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-21T13:58:40.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.enabled: false 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.target_file_size_base: 67108864 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-21T13:58:40.860 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.arena_block_size: 1048576 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.disable_auto_compactions: 0 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-21T13:58:40.861 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.table_properties_collectors: 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.inplace_update_support: 0 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-21T13:58:40.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.bloom_locality: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.max_successive_merges: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.paranoid_file_checks: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.force_consistency_checks: 1 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.report_bg_io_stats: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.ttl: 2592000 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-21T13:58:40.864 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.enable_blob_files: false 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.min_blob_size: 0 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.blob_file_size: 268435456 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.blob_file_starting_level: 0 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5a26f7e8-fd5a-4808-beb9-059a6b1da238 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029520830293, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-21T13:58:40.865 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029520830687, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711029520, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5a26f7e8-fd5a-4808-beb9-059a6b1da238", "db_session_id": "X1LX1BMZ68LAPJAT5TXQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711029520830768, "job": 1, "event": "recovery_finished"} 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562c1a52c000 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: DB pointer 0x562c1a516000 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** DB Stats ** 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:58:40.866 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** Compaction Stats [default] ** 2024-03-21T13:58:40.867 INFO:journalctl@ceph.mon.c.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-21T13:58:40.868 INFO:journalctl@ceph.mon.c.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: Block cache BinnedLRUCache@0x562c1a43d090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: using public_addrv [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: starting mon.c rank -1 at public addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] at bind addrs [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(???) e0 preinit fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).mds e1 new map 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: e1 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: legacy client fscid: -1 2024-03-21T13:58:40.869 INFO:journalctl@ceph.mon.c.smithi003.stdout: 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout: No filesystems configured 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T13:58:40.870 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1347494758' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1347494758' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-21T13:58:40.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:40.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi003:[v2:172.21.15.3:3301,v1:172.21.15.3:6790]=c;smithi118:172.21.15.118=b;count:3 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Deploying daemon mon.b on smithi118 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/911605623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.a calling monitor election 2024-03-21T13:58:40.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.b calling monitor election 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: fsmap 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:40.874 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mgrmap e13: y(active, since 40s) 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: overall HEALTH_OK 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: Deploying daemon mon.c on smithi003 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/329083272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:40.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 ceph-mon[40510]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-21T13:58:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 bash[40491]: ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 2024-03-21T13:58:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:40 smithi003 systemd[1]: Started Ceph mon.c for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: mon.c calling monitor election 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: mon.c calling monitor election 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: mon.b calling monitor election 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: mon.a calling monitor election 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:58:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: fsmap 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: mgrmap e13: y(active, since 49s) 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: overall HEALTH_OK 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:58:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: mon.c calling monitor election 2024-03-21T13:58:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: mon.c calling monitor election 2024-03-21T13:58:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: mon.b calling monitor election 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: mon.a calling monitor election 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: fsmap 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: mgrmap e13: y(active, since 49s) 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: overall HEALTH_OK 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: mon.c calling monitor election 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: mon.c calling monitor election 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: mon.b calling monitor election 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: mon.a calling monitor election 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],c=[v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: fsmap 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: osdmap e4: 0 total, 0 up, 0 in 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: mgrmap e13: y(active, since 49s) 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T13:58:47.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: overall HEALTH_OK 2024-03-21T13:58:47.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:47.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:47.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:47.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:47.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T13:58:47.355 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-21T13:58:47.355 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":3,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","modified":"2024-03-21T13:58:40.851234Z","created":"2024-03-21T13:56:51.623941Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3301","nonce":0},{"type":"v1","addr":"172.21.15.3:6790","nonce":0}]},"addr":"172.21.15.3:6790/0","public_addr":"172.21.15.3:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-21T13:58:47.358 INFO:teuthology.orchestra.run.smithi118.stderr:dumped monmap epoch 3 2024-03-21T13:58:48.039 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-21T13:58:48.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-21T13:58:48.281 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:48.403 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/3956052410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/3956052410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:48.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:48.407 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/3956052410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:48.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:48.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:48.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:49 smithi118 ceph-mon[33862]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T13:58:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:49 smithi118 ceph-mon[33862]: Reconfiguring daemon mon.a on smithi003 2024-03-21T13:58:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:49 smithi118 ceph-mon[33862]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:49.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:49 smithi003 ceph-mon[29576]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T13:58:49.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:49 smithi003 ceph-mon[29576]: Reconfiguring daemon mon.a on smithi003 2024-03-21T13:58:49.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:49 smithi003 ceph-mon[29576]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:49.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:49 smithi003 ceph-mon[40510]: Reconfiguring mon.a (unknown last config time)... 2024-03-21T13:58:49.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:49 smithi003 ceph-mon[40510]: Reconfiguring daemon mon.a on smithi003 2024-03-21T13:58:49.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:49 smithi003 ceph-mon[40510]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:50.249 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:50.249 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-03-21T13:58:50.249 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T13:58:50.249 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] [v2:172.21.15.3:3301/0,v1:172.21.15.3:6790/0] 2024-03-21T13:58:50.440 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1605296264' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:50.441 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1605296264' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1605296264' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:51.008 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-21T13:58:51.008 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:51.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T13:58:51.041 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:51.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:51.109 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:58:51.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.conf 2024-03-21T13:58:51.143 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:58:51.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-21T13:58:51.213 INFO:tasks.cephadm:Adding mgr.y on smithi003 2024-03-21T13:58:51.213 INFO:tasks.cephadm:Adding mgr.x on smithi118 2024-03-21T13:58:51.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi003=y;smithi118=x' 2024-03-21T13:58:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:51 smithi118 ceph-mon[33862]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:51.489 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T13:58:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:51 smithi003 ceph-mon[29576]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:51 smithi003 ceph-mon[40510]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:52.745 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:52.745 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:52.746 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: Reconfiguring mon.c (monmap changed)... 2024-03-21T13:58:52.746 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:52.746 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:52.746 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:52.746 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:52 smithi118 ceph-mon[33862]: Reconfiguring daemon mon.c on smithi003 2024-03-21T13:58:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: Reconfiguring mon.c (monmap changed)... 2024-03-21T13:58:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[29576]: Reconfiguring daemon mon.c on smithi003 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: Reconfiguring mon.c (monmap changed)... 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:52 smithi003 ceph-mon[40510]: Reconfiguring daemon mon.c on smithi003 2024-03-21T13:58:53.220 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled mgr update... 2024-03-21T13:58:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:53.792 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.793 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.793 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:53.793 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:53.793 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:53.793 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.802 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:53.802 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.803 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.803 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-21T13:58:53.803 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-21T13:58:53.803 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:53.803 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:53.827 DEBUG:teuthology.orchestra.run.smithi118:mgr.x> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.x.service 2024-03-21T13:58:53.829 INFO:tasks.cephadm:Deploying OSDs... 2024-03-21T13:58:53.829 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T13:58:53.829 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T13:58:53.848 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T13:58:53.848 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-03-21T13:58:53.906 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-21 13:58:40.307504300 +0000 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-21 13:53:19.860667436 +0000 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-21 13:53:19.860667436 +0000 2024-03-21T13:58:53.907 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-21 13:53:19.860667436 +0000 2024-03-21T13:58:53.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T13:58:53.977 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-21T13:58:53.978 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-21T13:58:53.978 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000217237 s, 2.4 MB/s 2024-03-21T13:58:53.979 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T13:58:54.037 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-03-21T13:58:54.093 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-21 13:58:40.307504300 +0000 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-21 13:53:20.210662879 +0000 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-21 13:53:20.210662879 +0000 2024-03-21T13:58:54.094 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-21 13:53:20.210662879 +0000 2024-03-21T13:58:54.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T13:58:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-21T13:58:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-21T13:58:54.163 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000249432 s, 2.1 MB/s 2024-03-21T13:58:54.164 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T13:58:54.222 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-03-21T13:58:54.279 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-21 13:58:40.308504287 +0000 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-21 13:53:20.558658349 +0000 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-21 13:53:20.558658349 +0000 2024-03-21T13:58:54.280 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-21 13:53:20.558658349 +0000 2024-03-21T13:58:54.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T13:58:54.349 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-21T13:58:54.349 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-21T13:58:54.349 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000209958 s, 2.4 MB/s 2024-03-21T13:58:54.350 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T13:58:54.407 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-03-21 13:58:40.308504287 +0000 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-03-21 13:53:20.893653987 +0000 2024-03-21T13:58:54.463 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-03-21 13:53:20.893653987 +0000 2024-03-21T13:58:54.464 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-03-21 13:53:20.893653987 +0000 2024-03-21T13:58:54.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T13:58:54.530 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-03-21T13:58:54.530 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-03-21T13:58:54.530 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000204242 s, 2.5 MB/s 2024-03-21T13:58:54.531 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T13:58:54.587 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T13:58:54.587 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-03-21T13:58:54.603 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-21T13:58:54.603 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-03-21T13:58:54.658 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-21 13:58:31.294454053 +0000 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-21 13:53:16.605071048 +0000 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-21 13:53:16.605071048 +0000 2024-03-21T13:58:54.659 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-21 13:53:16.605071048 +0000 2024-03-21T13:58:54.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-21T13:58:54.724 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-21T13:58:54.725 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-21T13:58:54.725 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 8.4535e-05 s, 6.1 MB/s 2024-03-21T13:58:54.726 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-21T13:58:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[40510]: Reconfiguring mon.b (monmap changed)... 2024-03-21T13:58:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[40510]: Reconfiguring daemon mon.b on smithi118 2024-03-21T13:58:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[40510]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi118=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[40510]: Saving service mgr spec with placement smithi003=y;smithi118=x;count:2 2024-03-21T13:58:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[29576]: Reconfiguring mon.b (monmap changed)... 2024-03-21T13:58:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[29576]: Reconfiguring daemon mon.b on smithi118 2024-03-21T13:58:54.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[29576]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi118=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:54.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:54 smithi003 ceph-mon[29576]: Saving service mgr spec with placement smithi003=y;smithi118=x;count:2 2024-03-21T13:58:54.782 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-21 13:58:31.294454053 +0000 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-21 13:53:16.927063531 +0000 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-21 13:53:16.927063531 +0000 2024-03-21T13:58:54.837 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-21 13:53:16.927063531 +0000 2024-03-21T13:58:54.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-21T13:58:54.892 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:54 smithi118 ceph-mon[33862]: Reconfiguring mon.b (monmap changed)... 2024-03-21T13:58:54.892 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:54 smithi118 ceph-mon[33862]: Reconfiguring daemon mon.b on smithi118 2024-03-21T13:58:54.892 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:54 smithi118 ceph-mon[33862]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi003=y;smithi118=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:58:54.892 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:54 smithi118 ceph-mon[33862]: Saving service mgr spec with placement smithi003=y;smithi118=x;count:2 2024-03-21T13:58:54.904 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-21T13:58:54.904 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-21T13:58:54.904 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000130723 s, 3.9 MB/s 2024-03-21T13:58:54.905 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-21T13:58:54.961 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-21 13:58:31.294454053 +0000 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-21 13:53:17.264055664 +0000 2024-03-21T13:58:55.018 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-21 13:53:17.264055664 +0000 2024-03-21T13:58:55.019 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-21 13:53:17.264055664 +0000 2024-03-21T13:58:55.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-21T13:58:55.085 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-21T13:58:55.085 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-21T13:58:55.085 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.2017e-05 s, 5.6 MB/s 2024-03-21T13:58:55.086 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-21T13:58:55.142 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-21 13:58:31.295454030 +0000 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-21 13:53:17.574048428 +0000 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-21 13:53:17.574048428 +0000 2024-03-21T13:58:55.198 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-21 13:53:17.574048428 +0000 2024-03-21T13:58:55.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-21T13:58:55.263 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-21T13:58:55.264 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-21T13:58:55.264 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 9.0744e-05 s, 5.6 MB/s 2024-03-21T13:58:55.265 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-21T13:58:55.320 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/vg_nvme/lv_4... 2024-03-21T13:58:55.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T13:58:55.572 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:55.588 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:55.588 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.588 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.588 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:55.588 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:55.589 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:55.589 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.589 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:58:55.589 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T13:58:55.589 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:58:55.589 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:55.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:55.694 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:58:55.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:56.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:56 smithi118 ceph-mon[33862]: Deploying daemon mgr.x on smithi118 2024-03-21T13:58:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:56 smithi003 ceph-mon[29576]: Deploying daemon mgr.x on smithi118 2024-03-21T13:58:56.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:56 smithi003 ceph-mon[40510]: Deploying daemon mgr.x on smithi118 2024-03-21T13:58:56.931 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:56 smithi118 systemd[1]: Starting Ceph mgr.x for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T13:58:57.254 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 podman[35696]: 2024-03-21 13:58:56.904241354 +0000 UTC m=+0.018934013 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T13:58:57.254 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 podman[35696]: 2024-03-21 13:58:57.035427625 +0000 UTC m=+0.150120272 container create 8b7dde2a0e5ead6dff5bb9b34e96b355b356c2e7fc712fa25896ac14a83d0f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009) 2024-03-21T13:58:57.254 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 podman[35696]: 2024-03-21 13:58:57.254521348 +0000 UTC m=+0.369213995 container init 8b7dde2a0e5ead6dff5bb9b34e96b355b356c2e7fc712fa25896ac14a83d0f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009) 2024-03-21T13:58:57.508 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 podman[35696]: 2024-03-21 13:58:57.257483378 +0000 UTC m=+0.372176033 container start 8b7dde2a0e5ead6dff5bb9b34e96b355b356c2e7fc712fa25896ac14a83d0f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T13:58:57.508 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-mgr[35715]: -- 172.21.15.118:0/3389163357 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556715c7e5a0 con 0x556715c52800 2024-03-21T13:58:57.508 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 bash[35696]: 8b7dde2a0e5ead6dff5bb9b34e96b355b356c2e7fc712fa25896ac14a83d0f82 2024-03-21T13:58:57.508 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 systemd[1]: Started Ceph mgr.x for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T13:58:57.629 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T13:58:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T13:58:57.630 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-03-21T13:58:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-03-21T13:58:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113988 s, 920 MB/s 2024-03-21T13:58:57.630 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-03-21T13:58:57.832 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-mon[33862]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:57.832 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.832 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.832 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.833 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[29576]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[40510]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:57.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:58.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:57.832+0000 7fc93df6e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T13:58:58.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:57.929+0000 7fc93df6e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T13:58:58.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_4 2024-03-21T13:58:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:58:58.613 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:58.818 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:58.818 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:58:58.922 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:58 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:58.612+0000 7fc93df6e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T13:58:58.922 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:58 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:58.708+0000 7fc93df6e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T13:58:58.922 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:58 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:58.856+0000 7fc93df6e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T13:58:59.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:58 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:58.967+0000 7fc93df6e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T13:58:59.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:59.065+0000 7fc93df6e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T13:58:59.644 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:59.348+0000 7fc93df6e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T13:58:59.644 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:59.459+0000 7fc93df6e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T13:58:59.644 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:59.547+0000 7fc93df6e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T13:58:59.644 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:59.643+0000 7fc93df6e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:58:59.645 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:59.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:59.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:58:59.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-21T13:58:59.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-21T13:58:59.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:58:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:58:59.897 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:58:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:58:59.818+0000 7fc93df6e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T13:59:00.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:00 smithi003 ceph-mon[40510]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T13:59:00.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:00 smithi003 ceph-mon[40510]: Reconfiguring daemon mgr.y on smithi003 2024-03-21T13:59:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:00 smithi003 ceph-mon[29576]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T13:59:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:00 smithi003 ceph-mon[29576]: Reconfiguring daemon mgr.y on smithi003 2024-03-21T13:59:00.780 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:00 smithi118 ceph-mon[33862]: Reconfiguring mgr.y (unknown last config time)... 2024-03-21T13:59:00.780 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:00 smithi118 ceph-mon[33862]: Reconfiguring daemon mgr.y on smithi003 2024-03-21T13:59:01.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:00 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:00.780+0000 7fc93df6e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T13:59:01.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:00 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:00.908+0000 7fc93df6e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T13:59:01.513 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:01.234+0000 7fc93df6e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T13:59:01.513 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:01.322+0000 7fc93df6e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T13:59:01.513 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:01.512+0000 7fc93df6e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:01.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:01.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:01.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:01.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:01.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:01.710 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:01.692+0000 7fc93df6e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:02.227 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:01 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:01.973+0000 7fc93df6e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T13:59:02.227 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:02.226+0000 7fc93df6e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T13:59:02.524 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:02.460+0000 7fc93df6e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T13:59:02.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[29576]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:02.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:02.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:02.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:02.626 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:02.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[40510]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:02.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:02.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:02.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:02.627 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-mon[33862]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:03.532 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:03.254+0000 7fc93df6e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T13:59:03.841 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-mon[33862]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:03.841 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4199589761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb"}]: dispatch 2024-03-21T13:59:03.841 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4199589761' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb"}]': finished 2024-03-21T13:59:03.841 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-mon[33862]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T13:59:03.841 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[29576]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4199589761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb"}]: dispatch 2024-03-21T13:59:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4199589761' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb"}]': finished 2024-03-21T13:59:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[29576]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T13:59:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[40510]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4199589761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb"}]: dispatch 2024-03-21T13:59:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4199589761' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb"}]': finished 2024-03-21T13:59:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[40510]: osdmap e5: 1 total, 0 up, 1 in 2024-03-21T13:59:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:03 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:04.153 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:03.840+0000 7fc93df6e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T13:59:04.154 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:03 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:03.938+0000 7fc93df6e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T13:59:04.154 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T13:59:04.153+0000 7fc93df6e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T13:59:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2597601883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-mon[33862]: Standby manager daemon x started 2024-03-21T13:59:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:59:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:59:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:59:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:04 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:59:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2597601883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[40510]: Standby manager daemon x started 2024-03-21T13:59:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:59:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:59:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:59:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:59:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2597601883' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[29576]: Standby manager daemon x started 2024-03-21T13:59:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T13:59:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T13:59:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T13:59:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:04 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/151976260' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T13:59:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:05 smithi118 ceph-mon[33862]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-21T13:59:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:59:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:05 smithi118 ceph-mon[33862]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:05 smithi003 ceph-mon[29576]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-21T13:59:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:59:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:05 smithi003 ceph-mon[29576]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:05 smithi003 ceph-mon[40510]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-21T13:59:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T13:59:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:05 smithi003 ceph-mon[40510]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:06 smithi118 ceph-mon[33862]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:07.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:06 smithi003 ceph-mon[29576]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:07.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:06 smithi003 ceph-mon[40510]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:08.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:08 smithi003 ceph-mon[29576]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:08.929 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:08 smithi003 ceph-mon[40510]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:08 smithi118 ceph-mon[33862]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:10.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:10 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T13:59:10.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:10 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:10.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:10 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T13:59:10.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:10 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:10 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-21T13:59:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:10 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:11.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:11 smithi003 ceph-mon[29576]: Deploying daemon osd.0 on smithi003 2024-03-21T13:59:11.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:11 smithi003 ceph-mon[29576]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:11.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:11 smithi003 ceph-mon[40510]: Deploying daemon osd.0 on smithi003 2024-03-21T13:59:11.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:11 smithi003 ceph-mon[40510]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:11 smithi118 ceph-mon[33862]: Deploying daemon osd.0 on smithi003 2024-03-21T13:59:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:11 smithi118 ceph-mon[33862]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:12 smithi118 ceph-mon[33862]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:12.969 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:12 smithi003 ceph-mon[29576]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:12.970 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:12 smithi003 ceph-mon[40510]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:15.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:15 smithi003 ceph-mon[29576]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:15.335 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:15 smithi003 ceph-mon[40510]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:15 smithi118 ceph-mon[33862]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:16.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:16 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:16.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:16 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:16.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:16 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:16.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:16 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:16.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:16 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:16.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:16 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:16 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:16 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:16 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:17.594 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:17 smithi003 ceph-mon[29576]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:17.594 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:17 smithi003 ceph-mon[40510]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:17 smithi118 ceph-mon[33862]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:19.125 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='osd.0 v2:172.21.15.3:6801/930096433' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:59:19.420 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='osd.0 v2:172.21.15.3:6801/930096433' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.421 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.422 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='osd.0 v2:172.21.15.3:6801/930096433' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:59:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-21T13:59:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:19.718 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.0.service 2024-03-21T13:59:19.720 INFO:tasks.cephadm:Deploying osd.1 on smithi003 with /dev/vg_nvme/lv_3... 2024-03-21T13:59:19.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T13:59:20.392 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:59:20.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[29576]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T13:59:20.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[29576]: from='osd.0 v2:172.21.15.3:6801/930096433' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:20.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[29576]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T13:59:20.551 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:20.551 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[29576]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:20.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[40510]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T13:59:20.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[40510]: from='osd.0 v2:172.21.15.3:6801/930096433' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:20.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[40510]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T13:59:20.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:20.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-mon[40510]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:20 smithi118 ceph-mon[33862]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-21T13:59:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:20 smithi118 ceph-mon[33862]: from='osd.0 v2:172.21.15.3:6801/930096433' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:20 smithi118 ceph-mon[33862]: osdmap e6: 1 total, 0 up, 1 in 2024-03-21T13:59:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:20 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:20 smithi118 ceph-mon[33862]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:20.845 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0[44524]: 2024-03-21T13:59:20.691+0000 7fa42f515700 -1 osd.0 0 waiting for initial osdmap 2024-03-21T13:59:20.845 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 13:59:20 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0[44524]: 2024-03-21T13:59:20.693+0000 7fa427300700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: from='osd.0 ' entity='osd.0' 2024-03-21T13:59:21.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: from='osd.0 ' entity='osd.0' 2024-03-21T13:59:21.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: osdmap e7: 1 total, 0 up, 1 in 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: from='osd.0 ' entity='osd.0' 2024-03-21T13:59:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T13:59:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T13:59:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[40510]: osd.0 v2:172.21.15.3:6801/930096433 boot 2024-03-21T13:59:22.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[40510]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T13:59:22.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T13:59:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T13:59:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[29576]: osd.0 v2:172.21.15.3:6801/930096433 boot 2024-03-21T13:59:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[29576]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T13:59:22.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:22 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:22 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T13:59:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:22 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T13:59:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:22 smithi118 ceph-mon[33862]: osd.0 v2:172.21.15.3:6801/930096433 boot 2024-03-21T13:59:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:22 smithi118 ceph-mon[33862]: osdmap e8: 1 total, 1 up, 1 in 2024-03-21T13:59:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:22 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T13:59:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T13:59:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T13:59:22.721 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-03-21T13:59:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-03-21T13:59:22.721 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129788 s, 808 MB/s 2024-03-21T13:59:22.721 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-03-21T13:59:23.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:23 smithi003 ceph-mon[29576]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:23.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:23 smithi003 ceph-mon[40510]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:23.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_3 2024-03-21T13:59:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:23 smithi118 ceph-mon[33862]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:59:24.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.335 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:24.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:24.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:24.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:24 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: osdmap e9: 1 total, 1 up, 1 in 2024-03-21T13:59:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:24.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:24.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:24 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:25.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:25 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi003 2024-03-21T13:59:25.397 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:25 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi003 to 16069M 2024-03-21T13:59:25.398 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:25 smithi003 ceph-mon[29576]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:25.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:25 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi003 2024-03-21T13:59:25.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:25 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi003 to 16069M 2024-03-21T13:59:25.398 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:25 smithi003 ceph-mon[40510]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:25 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi003 2024-03-21T13:59:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:25 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi003 to 16069M 2024-03-21T13:59:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:25 smithi118 ceph-mon[33862]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:26.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:26.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:26.611 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:26.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:26.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:26.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:27 smithi003 ceph-mon[29576]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:27 smithi003 ceph-mon[29576]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:27.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:27 smithi003 ceph-mon[40510]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:27.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:27 smithi003 ceph-mon[40510]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:27 smithi118 ceph-mon[33862]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:27 smithi118 ceph-mon[33862]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:28 smithi118 ceph-mon[33862]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:28 smithi003 ceph-mon[29576]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:29.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:28 smithi003 ceph-mon[40510]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1139414221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d"}]: dispatch 2024-03-21T13:59:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1139414221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d"}]': finished 2024-03-21T13:59:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:29 smithi118 ceph-mon[33862]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T13:59:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3348053240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1139414221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d"}]: dispatch 2024-03-21T13:59:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1139414221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d"}]': finished 2024-03-21T13:59:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[29576]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T13:59:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3348053240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1139414221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d"}]: dispatch 2024-03-21T13:59:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1139414221' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d"}]': finished 2024-03-21T13:59:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[40510]: osdmap e10: 2 total, 1 up, 2 in 2024-03-21T13:59:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3348053240' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:30 smithi118 ceph-mon[33862]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:31.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:30 smithi003 ceph-mon[29576]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:31.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:30 smithi003 ceph-mon[40510]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:32 smithi118 ceph-mon[33862]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:32.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:32 smithi003 ceph-mon[29576]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:32.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:32 smithi003 ceph-mon[40510]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:35.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:35 smithi003 ceph-mon[40510]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:35.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T13:59:35.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:35 smithi003 ceph-mon[29576]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T13:59:35.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:35 smithi118 ceph-mon[33862]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-21T13:59:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:36 smithi118 ceph-mon[33862]: Deploying daemon osd.1 on smithi003 2024-03-21T13:59:36.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:36 smithi003 ceph-mon[29576]: Deploying daemon osd.1 on smithi003 2024-03-21T13:59:36.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:36 smithi003 ceph-mon[40510]: Deploying daemon osd.1 on smithi003 2024-03-21T13:59:37.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:37 smithi003 ceph-mon[29576]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:37.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:37 smithi003 ceph-mon[40510]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:37 smithi118 ceph-mon[33862]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:38.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:38 smithi003 ceph-mon[29576]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:38.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:38 smithi003 ceph-mon[40510]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:38 smithi118 ceph-mon[33862]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:40.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:40 smithi118 ceph-mon[33862]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:40.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:40 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:40 smithi003 ceph-mon[40510]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:40 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:40 smithi003 ceph-mon[29576]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:40 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:42.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:42.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:42 smithi118 ceph-mon[33862]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:42 smithi003 ceph-mon[29576]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:42 smithi003 ceph-mon[40510]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:44.487 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 1 on host 'smithi003' 2024-03-21T13:59:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:59:44.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:44.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:44 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:45.072 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:45.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:44 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T13:59:45.077 DEBUG:teuthology.orchestra.run.smithi003:osd.1> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.1.service 2024-03-21T13:59:45.079 INFO:tasks.cephadm:Deploying osd.2 on smithi003 with /dev/vg_nvme/lv_2... 2024-03-21T13:59:45.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T13:59:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:45 smithi118 ceph-mon[33862]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:45 smithi118 ceph-mon[33862]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:59:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:45 smithi118 ceph-mon[33862]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T13:59:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:45 smithi118 ceph-mon[33862]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:45 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[29576]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[29576]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:59:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[29576]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T13:59:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[29576]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[40510]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[40510]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-21T13:59:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[40510]: osdmap e11: 2 total, 1 up, 2 in 2024-03-21T13:59:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[40510]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T13:59:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:45 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.266 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1[50259]: 2024-03-21T13:59:46.044+0000 7f88e9998700 -1 osd.1 0 waiting for initial osdmap 2024-03-21T13:59:46.266 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1[50259]: 2024-03-21T13:59:46.047+0000 7f88e0f82700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T13:59:46.351 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:59:46.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[29576]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[29576]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[29576]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[29576]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[40510]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[40510]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[40510]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' 2024-03-21T13:59:46.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:46 smithi003 ceph-mon[40510]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:46 smithi118 ceph-mon[33862]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T13:59:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:46 smithi118 ceph-mon[33862]: osdmap e12: 2 total, 1 up, 2 in 2024-03-21T13:59:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:46 smithi118 ceph-mon[33862]: from='osd.1 v2:172.21.15.3:6805/1526485482' entity='osd.1' 2024-03-21T13:59:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:46 smithi118 ceph-mon[33862]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[29576]: osd.1 v2:172.21.15.3:6805/1526485482 boot 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[29576]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T13:59:47.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T13:59:47.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:47.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[40510]: osd.1 v2:172.21.15.3:6805/1526485482 boot 2024-03-21T13:59:47.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[40510]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T13:59:47.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:47 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:47 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:47 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:47 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:47 smithi118 ceph-mon[33862]: osd.1 v2:172.21.15.3:6805/1526485482 boot 2024-03-21T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:47 smithi118 ceph-mon[33862]: osdmap e13: 2 total, 2 up, 2 in 2024-03-21T13:59:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:47 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T13:59:49.308 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T13:59:49.308 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T13:59:49.308 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-03-21T13:59:49.308 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-03-21T13:59:49.308 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131024 s, 800 MB/s 2024-03-21T13:59:49.308 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-03-21T13:59:49.628 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi003 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi003 to 8034M 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:49.629 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[29576]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi003 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi003 to 8034M 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:49.630 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:49 smithi003 ceph-mon[40510]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T13:59:49.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi003 2024-03-21T13:59:49.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi003 to 8034M 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:49.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:49 smithi118 ceph-mon[33862]: osdmap e14: 2 total, 2 up, 2 in 2024-03-21T13:59:49.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_2 2024-03-21T13:59:50.204 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T13:59:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:50 smithi118 ceph-mon[33862]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:50 smithi003 ceph-mon[29576]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:51.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:50 smithi003 ceph-mon[40510]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:52.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:52.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:52.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:52.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T13:59:52.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T13:59:52.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T13:59:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:53 smithi003 ceph-mon[29576]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:53 smithi003 ceph-mon[29576]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:53 smithi003 ceph-mon[40510]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:53 smithi003 ceph-mon[40510]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:53 smithi118 ceph-mon[33862]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T13:59:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:53 smithi118 ceph-mon[33862]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:55.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:55 smithi118 ceph-mon[33862]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:55.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:55 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1779753239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c"}]: dispatch 2024-03-21T13:59:55.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:55 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1779753239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c"}]': finished 2024-03-21T13:59:55.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:55 smithi118 ceph-mon[33862]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T13:59:55.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:55 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:59:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[29576]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1779753239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c"}]: dispatch 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1779753239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c"}]': finished 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[29576]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[40510]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1779753239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c"}]: dispatch 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1779753239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c"}]': finished 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[40510]: osdmap e15: 3 total, 2 up, 3 in 2024-03-21T13:59:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:55 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T13:59:56.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1629245146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1629245146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:56.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1629245146' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T13:59:57.630 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:57 smithi003 ceph-mon[29576]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:57.631 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:57 smithi003 ceph-mon[40510]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:57 smithi118 ceph-mon[33862]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 13:59:58 smithi118 ceph-mon[33862]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:58.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 13:59:58 smithi003 ceph-mon[29576]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T13:59:58.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 13:59:58 smithi003 ceph-mon[40510]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:00.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:00 smithi003 ceph-mon[29576]: overall HEALTH_OK 2024-03-21T14:00:00.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:00 smithi003 ceph-mon[40510]: overall HEALTH_OK 2024-03-21T14:00:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:00 smithi118 ceph-mon[33862]: overall HEALTH_OK 2024-03-21T14:00:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:01 smithi003 ceph-mon[29576]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:01.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:01 smithi003 ceph-mon[40510]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:01 smithi118 ceph-mon[33862]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:02.210 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T14:00:02.210 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:02.210 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:02 smithi003 ceph-mon[29576]: Deploying daemon osd.2 on smithi003 2024-03-21T14:00:02.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T14:00:02.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:02.211 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:02 smithi003 ceph-mon[40510]: Deploying daemon osd.2 on smithi003 2024-03-21T14:00:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-21T14:00:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:02 smithi118 ceph-mon[33862]: Deploying daemon osd.2 on smithi003 2024-03-21T14:00:03.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:03 smithi003 ceph-mon[29576]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:03.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:03 smithi003 ceph-mon[40510]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:03 smithi118 ceph-mon[33862]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:04.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:04 smithi003 ceph-mon[40510]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:04.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:04 smithi003 ceph-mon[29576]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:04 smithi118 ceph-mon[33862]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:06.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:06 smithi118 ceph-mon[33862]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:07.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:06 smithi003 ceph-mon[29576]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:07.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:06 smithi003 ceph-mon[40510]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:08.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:08.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:08.899 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:08 smithi003 ceph-mon[29576]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:08.899 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:08 smithi003 ceph-mon[40510]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:09.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:08 smithi118 ceph-mon[33862]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:10.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:09 smithi003 ceph-mon[29576]: from='osd.2 v2:172.21.15.3:6809/3751378277' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:00:10.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:09 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:00:10.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:09 smithi003 ceph-mon[40510]: from='osd.2 v2:172.21.15.3:6809/3751378277' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:00:10.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:09 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:00:10.125 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 2 on host 'smithi003' 2024-03-21T14:00:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:09 smithi118 ceph-mon[33862]: from='osd.2 v2:172.21.15.3:6809/3751378277' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:00:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:09 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:00:10.749 DEBUG:teuthology.orchestra.run.smithi003:osd.2> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T14:00:10.752 INFO:tasks.cephadm:Deploying osd.3 on smithi003 with /dev/vg_nvme/lv_1... 2024-03-21T14:00:10.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T14:00:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T14:00:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T14:00:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: from='osd.2 v2:172.21.15.3:6809/3751378277' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[29576]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: from='osd.2 v2:172.21.15.3:6809/3751378277' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:10 smithi003 ceph-mon[40510]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:11.053 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: osdmap e16: 3 total, 2 up, 3 in 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: from='osd.2 v2:172.21.15.3:6809/3751378277' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:10 smithi118 ceph-mon[33862]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:11.504 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[56005]: 2024-03-21T14:00:11.309+0000 7f02c2523700 -1 osd.2 0 waiting for initial osdmap 2024-03-21T14:00:11.505 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[56005]: 2024-03-21T14:00:11.314+0000 7f02ba30e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:11 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' 2024-03-21T14:00:12.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: osdmap e17: 3 total, 2 up, 3 in 2024-03-21T14:00:12.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:12.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:12.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:12.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:11 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' 2024-03-21T14:00:12.925 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T14:00:12.925 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T14:00:12.925 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-03-21T14:00:12.925 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-03-21T14:00:12.925 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0147998 s, 709 MB/s 2024-03-21T14:00:12.925 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-03-21T14:00:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:12 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:12 smithi118 ceph-mon[33862]: osd.2 v2:172.21.15.3:6809/3751378277 boot 2024-03-21T14:00:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:12 smithi118 ceph-mon[33862]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T14:00:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:12 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:12 smithi118 ceph-mon[33862]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[29576]: osd.2 v2:172.21.15.3:6809/3751378277 boot 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[29576]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[29576]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[40510]: osd.2 v2:172.21.15.3:6809/3751378277 boot 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[40510]: osdmap e18: 3 total, 3 up, 3 in 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:00:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:12 smithi003 ceph-mon[40510]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-21T14:00:13.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_1 2024-03-21T14:00:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:13 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:00:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:13 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:00:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:13 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:00:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:13 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T14:00:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:13 smithi118 ceph-mon[33862]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T14:00:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:13 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:00:14.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:00:14.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:00:14.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[29576]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[40510]: osdmap e19: 3 total, 3 up, 3 in 2024-03-21T14:00:14.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:13 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:00:14.606 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:00:15.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:15 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:00:15.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:15 smithi003 ceph-mon[29576]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T14:00:15.410 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:15 smithi003 ceph-mon[29576]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:15.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:15 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:00:15.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:15 smithi003 ceph-mon[40510]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T14:00:15.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:15 smithi003 ceph-mon[40510]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:15.410 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[57616]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T14:00:15.410 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[57616]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:15.410 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[57616]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:15.410 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[57616]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:15 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:00:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:15 smithi118 ceph-mon[33862]: osdmap e20: 3 total, 3 up, 3 in 2024-03-21T14:00:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:15 smithi118 ceph-mon[33862]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:15.705 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[58729]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T14:00:15.705 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[58729]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:15.705 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[58729]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:15.705 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[58729]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:16.006 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[60208]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T14:00:16.006 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[60208]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:16.006 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[60208]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:16.007 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:15 smithi003 sudo[60208]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:16.309 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T14:00:16.309 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:16.309 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60212]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:16.309 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60212]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:16 smithi003 ceph-mon[29576]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T14:00:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:16 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:16.633 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:16 smithi003 ceph-mon[40510]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T14:00:16.633 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:16 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:16.634 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60217]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T14:00:16.634 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60217]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:16.634 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60217]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:16.634 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60217]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:16 smithi118 ceph-mon[33862]: osdmap e21: 3 total, 3 up, 3 in 2024-03-21T14:00:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:16 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:16.901 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60285]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T14:00:16.902 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60285]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:16.902 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60285]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:16.902 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60285]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:17.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60308]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T14:00:17.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60308]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:17.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60308]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:17.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:16 smithi003 sudo[60308]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60315]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T14:00:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60315]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60315]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60315]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:17.668 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36412]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T14:00:17.668 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36412]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:17.668 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36412]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:17.668 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36412]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 sudo[36415]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T14:00:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi003 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi003 to 5356M 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:17.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:17.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T14:00:17.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi003 2024-03-21T14:00:17.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi003 to 5356M 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:17.954 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi003 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.955 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi003 to 5356M 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:17.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60414]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60414]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:17.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60414]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:18.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:17 smithi003 sudo[60414]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:18.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 sudo[60463]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-21T14:00:18.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 sudo[60463]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-21T14:00:18.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 sudo[60463]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-21T14:00:18.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 sudo[60463]: pam_unix(sudo:session): session closed for user root 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:18.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:18.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:18.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:18.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:18.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:18.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:18 smithi118 ceph-mon[33862]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:18.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[29576]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:18.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-21T14:00:18.983 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:18 smithi003 ceph-mon[40510]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:20.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2892299565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c"}]: dispatch 2024-03-21T14:00:20.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2892299565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c"}]': finished 2024-03-21T14:00:20.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[40510]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T14:00:20.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2892299565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c"}]: dispatch 2024-03-21T14:00:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2892299565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c"}]': finished 2024-03-21T14:00:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[29576]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T14:00:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:20 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2892299565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c"}]: dispatch 2024-03-21T14:00:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2892299565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c"}]': finished 2024-03-21T14:00:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:20 smithi118 ceph-mon[33862]: osdmap e22: 4 total, 3 up, 4 in 2024-03-21T14:00:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:20 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:21 smithi003 ceph-mon[29576]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3984938612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:00:21.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:21 smithi003 ceph-mon[40510]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:21.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:21 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3984938612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:00:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:21 smithi118 ceph-mon[33862]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:21 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3984938612' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:00:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:22 smithi118 ceph-mon[33862]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:22 smithi003 ceph-mon[29576]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:22 smithi003 ceph-mon[40510]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:24 smithi118 ceph-mon[33862]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-21T14:00:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:24 smithi003 ceph-mon[29576]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-21T14:00:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:24 smithi003 ceph-mon[40510]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-21T14:00:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T14:00:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T14:00:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-21T14:00:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:27.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:27 smithi003 ceph-mon[29576]: Deploying daemon osd.3 on smithi003 2024-03-21T14:00:27.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:27 smithi003 ceph-mon[29576]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-21T14:00:27.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:27 smithi003 ceph-mon[40510]: Deploying daemon osd.3 on smithi003 2024-03-21T14:00:27.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:27 smithi003 ceph-mon[40510]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-21T14:00:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:27 smithi118 ceph-mon[33862]: Deploying daemon osd.3 on smithi003 2024-03-21T14:00:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:27 smithi118 ceph-mon[33862]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-21T14:00:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:29 smithi003 ceph-mon[29576]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-21T14:00:29.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:29 smithi003 ceph-mon[40510]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-21T14:00:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:29 smithi118 ceph-mon[33862]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-21T14:00:31.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:31 smithi003 ceph-mon[29576]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 75 KiB/s wr, 5 op/s 2024-03-21T14:00:31.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:31 smithi003 ceph-mon[40510]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 75 KiB/s wr, 5 op/s 2024-03-21T14:00:31.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:31 smithi118 ceph-mon[33862]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 75 KiB/s wr, 5 op/s 2024-03-21T14:00:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[29576]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T14:00:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:33 smithi003 ceph-mon[40510]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T14:00:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:33 smithi118 ceph-mon[33862]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T14:00:35.612 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[29576]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.613 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:35.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:35.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.614 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:35 smithi003 ceph-mon[40510]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T14:00:35.614 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 3 on host 'smithi003' 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:35 smithi118 ceph-mon[33862]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-21T14:00:36.243 DEBUG:teuthology.orchestra.run.smithi003:osd.3> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.3.service 2024-03-21T14:00:36.245 INFO:tasks.cephadm:Deploying osd.4 on smithi118 with /dev/vg_nvme/lv_4... 2024-03-21T14:00:36.245 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-21T14:00:36.494 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:36.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:36.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: osdmap e23: 4 total, 3 up, 4 in 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:36.616 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:36 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:36.970 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:00:36.751+0000 7f5c6ee10700 -1 osd.3 0 waiting for initial osdmap 2024-03-21T14:00:36.971 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:00:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:00:36.755+0000 7f5c66bfb700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T14:00:37.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T14:00:37.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:37.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' 2024-03-21T14:00:37.551 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:37 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: osdmap e24: 4 total, 3 up, 4 in 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: from='osd.3 v2:172.21.15.3:6813/1918209225' entity='osd.3' 2024-03-21T14:00:37.584 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:38.440 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-21T14:00:38.440 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-21T14:00:38.440 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-21T14:00:38.440 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-21T14:00:38.440 INFO:teuthology.orchestra.run.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00868874 s, 1.2 GB/s 2024-03-21T14:00:38.440 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-21T14:00:38.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:00:38.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:00:38.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[29576]: osd.3 v2:172.21.15.3:6813/1918209225 boot 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[29576]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[29576]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[40510]: osd.3 v2:172.21.15.3:6813/1918209225 boot 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[40510]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:38.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:38 smithi003 ceph-mon[40510]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:39.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:38 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:00:39.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:38 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:00:39.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:38 smithi118 ceph-mon[33862]: osd.3 v2:172.21.15.3:6813/1918209225 boot 2024-03-21T14:00:39.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:38 smithi118 ceph-mon[33862]: osdmap e25: 4 total, 4 up, 4 in 2024-03-21T14:00:39.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:38 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:00:39.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:38 smithi118 ceph-mon[33862]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:39.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_4 2024-03-21T14:00:39.327 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi003 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi003 to 4017M 2024-03-21T14:00:39.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:39.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:39.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:39.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi003 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi003 to 4017M 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: osdmap e26: 4 total, 4 up, 4 in 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi003 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi003 to 4017M 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:00:40.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:40 smithi118 ceph-mon[33862]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:40 smithi003 ceph-mon[29576]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:40 smithi003 ceph-mon[40510]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:42.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:41 smithi118 ceph-mon[33862]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:00:42.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:00:42.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:00:42.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[29576]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:00:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:00:42.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:00:42.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[40510]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:00:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:00:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:00:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:43.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:42 smithi118 ceph-mon[33862]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:43.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:42 smithi003 ceph-mon[29576]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:43.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:42 smithi003 ceph-mon[40510]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:43 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/2990059178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]: dispatch 2024-03-21T14:00:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]: dispatch 2024-03-21T14:00:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]': finished 2024-03-21T14:00:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:43 smithi118 ceph-mon[33862]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T14:00:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:43 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:00:44.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/2990059178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]: dispatch 2024-03-21T14:00:44.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]: dispatch 2024-03-21T14:00:44.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]': finished 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[29576]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/2990059178' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]: dispatch 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]: dispatch 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5"}]': finished 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[40510]: osdmap e27: 5 total, 4 up, 5 in 2024-03-21T14:00:44.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:43 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:00:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:44 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/715001649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:00:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:44 smithi118 ceph-mon[33862]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:44 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/715001649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:00:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:44 smithi003 ceph-mon[29576]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:45.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:44 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/715001649' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:00:45.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:44 smithi003 ceph-mon[40510]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:46 smithi118 ceph-mon[33862]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:47.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:46 smithi003 ceph-mon[29576]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:47.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:46 smithi003 ceph-mon[40510]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:48.873 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:48 smithi118 ceph-mon[33862]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:48 smithi003 ceph-mon[29576]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:48 smithi003 ceph-mon[40510]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T14:00:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T14:00:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:50.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-21T14:00:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:00:51.329 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:51 smithi118 ceph-mon[33862]: Deploying daemon osd.4 on smithi118 2024-03-21T14:00:51.329 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:51 smithi118 ceph-mon[33862]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:51.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:51 smithi003 ceph-mon[29576]: Deploying daemon osd.4 on smithi118 2024-03-21T14:00:51.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:51 smithi003 ceph-mon[29576]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:51.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:51 smithi003 ceph-mon[40510]: Deploying daemon osd.4 on smithi118 2024-03-21T14:00:51.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:51 smithi003 ceph-mon[40510]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:52 smithi118 ceph-mon[33862]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:53.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:52 smithi003 ceph-mon[29576]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:53.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:52 smithi003 ceph-mon[40510]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:55.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:55 smithi118 ceph-mon[33862]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:55 smithi003 ceph-mon[29576]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:55 smithi003 ceph-mon[40510]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:56 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:56 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:56 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:56 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:57.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:56 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:57.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:56 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:57.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:56 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:00:57.604 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:57 smithi118 ceph-mon[33862]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:57 smithi003 ceph-mon[29576]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:57 smithi003 ceph-mon[40510]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:58.891 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:58 smithi118 ceph-mon[33862]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:58.891 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:58 smithi118 ceph-mon[33862]: from='osd.4 v2:172.21.15.118:6800/296472230' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T14:00:58.891 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:58 smithi118 ceph-mon[33862]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T14:00:58.972 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 4 on host 'smithi118' 2024-03-21T14:00:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:58 smithi003 ceph-mon[29576]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:58 smithi003 ceph-mon[29576]: from='osd.4 v2:172.21.15.118:6800/296472230' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T14:00:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:58 smithi003 ceph-mon[29576]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T14:00:59.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:58 smithi003 ceph-mon[40510]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:00:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:58 smithi003 ceph-mon[40510]: from='osd.4 v2:172.21.15.118:6800/296472230' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T14:00:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:58 smithi003 ceph-mon[40510]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-21T14:00:59.563 DEBUG:teuthology.orchestra.run.smithi118:osd.4> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.4.service 2024-03-21T14:00:59.566 INFO:tasks.cephadm:Deploying osd.5 on smithi118 with /dev/vg_nvme/lv_3... 2024-03-21T14:00:59.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-21T14:00:59.891 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: from='osd.4 v2:172.21.15.118:6800/296472230' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:00:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T14:01:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: from='osd.4 v2:172.21.15.118:6800/296472230' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T14:01:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: from='osd.4 v2:172.21.15.118:6800/296472230' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: osdmap e28: 5 total, 4 up, 5 in 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:00:59 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.281 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:00:59.976+0000 7fdd04174700 -1 osd.4 0 waiting for initial osdmap 2024-03-21T14:01:00.281 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:00:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:00:59.979+0000 7fdcfbf5f700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:01:00.978 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:00.978 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T14:01:00.978 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:00.979 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:00.979 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.979 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:00.979 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: from='osd.4 ' entity='osd.4' 2024-03-21T14:01:00.979 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:00 smithi118 ceph-mon[33862]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:01:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T14:01:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: from='osd.4 ' entity='osd.4' 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[29576]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: osdmap e29: 5 total, 4 up, 5 in 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: from='osd.4 ' entity='osd.4' 2024-03-21T14:01:01.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:00 smithi003 ceph-mon[40510]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-21T14:01:01.780 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:01 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:01:01.781 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:01 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:01:01.781 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:01 smithi118 ceph-mon[33862]: osd.4 v2:172.21.15.118:6800/296472230 boot 2024-03-21T14:01:01.781 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:01 smithi118 ceph-mon[33862]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T14:01:01.781 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:01 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:01.781 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:01 smithi118 ceph-mon[33862]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T14:01:01.849 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-21T14:01:01.850 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-21T14:01:01.850 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-21T14:01:01.850 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-21T14:01:01.850 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00996076 s, 1.1 GB/s 2024-03-21T14:01:01.850 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-21T14:01:02.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:01:02.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[29576]: osd.4 v2:172.21.15.118:6800/296472230 boot 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[29576]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[29576]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[40510]: osd.4 v2:172.21.15.118:6800/296472230 boot 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[40510]: osdmap e30: 5 total, 5 up, 5 in 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:01:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:01 smithi003 ceph-mon[40510]: osdmap e31: 5 total, 5 up, 5 in 2024-03-21T14:01:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:02 smithi118 ceph-mon[33862]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:02 smithi118 ceph-mon[33862]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T14:01:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:02 smithi003 ceph-mon[29576]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:02 smithi003 ceph-mon[29576]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T14:01:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:02 smithi003 ceph-mon[40510]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:02 smithi003 ceph-mon[40510]: osdmap e32: 5 total, 5 up, 5 in 2024-03-21T14:01:03.374 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_3 2024-03-21T14:01:03.954 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:01:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:04 smithi003 ceph-mon[29576]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T14:01:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:04 smithi003 ceph-mon[29576]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:04 smithi003 ceph-mon[40510]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T14:01:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:04 smithi003 ceph-mon[40510]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:04 smithi118 ceph-mon[33862]: osdmap e33: 5 total, 5 up, 5 in 2024-03-21T14:01:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:04 smithi118 ceph-mon[33862]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi118 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi118 to 17088M 2024-03-21T14:01:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:05 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T14:01:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi118 2024-03-21T14:01:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi118 to 17088M 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: osdmap e34: 5 total, 5 up, 5 in 2024-03-21T14:01:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi118 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi118 to 17088M 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:06.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:05 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:07.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[29576]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:07.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:07.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:07.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:07.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[29576]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:07.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[40510]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:07.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:07.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:07.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:07.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:06 smithi003 ceph-mon[40510]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:07.092 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:06 smithi118 ceph-mon[33862]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:07.093 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:06 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:07.093 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:06 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:07.093 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:06 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:07.093 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:06 smithi118 ceph-mon[33862]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:09 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:01:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:09 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:01:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:09 smithi118 ceph-mon[33862]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T14:01:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:09 smithi118 ceph-mon[33862]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-21T14:01:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:01:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:01:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[40510]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T14:01:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[40510]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-21T14:01:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:01:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:01:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[29576]: osdmap e35: 5 total, 5 up, 5 in 2024-03-21T14:01:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:09 smithi003 ceph-mon[29576]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-03-21T14:01:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:10 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/1593509477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]: dispatch 2024-03-21T14:01:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:10 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]: dispatch 2024-03-21T14:01:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:10 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]': finished 2024-03-21T14:01:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:10 smithi118 ceph-mon[33862]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T14:01:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:10 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:10 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/2736684248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:01:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/1593509477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]: dispatch 2024-03-21T14:01:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]: dispatch 2024-03-21T14:01:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]': finished 2024-03-21T14:01:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[29576]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T14:01:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/2736684248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:01:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/1593509477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]: dispatch 2024-03-21T14:01:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]: dispatch 2024-03-21T14:01:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bff24605-38ce-47ce-9142-4ced3766476e"}]': finished 2024-03-21T14:01:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[40510]: osdmap e36: 6 total, 5 up, 6 in 2024-03-21T14:01:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:10 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/2736684248' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:01:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:11 smithi118 ceph-mon[33862]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:01:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:11 smithi003 ceph-mon[29576]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:01:11.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:11 smithi003 ceph-mon[40510]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:01:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:12 smithi118 ceph-mon[33862]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-21T14:01:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:12 smithi003 ceph-mon[29576]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-21T14:01:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:12 smithi003 ceph-mon[40510]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-21T14:01:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:13 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:01:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:13 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:01:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:13 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:01:14.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:13 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:01:14.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:13 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:01:14.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:13 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:01:14.797 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:14 smithi118 ceph-mon[33862]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T14:01:14.797 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:14 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:14.797 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:14 smithi118 ceph-mon[33862]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:14 smithi003 ceph-mon[29576]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T14:01:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:14 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:14 smithi003 ceph-mon[29576]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:14 smithi003 ceph-mon[40510]: osdmap e37: 6 total, 5 up, 6 in 2024-03-21T14:01:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:14 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:14 smithi003 ceph-mon[40510]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:16.642 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:16 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T14:01:16.642 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:16 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:16.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:16 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T14:01:16.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:16 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:16.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:16 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-21T14:01:16.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:16 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:17.626 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:17 smithi118 ceph-mon[33862]: Deploying daemon osd.5 on smithi118 2024-03-21T14:01:17.627 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:17 smithi118 ceph-mon[33862]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:17.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:17 smithi003 ceph-mon[29576]: Deploying daemon osd.5 on smithi118 2024-03-21T14:01:17.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:17 smithi003 ceph-mon[29576]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:17.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:17 smithi003 ceph-mon[40510]: Deploying daemon osd.5 on smithi118 2024-03-21T14:01:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:17 smithi003 ceph-mon[40510]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:18.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:18 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:01:18.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:18 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:01:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:18 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:01:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:18 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:01:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:18 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:01:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:18 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:01:19.535 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:19 smithi118 ceph-mon[33862]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:19.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:19 smithi003 ceph-mon[29576]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:19.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:19 smithi003 ceph-mon[40510]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:20.902 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:20 smithi118 ceph-mon[33862]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:20 smithi003 ceph-mon[29576]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:21.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:20 smithi003 ceph-mon[40510]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:22 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:22 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:22 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:22 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:22 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:22 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:22.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:22 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:22 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:22 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:23 smithi003 ceph-mon[29576]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:23.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:23 smithi003 ceph-mon[40510]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:23 smithi118 ceph-mon[33862]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:25.126 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 5 on host 'smithi118' 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='osd.5 v2:172.21.15.118:6804/326385780' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T14:01:25.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T14:01:25.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:25 smithi118 ceph-mon[33862]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:25.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:25.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:25.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='osd.5 v2:172.21.15.118:6804/326385780' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[29576]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='osd.5 v2:172.21.15.118:6804/326385780' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-21T14:01:25.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:25 smithi003 ceph-mon[40510]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:25.733 DEBUG:teuthology.orchestra.run.smithi118:osd.5> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.5.service 2024-03-21T14:01:25.735 INFO:tasks.cephadm:Deploying osd.6 on smithi118 with /dev/vg_nvme/lv_2... 2024-03-21T14:01:25.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-21T14:01:26.294 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='osd.5 v2:172.21.15.118:6804/326385780' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:26.295 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:26.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T14:01:26.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='osd.5 v2:172.21.15.118:6804/326385780' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:26.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T14:01:26.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:26.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:26.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='osd.5 v2:172.21.15.118:6804/326385780' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: osdmap e38: 6 total, 5 up, 6 in 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:26.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:26 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:26.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:01:26.294+0000 7f3bf210e700 -1 osd.5 0 waiting for initial osdmap 2024-03-21T14:01:26.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:01:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:01:26.298+0000 7f3be96f8700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:01:26.854 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:01:27.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:27.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: osdmap e39: 6 total, 5 up, 6 in 2024-03-21T14:01:27.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:27.386 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: from='osd.5 ' entity='osd.5' 2024-03-21T14:01:27.387 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:27.387 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: osd.5 v2:172.21.15.118:6804/326385780 boot 2024-03-21T14:01:27.387 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T14:01:27.387 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:27 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: osdmap e39: 6 total, 5 up, 6 in 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: from='osd.5 ' entity='osd.5' 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: osd.5 v2:172.21.15.118:6804/326385780 boot 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: osdmap e39: 6 total, 5 up, 6 in 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: from='osd.5 ' entity='osd.5' 2024-03-21T14:01:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-21T14:01:27.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: osd.5 v2:172.21.15.118:6804/326385780 boot 2024-03-21T14:01:27.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: osdmap e40: 6 total, 6 up, 6 in 2024-03-21T14:01:27.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:27 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:01:28.670 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:28 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:01:28.670 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:28 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:01:28.671 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:28 smithi118 ceph-mon[33862]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T14:01:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:28 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:01:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:28 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:01:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:28 smithi003 ceph-mon[29576]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T14:01:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:28 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:01:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:28 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:01:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:28 smithi003 ceph-mon[40510]: osdmap e41: 6 total, 6 up, 6 in 2024-03-21T14:01:29.856 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-21T14:01:29.857 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-21T14:01:29.857 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-21T14:01:29.857 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-21T14:01:29.857 INFO:teuthology.orchestra.run.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00902745 s, 1.2 GB/s 2024-03-21T14:01:29.857 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi118 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi118 to 8544M 2024-03-21T14:01:29.857 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:29.858 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:29.858 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:29.858 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:29.858 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:29.858 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:29 smithi118 ceph-mon[33862]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T14:01:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi118 2024-03-21T14:01:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi118 to 8544M 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[29576]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi118 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi118 to 8544M 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:29 smithi003 ceph-mon[40510]: osdmap e42: 6 total, 6 up, 6 in 2024-03-21T14:01:30.463 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_2 2024-03-21T14:01:30.703 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:01:31.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:31 smithi118 ceph-mon[33862]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T14:01:31.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:31 smithi118 ceph-mon[33862]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:31.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:31 smithi003 ceph-mon[29576]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T14:01:31.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:31 smithi003 ceph-mon[29576]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:31.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:31 smithi003 ceph-mon[40510]: osdmap e43: 6 total, 6 up, 6 in 2024-03-21T14:01:31.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:31 smithi003 ceph-mon[40510]: pgmap v140: 1 pgs: 1 remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:32.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:32 smithi118 ceph-mon[33862]: osdmap e44: 6 total, 6 up, 6 in 2024-03-21T14:01:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:32 smithi003 ceph-mon[29576]: osdmap e44: 6 total, 6 up, 6 in 2024-03-21T14:01:32.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:32 smithi003 ceph-mon[40510]: osdmap e44: 6 total, 6 up, 6 in 2024-03-21T14:01:33.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:33 smithi118 ceph-mon[33862]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:33.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:33.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:33.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:33.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:33 smithi118 ceph-mon[33862]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-21T14:01:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[29576]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[29576]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-21T14:01:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[40510]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:33 smithi003 ceph-mon[40510]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-21T14:01:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:35 smithi118 ceph-mon[33862]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:01:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/1594594894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]: dispatch 2024-03-21T14:01:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:35 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]: dispatch 2024-03-21T14:01:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:35 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]': finished 2024-03-21T14:01:35.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:35 smithi118 ceph-mon[33862]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T14:01:35.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:35 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[29576]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:01:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/1594594894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]: dispatch 2024-03-21T14:01:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]: dispatch 2024-03-21T14:01:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]': finished 2024-03-21T14:01:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[29576]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[40510]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/1594594894' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]: dispatch 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]: dispatch 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73a781e7-8660-4885-b976-2aeb8515109a"}]': finished 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[40510]: osdmap e45: 7 total, 6 up, 7 in 2024-03-21T14:01:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:35 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:36 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/1162811766' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:01:36.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:36 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/1162811766' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:01:36.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:36 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/1162811766' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:01:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:37 smithi118 ceph-mon[33862]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-21T14:01:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:37 smithi003 ceph-mon[29576]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-21T14:01:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:37 smithi003 ceph-mon[40510]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-03-21T14:01:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:38 smithi118 ceph-mon[33862]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T14:01:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:38 smithi003 ceph-mon[29576]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T14:01:39.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:38 smithi003 ceph-mon[40510]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-21T14:01:40.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:40 smithi118 ceph-mon[33862]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-21T14:01:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:40 smithi003 ceph-mon[29576]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-21T14:01:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:40 smithi003 ceph-mon[40510]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-21T14:01:41.856 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T14:01:41.856 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:41.856 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:41 smithi118 ceph-mon[33862]: Deploying daemon osd.6 on smithi118 2024-03-21T14:01:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T14:01:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:41 smithi003 ceph-mon[29576]: Deploying daemon osd.6 on smithi118 2024-03-21T14:01:42.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-21T14:01:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:41 smithi003 ceph-mon[40510]: Deploying daemon osd.6 on smithi118 2024-03-21T14:01:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:42 smithi003 ceph-mon[29576]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:42 smithi003 ceph-mon[40510]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:43.044 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:42 smithi118 ceph-mon[33862]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:44 smithi118 ceph-mon[33862]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:44 smithi003 ceph-mon[29576]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:44 smithi003 ceph-mon[40510]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:46 smithi118 ceph-mon[33862]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:47.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:46 smithi003 ceph-mon[29576]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:47.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:46 smithi003 ceph-mon[40510]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:48 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:48 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:48.425 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:48.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:48.426 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:48 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:49.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:49 smithi118 ceph-mon[33862]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:49.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:49 smithi003 ceph-mon[29576]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:49.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:49 smithi003 ceph-mon[40510]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:50.886 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 6 on host 'smithi118' 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='osd.6 v2:172.21.15.118:6808/1322208109' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.172 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='osd.6 v2:172.21.15.118:6808/1322208109' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='osd.6 v2:172.21.15.118:6808/1322208109' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:01:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:01:51.460 DEBUG:teuthology.orchestra.run.smithi118:osd.6> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T14:01:51.463 INFO:tasks.cephadm:Deploying osd.7 on smithi118 with /dev/vg_nvme/lv_1... 2024-03-21T14:01:51.463 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-21T14:01:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T14:01:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-mon[33862]: from='osd.6 v2:172.21.15.118:6808/1322208109' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-mon[33862]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T14:01:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:52.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[50523]: 2024-03-21T14:01:51.968+0000 7f6c62010700 -1 osd.6 0 waiting for initial osdmap 2024-03-21T14:01:52.182 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:01:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[50523]: 2024-03-21T14:01:51.974+0000 7f6c5de03700 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:01:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T14:01:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[29576]: from='osd.6 v2:172.21.15.118:6808/1322208109' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[29576]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T14:01:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T14:01:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[40510]: from='osd.6 v2:172.21.15.118:6808/1322208109' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[40510]: osdmap e46: 7 total, 6 up, 7 in 2024-03-21T14:01:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:51 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:01:52.822 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:01:53.124 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:52 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:53.124 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:52 smithi118 ceph-mon[33862]: osdmap e47: 7 total, 6 up, 7 in 2024-03-21T14:01:53.124 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:53.124 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:52 smithi118 ceph-mon[33862]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[29576]: osdmap e47: 7 total, 6 up, 7 in 2024-03-21T14:01:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[29576]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:01:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[40510]: osdmap e47: 7 total, 6 up, 7 in 2024-03-21T14:01:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:52 smithi003 ceph-mon[40510]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-03-21T14:01:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:01:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:01:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[29576]: osd.6 v2:172.21.15.118:6808/1322208109 boot 2024-03-21T14:01:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[29576]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T14:01:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:01:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:01:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[40510]: osd.6 v2:172.21.15.118:6808/1322208109 boot 2024-03-21T14:01:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[40510]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T14:01:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:54.280 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:53 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:01:54.281 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:53 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:01:54.281 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:53 smithi118 ceph-mon[33862]: osd.6 v2:172.21.15.118:6808/1322208109 boot 2024-03-21T14:01:54.281 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:53 smithi118 ceph-mon[33862]: osdmap e48: 7 total, 7 up, 7 in 2024-03-21T14:01:54.281 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:01:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T14:01:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi118 2024-03-21T14:01:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi118 to 5696M 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[29576]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T14:01:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi118 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi118 to 5696M 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:54 smithi003 ceph-mon[40510]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: osdmap e49: 7 total, 7 up, 7 in 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi118 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:01:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi118 to 5696M 2024-03-21T14:01:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:01:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:01:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:54 smithi118 ceph-mon[33862]: osdmap e50: 7 total, 7 up, 7 in 2024-03-21T14:01:55.569 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-21T14:01:55.569 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-21T14:01:55.569 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-21T14:01:55.569 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-21T14:01:55.569 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00859734 s, 1.2 GB/s 2024-03-21T14:01:55.570 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-21T14:01:56.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_1 2024-03-21T14:01:56.670 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:01:57.059 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:56 smithi118 ceph-mon[33862]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T14:01:57.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:56 smithi118 ceph-mon[33862]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:56 smithi003 ceph-mon[29576]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T14:01:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:56 smithi003 ceph-mon[29576]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:56 smithi003 ceph-mon[40510]: osdmap e51: 7 total, 7 up, 7 in 2024-03-21T14:01:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:56 smithi003 ceph-mon[40510]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:58.057 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:57 smithi118 ceph-mon[33862]: osdmap e52: 7 total, 7 up, 7 in 2024-03-21T14:01:58.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:57 smithi003 ceph-mon[29576]: osdmap e52: 7 total, 7 up, 7 in 2024-03-21T14:01:58.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:57 smithi003 ceph-mon[40510]: osdmap e52: 7 total, 7 up, 7 in 2024-03-21T14:01:59.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:01:59.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[29576]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:59.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:58 smithi003 ceph-mon[40510]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:01:59.336 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:58 smithi118 ceph-mon[33862]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:00.040 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:01:59 smithi118 ceph-mon[33862]: osdmap e53: 7 total, 7 up, 7 in 2024-03-21T14:02:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:01:59 smithi003 ceph-mon[29576]: osdmap e53: 7 total, 7 up, 7 in 2024-03-21T14:02:00.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:01:59 smithi003 ceph-mon[40510]: osdmap e53: 7 total, 7 up, 7 in 2024-03-21T14:02:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:00 smithi003 ceph-mon[29576]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-21T14:02:01.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:01 smithi003 ceph-mon[40510]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-21T14:02:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:00 smithi118 ceph-mon[33862]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-21T14:02:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:01 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/4247383037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]: dispatch 2024-03-21T14:02:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]: dispatch 2024-03-21T14:02:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[29576]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]': finished 2024-03-21T14:02:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[29576]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T14:02:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/464798463' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/4247383037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]: dispatch 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]: dispatch 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[40510]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]': finished 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[40510]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/464798463' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:02:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/4247383037' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]: dispatch 2024-03-21T14:02:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:02 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]: dispatch 2024-03-21T14:02:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:02 smithi118 ceph-mon[33862]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0"}]': finished 2024-03-21T14:02:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:02 smithi118 ceph-mon[33862]: osdmap e54: 8 total, 7 up, 8 in 2024-03-21T14:02:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:02 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/464798463' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-21T14:02:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:03 smithi118 ceph-mon[33862]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:02:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:03 smithi003 ceph-mon[29576]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:02:03.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:03 smithi003 ceph-mon[40510]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:02:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:04 smithi003 ceph-mon[29576]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-21T14:02:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:04 smithi003 ceph-mon[40510]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-21T14:02:05.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:04 smithi118 ceph-mon[33862]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-21T14:02:07.871 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:07 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:02:07.871 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:07 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:02:07.871 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:07 smithi118 ceph-mon[33862]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T14:02:07.871 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:07.871 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:07 smithi118 ceph-mon[33862]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:02:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:02:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[29576]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T14:02:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[29576]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:02:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:02:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[40510]: osdmap e55: 8 total, 7 up, 8 in 2024-03-21T14:02:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:07 smithi003 ceph-mon[40510]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:08.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:08 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T14:02:08.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:08 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:08 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T14:02:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:08 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:09.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:08 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-21T14:02:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:08 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:09 smithi118 ceph-mon[33862]: Deploying daemon osd.7 on smithi118 2024-03-21T14:02:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:09 smithi118 ceph-mon[33862]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:09.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:09 smithi003 ceph-mon[29576]: Deploying daemon osd.7 on smithi118 2024-03-21T14:02:09.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:09 smithi003 ceph-mon[29576]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:09.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:09 smithi003 ceph-mon[40510]: Deploying daemon osd.7 on smithi118 2024-03-21T14:02:09.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:09 smithi003 ceph-mon[40510]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:10 smithi118 ceph-mon[33862]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:10 smithi003 ceph-mon[29576]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:11.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:10 smithi003 ceph-mon[40510]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:12.757 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:12 smithi118 ceph-mon[33862]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:12 smithi003 ceph-mon[29576]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:12 smithi003 ceph-mon[40510]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:14.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:14 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:14.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:14 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:14.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:14 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:14.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:14 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:14.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:14 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:14.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:14 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:14.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:14 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:14.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:14 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:14.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:14 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:15 smithi118 ceph-mon[33862]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:15.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:15 smithi003 ceph-mon[29576]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:15.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:15 smithi003 ceph-mon[40510]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:17.104 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 7 on host 'smithi118' 2024-03-21T14:02:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: from='osd.7 v2:172.21.15.118:6812/2128678671' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T14:02:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T14:02:17.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:17.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:17 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.717 DEBUG:teuthology.orchestra.run.smithi118:osd.7> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.7.service 2024-03-21T14:02:17.719 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-21T14:02:17.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-21T14:02:17.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: from='osd.7 v2:172.21.15.118:6812/2128678671' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: from='osd.7 v2:172.21.15.118:6812/2128678671' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:17.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:17 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:17.975 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:02:18.643 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-mon[33862]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T14:02:18.643 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-mon[33862]: from='osd.7 v2:172.21.15.118:6812/2128678671' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:02:18.643 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-mon[33862]: osdmap e56: 8 total, 7 up, 8 in 2024-03-21T14:02:18.643 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:18.643 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-mon[33862]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:02:18.643 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:02:18.635+0000 7fb6925c7700 -1 osd.7 0 waiting for initial osdmap 2024-03-21T14:02:18.643 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:02:18 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:02:18.642+0000 7fb689bb1700 -1 osd.7 57 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:02:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[29576]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T14:02:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[29576]: from='osd.7 v2:172.21.15.118:6812/2128678671' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:02:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[29576]: osdmap e56: 8 total, 7 up, 8 in 2024-03-21T14:02:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:18.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[29576]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[40510]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[40510]: from='osd.7 v2:172.21.15.118:6812/2128678671' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[40510]: osdmap e56: 8 total, 7 up, 8 in 2024-03-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:18 smithi003 ceph-mon[40510]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:02:19.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:02:19.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: osdmap e57: 8 total, 7 up, 8 in 2024-03-21T14:02:19.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: from='osd.7 ' entity='osd.7' 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: osd.7 v2:172.21.15.118:6812/2128678671 boot 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: osdmap e57: 8 total, 7 up, 8 in 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: from='osd.7 ' entity='osd.7' 2024-03-21T14:02:19.658 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: osd.7 v2:172.21.15.118:6812/2128678671 boot 2024-03-21T14:02:19.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T14:02:19.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:19 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: osdmap e57: 8 total, 7 up, 8 in 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 206 MiB used, 626 GiB / 626 GiB avail 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: from='osd.7 ' entity='osd.7' 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: osd.7 v2:172.21.15.118:6812/2128678671 boot 2024-03-21T14:02:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: osdmap e58: 8 total, 8 up, 8 in 2024-03-21T14:02:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:19 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:02:19.902 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:02:20.577 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:20 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:02:20.577 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:20 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:02:20.577 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:20 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:02:20.577 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:20 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:02:20.577 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1507236108' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T14:02:20.577 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:20 smithi118 ceph-mon[33862]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T14:02:20.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:02:20.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:02:20.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:02:20.586 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1507236108' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[29576]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1507236108' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-21T14:02:20.587 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:20 smithi003 ceph-mon[40510]: osdmap e59: 8 total, 8 up, 8 in 2024-03-21T14:02:20.614 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1711029739,"num_in_osds":8,"osd_in_since":1711029721,"num_remapped_pgs":1} 2024-03-21T14:02:20.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T14:02:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: Detected new or changed devices on smithi118 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:21 smithi118 ceph-mon[33862]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: Detected new or changed devices on smithi118 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[29576]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: Detected new or changed devices on smithi118 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:21.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:21.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:21 smithi003 ceph-mon[40510]: osdmap e60: 8 total, 8 up, 8 in 2024-03-21T14:02:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi118 to 4272M 2024-03-21T14:02:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:02:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:02:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[29576]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T14:02:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi118 to 4272M 2024-03-21T14:02:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:02:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:02:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:22 smithi003 ceph-mon[40510]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T14:02:22.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:02:22.793 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":61,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","created":"2024-03-21T13:56:54.530125+0000","modified":"2024-03-21T14:02:22.327850+0000","last_up_change":"2024-03-21T14:02:19.321454+0000","last_in_change":"2024-03-21T14:02:01.236118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T14:00:12.925535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"daf9bc3c-4e73-4379-afb6-7ee181b686fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":930096433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":930096433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":930096433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":930096433}]},"public_addr":"172.21.15.3:6801/930096433","cluster_addr":"172.21.15.3:6802/930096433","heartbeat_back_addr":"172.21.15.3:6804/930096433","heartbeat_front_addr":"172.21.15.3:6803/930096433","state":["exists","up"]},{"osd":1,"uuid":"1652f24d-aa0e-471c-bf54-03814b1ee11d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1526485482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1526485482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1526485482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1526485482}]},"public_addr":"172.21.15.3:6805/1526485482","cluster_addr":"172.21.15.3:6806/1526485482","heartbeat_back_addr":"172.21.15.3:6808/1526485482","heartbeat_front_addr":"172.21.15.3:6807/1526485482","state":["exists","up"]},{"osd":2,"uuid":"5e67d907-f8d1-4ecc-85d3-df88da5c364c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3751378277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3751378277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3751378277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3751378277}]},"public_addr":"172.21.15.3:6809/3751378277","cluster_addr":"172.21.15.3:6810/3751378277","heartbeat_back_addr":"172.21.15.3:6812/3751378277","heartbeat_front_addr":"172.21.15.3:6811/3751378277","state":["exists","up"]},{"osd":3,"uuid":"ad8dabdf-fce5-4312-9e80-dbbb6ba2947c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":1918209225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1918209225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1918209225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":1918209225}]},"public_addr":"172.21.15.3:6813/1918209225","cluster_addr":"172.21.15.3:6814/1918209225","heartbeat_back_addr":"172.21.15.3:6816/1918209225","heartbeat_front_addr":"172.21.15.3:6815/1918209225","state":["exists","up"]},{"osd":4,"uuid":"b5d486b1-2cb6-46c7-8664-994af07ff7b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":296472230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":296472230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":296472230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":296472230}]},"public_addr":"172.21.15.118:6800/296472230","cluster_addr":"172.21.15.118:6801/296472230","heartbeat_back_addr":"172.21.15.118:6803/296472230","heartbeat_front_addr":"172.21.15.118:6802/296472230","state":["exists","up"]},{"osd":5,"uuid":"bff24605-38ce-47ce-9142-4ced3766476e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":326385780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":326385780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":326385780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":326385780}]},"public_addr":"172.21.15.118:6804/326385780","cluster_addr":"172.21.15.118:6805/326385780","heartbeat_back_addr":"172.21.15.118:6807/326385780","heartbeat_front_addr":"172.21.15.118:6806/326385780","state":["exists","up"]},{"osd":6,"uuid":"73a781e7-8660-4885-b976-2aeb8515109a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1322208109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":1322208109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":1322208109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1322208109}]},"public_addr":"172.21.15.118:6808/1322208109","cluster_addr":"172.21.15.118:6809/1322208109","heartbeat_back_addr":"172.21.15.118:6811/1322208109","heartbeat_front_addr":"172.21.15.118:6810/1322208109","state":["exists","up"]},{"osd":7,"uuid":"3e95559b-79f6-40ba-9d98-d350bcdd60e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2128678671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":2128678671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":2128678671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2128678671}]},"public_addr":"172.21.15.118:6812/2128678671","cluster_addr":"172.21.15.118:6813/2128678671","heartbeat_back_addr":"172.21.15.118:6815/2128678671","heartbeat_front_addr":"172.21.15.118:6814/2128678671","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:01:58.399169+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:21.152955+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:01:12.779720+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:06.539457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:19.486841+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:01:25.291346+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:01:51.130228+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:17.436393+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3884352348":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2319227492":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2154949954":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/4162970769":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/2292819669":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/520312891":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2267426691":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/3211426112":"2024-03-22T13:57:56.576834+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T14:02:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:22 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi118 to 4272M 2024-03-21T14:02:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:22 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:02:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:22 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:02:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:22 smithi118 ceph-mon[33862]: osdmap e61: 8 total, 8 up, 8 in 2024-03-21T14:02:23.543 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T14:00:12.925535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-21T14:02:23.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-21T14:02:23.789 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:02:23.903 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:23 smithi003 ceph-mon[29576]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 368 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-21T14:02:23.903 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1413134884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:02:23.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:23 smithi003 ceph-mon[29576]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T14:02:23.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:23 smithi003 ceph-mon[40510]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 368 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-21T14:02:23.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:23 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1413134884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:02:23.904 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:23 smithi003 ceph-mon[40510]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T14:02:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:23 smithi118 ceph-mon[33862]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 368 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-21T14:02:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:23 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1413134884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:02:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:23 smithi118 ceph-mon[33862]: osdmap e62: 8 total, 8 up, 8 in 2024-03-21T14:02:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:25 smithi118 ceph-mon[33862]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-21T14:02:25.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:25 smithi003 ceph-mon[29576]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-21T14:02:25.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:25 smithi003 ceph-mon[40510]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-21T14:02:25.761 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-03-21T14:02:26.353 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4233832510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T14:02:26.353 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4233832510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T14:02:26.383 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi003 2024-03-21T14:02:26.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi003=foo.a' 2024-03-21T14:02:26.633 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4233832510' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-21T14:02:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:27 smithi118 ceph-mon[33862]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:02:27.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:27 smithi003 ceph-mon[40510]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:02:27.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:27 smithi003 ceph-mon[29576]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-21T14:02:28.403 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled rgw.foo.a update... 2024-03-21T14:02:29.001 DEBUG:teuthology.orchestra.run.smithi003:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-21T14:02:29.004 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi118 2024-03-21T14:02:29.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-21T14:02:29.240 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi003=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.625 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T14:02:29.626 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T14:02:29.626 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.626 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:29 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi003=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T14:02:29.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi003=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-21T14:02:29.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-21T14:02:29.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:29.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:29 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:30.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:30 smithi003 ceph-mon[29576]: Deploying daemon rgw.foo.a on smithi003 2024-03-21T14:02:30.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:30 smithi003 ceph-mon[40510]: Deploying daemon rgw.foo.a on smithi003 2024-03-21T14:02:30.668 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:30 smithi118 ceph-mon[33862]: Deploying daemon rgw.foo.a on smithi003 2024-03-21T14:02:31.254 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 systemd[1]: Starting Ceph rgw.foo.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:02:31.433 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'datapool' created 2024-03-21T14:02:31.616 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:31 smithi003 ceph-mon[29576]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:31.616 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:31 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/2349941071' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T14:02:31.616 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:31 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T14:02:31.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:31 smithi003 ceph-mon[40510]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:31.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:31 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/2349941071' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T14:02:31.617 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:31 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T14:02:31.617 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 podman[67204]: 2024-03-21 14:02:31.172548891 +0000 UTC m=+0.017269676 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:02:31.617 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 podman[67204]: 2024-03-21 14:02:31.280692253 +0000 UTC m=+0.125413038 container create afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-20-2326-ba76009) 2024-03-21T14:02:31.617 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 podman[67204]: 2024-03-21 14:02:31.508488864 +0000 UTC m=+0.353209660 container init afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-20-2326-ba76009) 2024-03-21T14:02:31.617 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 podman[67204]: 2024-03-21 14:02:31.511461237 +0000 UTC m=+0.356182031 container start afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:02:31.617 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 bash[67204]: afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 2024-03-21T14:02:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:31 smithi118 ceph-mon[33862]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:31 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/2349941071' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T14:02:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:31 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-21T14:02:31.959 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:02:31 smithi003 systemd[1]: Started Ceph rgw.foo.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:02:32.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-21T14:02:32.331 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:32.735 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:32 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: osdmap e63: 8 total, 8 up, 8 in 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:32.755 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:32 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:33.661 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-21T14:02:33.661 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T14:02:33.661 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2605341271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: pgmap v192: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T14:02:33.662 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:33 smithi118 ceph-mon[33862]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2605341271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: pgmap v192: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[40510]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: Saving service rgw.foo.a spec with placement smithi003=foo.a;count:1 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: osdmap e64: 8 total, 8 up, 8 in 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2605341271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: pgmap v192: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-21T14:02:33.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:33 smithi003 ceph-mon[29576]: osdmap e65: 8 total, 8 up, 8 in 2024-03-21T14:02:34.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: Checking dashboard <-> RGW credentials 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/1039806201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: Checking dashboard <-> RGW credentials 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/1039806201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:34 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: Checking dashboard <-> RGW credentials 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/1039806201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: osdmap e66: 8 total, 8 up, 8 in 2024-03-21T14:02:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:34.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:34 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-21T14:02:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[29576]: pgmap v195: 68 pgs: 10 creating+peering, 11 active+clean, 47 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T14:02:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T14:02:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T14:02:35.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[29576]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[40510]: pgmap v195: 68 pgs: 10 creating+peering, 11 active+clean, 47 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T14:02:35.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:35 smithi003 ceph-mon[40510]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T14:02:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:35 smithi118 ceph-mon[33862]: pgmap v195: 68 pgs: 10 creating+peering, 11 active+clean, 47 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T14:02:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:35 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T14:02:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:35 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-21T14:02:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:35 smithi118 ceph-mon[33862]: osdmap e67: 8 total, 8 up, 8 in 2024-03-21T14:02:37.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.118 --placement '1;smithi118=iscsi.a' 2024-03-21T14:02:37.314 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:37.719 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T14:02:37.719 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.719 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.719 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.719 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.720 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: pgmap v198: 100 pgs: 10 creating+peering, 24 active+clean, 66 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T14:02:37.720 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:37 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: pgmap v198: 100 pgs: 10 creating+peering, 24 active+clean, 66 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:37.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: osdmap e68: 8 total, 8 up, 8 in 2024-03-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: pgmap v198: 100 pgs: 10 creating+peering, 24 active+clean, 66 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-21T14:02:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:37 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:38.693 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:38 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T14:02:38.694 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:38 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T14:02:38.694 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:38 smithi118 ceph-mon[33862]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T14:02:38.694 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:38 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:02:38.694 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:38 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[29576]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:02:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[40510]: osdmap e69: 8 total, 8 up, 8 in 2024-03-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:02:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:38 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:02:39.104 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled iscsi.datapool update... 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: pgmap v201: 132 pgs: 8 creating+peering, 78 active+clean, 46 unknown; 578 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-21T14:02:39.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:39.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:39 smithi118 ceph-mon[33862]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: pgmap v201: 132 pgs: 8 creating+peering, 78 active+clean, 46 unknown; 578 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-21T14:02:39.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[40510]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: osdmap e70: 8 total, 8 up, 8 in 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: pgmap v201: 132 pgs: 8 creating+peering, 78 active+clean, 46 unknown; 578 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:39.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:39 smithi003 ceph-mon[29576]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:02:39.765 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-21T14:02:39.765 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T14:02:39.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-21T14:02:39.796 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T14:02:39.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-21T14:02:39.825 DEBUG:teuthology.orchestra.run.smithi118:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-21T14:02:39.868 INFO:tasks.cephadm:Adding prometheus.a on smithi118 2024-03-21T14:02:39.868 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi118=a' 2024-03-21T14:02:40.146 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:40.607 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.118", "placement": "1;smithi118=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:40.607 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: Saving service iscsi.datapool spec with placement smithi118=iscsi.a;count:1 2024-03-21T14:02:40.607 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T14:02:40.607 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T14:02:40.607 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T14:02:40.607 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.608 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.608 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.608 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:40 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.118", "placement": "1;smithi118=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:40.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: Saving service iscsi.datapool spec with placement smithi118=iscsi.a;count:1 2024-03-21T14:02:40.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T14:02:40.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.118", "placement": "1;smithi118=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: Saving service iscsi.datapool spec with placement smithi118=iscsi.a;count:1 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-21T14:02:40.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: osdmap e71: 8 total, 8 up, 8 in 2024-03-21T14:02:40.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1675140515' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2761509089' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:40.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:40 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-21T14:02:41.563 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T14:02:41.563 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T14:02:41.563 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T14:02:41.563 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: pgmap v204: 132 pgs: 14 creating+peering, 88 active+clean, 30 unknown; 578 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-21T14:02:41.563 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.564 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.564 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.564 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T14:02:41.564 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T14:02:41.564 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:41 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:41.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T14:02:41.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T14:02:41.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: pgmap v204: 132 pgs: 14 creating+peering, 88 active+clean, 30 unknown; 578 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-21T14:02:41.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: osdmap e72: 8 total, 8 up, 8 in 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: pgmap v204: 132 pgs: 14 creating+peering, 88 active+clean, 30 unknown; 578 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-21T14:02:41.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:41 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:42.030 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled prometheus update... 2024-03-21T14:02:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:42 smithi118 ceph-mon[33862]: Deploying daemon iscsi.iscsi.a on smithi118 2024-03-21T14:02:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:42 smithi118 ceph-mon[33862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:02:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:42 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:02:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:42 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[29576]: Deploying daemon iscsi.iscsi.a on smithi118 2024-03-21T14:02:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[29576]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:02:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:02:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[40510]: Deploying daemon iscsi.iscsi.a on smithi118 2024-03-21T14:02:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[40510]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:02:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:02:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:42 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:43.127 DEBUG:teuthology.orchestra.run.smithi118:prometheus.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-21T14:02:43.129 INFO:tasks.cephadm:Adding node-exporter.a on smithi003 2024-03-21T14:02:43.130 INFO:tasks.cephadm:Adding node-exporter.b on smithi118 2024-03-21T14:02:43.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi003=a;smithi118=b' 2024-03-21T14:02:43.640 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:43.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:43 smithi003 ceph-mon[29576]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi118=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:43.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:43 smithi003 ceph-mon[29576]: Saving service prometheus spec with placement smithi118=a;count:1 2024-03-21T14:02:43.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:43 smithi003 ceph-mon[29576]: pgmap v205: 132 pgs: 6 creating+peering, 114 active+clean, 12 unknown; 580 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.7 KiB/s wr, 95 op/s 2024-03-21T14:02:43.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:43 smithi003 ceph-mon[40510]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi118=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:43.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:43 smithi003 ceph-mon[40510]: Saving service prometheus spec with placement smithi118=a;count:1 2024-03-21T14:02:43.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:43 smithi003 ceph-mon[40510]: pgmap v205: 132 pgs: 6 creating+peering, 114 active+clean, 12 unknown; 580 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.7 KiB/s wr, 95 op/s 2024-03-21T14:02:43.762 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:43 smithi118 ceph-mon[33862]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi118=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:43.762 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:43 smithi118 ceph-mon[33862]: Saving service prometheus spec with placement smithi118=a;count:1 2024-03-21T14:02:43.762 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:43 smithi118 ceph-mon[33862]: pgmap v205: 132 pgs: 6 creating+peering, 114 active+clean, 12 unknown; 580 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 40 KiB/s rd, 3.7 KiB/s wr, 95 op/s 2024-03-21T14:02:45.110 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:44 smithi118 podman[61330]: 2024-03-21 14:02:44.983891642 +0000 UTC m=+0.124215957 container create 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T14:02:45.427 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-mon[33862]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T14:02:45.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-mon[33862]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 179 op/s 2024-03-21T14:02:45.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 podman[61330]: 2024-03-21 14:02:45.177597399 +0000 UTC m=+0.317921741 container init 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-21T14:02:45.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 podman[61330]: 2024-03-21 14:02:45.192173363 +0000 UTC m=+0.332497681 container start 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:02:45.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 bash[61330]: 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 2024-03-21T14:02:45.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 systemd[1]: Started Ceph iscsi.iscsi.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:02:45.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug Started the configuration object watcher 2024-03-21T14:02:45.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug Checking for config object changes every 1s 2024-03-21T14:02:45.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug Processing osd blocklist entries for this node 2024-03-21T14:02:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:45 smithi003 ceph-mon[29576]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T14:02:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:45 smithi003 ceph-mon[29576]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 179 op/s 2024-03-21T14:02:45.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:45 smithi003 ceph-mon[40510]: osdmap e73: 8 total, 8 up, 8 in 2024-03-21T14:02:45.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:45 smithi003 ceph-mon[40510]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 179 op/s 2024-03-21T14:02:46.012 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug Reading the configuration object to update local LIO configuration 2024-03-21T14:02:46.012 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug Configuration does not have an entry for this host(smithi118.front.sepia.ceph.com) - nothing to define to LIO 2024-03-21T14:02:46.012 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-21T14:02:46.096 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled node-exporter update... 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/635417771' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T14:02:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:46 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/635417771' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:02:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/635417771' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-21T14:02:46.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:46 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:47.029 DEBUG:teuthology.orchestra.run.smithi003:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-21T14:02:47.032 DEBUG:teuthology.orchestra.run.smithi118:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-21T14:02:47.034 INFO:tasks.cephadm:Adding alertmanager.a on smithi003 2024-03-21T14:02:47.035 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi003=a' 2024-03-21T14:02:47.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:02:47.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:02:47.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi118=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:47.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: Saving service node-exporter spec with placement smithi003=a;smithi118=b;count:2 2024-03-21T14:02:47.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.7 KiB/s wr, 155 op/s 2024-03-21T14:02:47.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:47.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:47 smithi118 ceph-mon[33862]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T14:02:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:02:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:02:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi118=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: Saving service node-exporter spec with placement smithi003=a;smithi118=b;count:2 2024-03-21T14:02:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.7 KiB/s wr, 155 op/s 2024-03-21T14:02:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[29576]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi003=a;smithi118=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: Saving service node-exporter spec with placement smithi003=a;smithi118=b;count:2 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 4.7 KiB/s wr, 155 op/s 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:47.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:47 smithi003 ceph-mon[40510]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-21T14:02:47.940 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:48 smithi003 ceph-mon[29576]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2024-03-21T14:02:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:48 smithi003 ceph-mon[40510]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2024-03-21T14:02:49.022 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:48 smithi118 ceph-mon[33862]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 4.2 KiB/s wr, 136 op/s 2024-03-21T14:02:49.789 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled alertmanager update... 2024-03-21T14:02:50.817 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:50.818 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:50 smithi118 ceph-mon[33862]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-21T14:02:50.844 DEBUG:teuthology.orchestra.run.smithi003:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-21T14:02:50.847 INFO:tasks.cephadm:Adding grafana.a on smithi118 2024-03-21T14:02:50.847 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi118=a' 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:51.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[29576]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi003=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: Saving service alertmanager spec with placement smithi003=a;count:1 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:51.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:50 smithi003 ceph-mon[40510]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 3.5 KiB/s wr, 112 op/s 2024-03-21T14:02:51.091 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:52.659 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:52 smithi118 ceph-mon[33862]: Deploying daemon prometheus.a on smithi118 2024-03-21T14:02:52.659 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:52 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:52 smithi003 ceph-mon[29576]: Deploying daemon prometheus.a on smithi118 2024-03-21T14:02:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:52 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:52 smithi003 ceph-mon[40510]: Deploying daemon prometheus.a on smithi118 2024-03-21T14:02:52.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:52 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:52.919 INFO:teuthology.orchestra.run.smithi118.stdout:Scheduled grafana update... 2024-03-21T14:02:53.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:53 smithi118 ceph-mon[33862]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 2.0 KiB/s wr, 64 op/s 2024-03-21T14:02:53.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:53 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:53.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:53 smithi003 ceph-mon[29576]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 2.0 KiB/s wr, 64 op/s 2024-03-21T14:02:53.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:53 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:53.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:53 smithi003 ceph-mon[40510]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 2.0 KiB/s wr, 64 op/s 2024-03-21T14:02:53.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:53 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:02:53.818 DEBUG:teuthology.orchestra.run.smithi118:grafana.a> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@grafana.a.service 2024-03-21T14:02:53.820 INFO:tasks.cephadm:Setting up client nodes... 2024-03-21T14:02:53.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T14:02:54.075 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:02:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:54 smithi003 ceph-mon[29576]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi118=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:54 smithi003 ceph-mon[29576]: Saving service grafana spec with placement smithi118=a;count:1 2024-03-21T14:02:54.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:54 smithi003 ceph-mon[40510]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi118=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:54.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:54 smithi003 ceph-mon[40510]: Saving service grafana spec with placement smithi118=a;count:1 2024-03-21T14:02:54.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:54 smithi118 ceph-mon[33862]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi118=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:02:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:54 smithi118 ceph-mon[33862]: Saving service grafana spec with placement smithi118=a;count:1 2024-03-21T14:02:55.757 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:55 smithi003 ceph-mon[29576]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 198 B/s wr, 6 op/s 2024-03-21T14:02:55.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:55 smithi003 ceph-mon[40510]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 198 B/s wr, 6 op/s 2024-03-21T14:02:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:55 smithi118 ceph-mon[33862]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 198 B/s wr, 6 op/s 2024-03-21T14:02:55.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:02:55 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:02:56.026 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-03-21T14:02:56.026 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQAQPvxlf6NgARAAAw0YXcC8kDrXyyCOaRMrqw== 2024-03-21T14:02:56.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:02:56.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/306056932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:56.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:56.610 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T14:02:56.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:02:56.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/306056932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:56.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:56.611 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:56 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T14:02:56.640 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-03-21T14:02:56.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-21T14:02:56.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-21T14:02:56.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-21T14:02:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:56 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:02:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/306056932' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:56 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:56 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T14:02:56.940 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:02:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-21T14:02:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T14:02:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:57 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-21T14:02:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:57 smithi118 ceph-mon[33862]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-21T14:02:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-21T14:02:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T14:02:57.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-21T14:02:57.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[29576]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-21T14:02:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-03-21T14:02:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-21T14:02:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-21T14:02:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:57 smithi003 ceph-mon[40510]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-21T14:02:58.748 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-21T14:02:58.748 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-21T14:02:58.748 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:58 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-21T14:02:58.748 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:58 smithi118 ceph-mon[33862]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T14:02:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-21T14:02:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-21T14:02:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-21T14:02:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[29576]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-03-21T14:02:58.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-21T14:02:58.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:58 smithi003 ceph-mon[40510]: osdmap e74: 8 total, 8 up, 8 in 2024-03-21T14:02:59.052 INFO:teuthology.orchestra.run.smithi118.stdout:[client.1] 2024-03-21T14:02:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQATPvxlU3fuAhAAV1qAnOzKbr/UGGlunI3ndg== 2024-03-21T14:02:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:59 smithi118 ceph-mon[33862]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T14:02:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:59 smithi118 ceph-mon[33862]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:02:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.118:0/3567699149' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:59 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:02:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:02:59 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T14:03:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[29576]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T14:03:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[29576]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.118:0/3567699149' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:03:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:03:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T14:03:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[40510]: osdmap e75: 8 total, 8 up, 8 in 2024-03-21T14:03:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[40510]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.118:0/3567699149' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:03:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-21T14:03:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:02:59 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-21T14:03:00.703 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-21T14:03:00.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-21T14:03:00.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-21T14:03:00.746 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-21T14:03:00.746 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-21T14:03:00.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-21T14:03:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:00 smithi118 ceph-mon[33862]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:01.000 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:00 smithi003 ceph-mon[29576]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:00 smithi003 ceph-mon[40510]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:02.895 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:03:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:02 smithi118 ceph-mon[33862]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 72 B/s, 2 objects/s recovering 2024-03-21T14:03:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:02 smithi003 ceph-mon[29576]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 72 B/s, 2 objects/s recovering 2024-03-21T14:03:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:02 smithi003 ceph-mon[40510]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 72 B/s, 2 objects/s recovering 2024-03-21T14:03:03.662 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":3687155565}]},"active_addr":"172.21.15.3:6800/3687155565","active_change":"2024-03-21T13:57:56.577001+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.9.5","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.9.5","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.9.5","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.9.5","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3411869253}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3883474825}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":384540368}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":45234419}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1598547836}]}]} 2024-03-21T14:03:03.666 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-21T14:03:03.666 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T14:03:03.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T14:03:03.816 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:03 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3883035970' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T14:03:03.915 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:03 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3883035970' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T14:03:04.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:03 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3883035970' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-21T14:03:04.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:04 smithi003 ceph-mon[29576]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 72 B/s, 2 objects/s recovering 2024-03-21T14:03:04.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:04 smithi003 ceph-mon[40510]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 72 B/s, 2 objects/s recovering 2024-03-21T14:03:04.972 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:04 smithi118 ceph-mon[33862]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 72 B/s, 2 objects/s recovering 2024-03-21T14:03:05.728 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:03:05.728 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","created":"2024-03-21T13:56:54.530125+0000","modified":"2024-03-21T14:03:05.699216+0000","last_up_change":"2024-03-21T14:02:19.321454+0000","last_in_change":"2024-03-21T14:02:01.236118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T14:00:12.925535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T14:02:31.373659+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T14:02:31.794831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T14:02:33.750394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T14:02:35.711498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T14:02:37.763124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"daf9bc3c-4e73-4379-afb6-7ee181b686fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":930096433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":930096433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":930096433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":930096433}]},"public_addr":"172.21.15.3:6801/930096433","cluster_addr":"172.21.15.3:6802/930096433","heartbeat_back_addr":"172.21.15.3:6804/930096433","heartbeat_front_addr":"172.21.15.3:6803/930096433","state":["exists","up"]},{"osd":1,"uuid":"1652f24d-aa0e-471c-bf54-03814b1ee11d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1526485482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1526485482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1526485482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1526485482}]},"public_addr":"172.21.15.3:6805/1526485482","cluster_addr":"172.21.15.3:6806/1526485482","heartbeat_back_addr":"172.21.15.3:6808/1526485482","heartbeat_front_addr":"172.21.15.3:6807/1526485482","state":["exists","up"]},{"osd":2,"uuid":"5e67d907-f8d1-4ecc-85d3-df88da5c364c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3751378277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3751378277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3751378277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3751378277}]},"public_addr":"172.21.15.3:6809/3751378277","cluster_addr":"172.21.15.3:6810/3751378277","heartbeat_back_addr":"172.21.15.3:6812/3751378277","heartbeat_front_addr":"172.21.15.3:6811/3751378277","state":["exists","up"]},{"osd":3,"uuid":"ad8dabdf-fce5-4312-9e80-dbbb6ba2947c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":1918209225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1918209225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1918209225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":1918209225}]},"public_addr":"172.21.15.3:6813/1918209225","cluster_addr":"172.21.15.3:6814/1918209225","heartbeat_back_addr":"172.21.15.3:6816/1918209225","heartbeat_front_addr":"172.21.15.3:6815/1918209225","state":["exists","up"]},{"osd":4,"uuid":"b5d486b1-2cb6-46c7-8664-994af07ff7b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":296472230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":296472230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":296472230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":296472230}]},"public_addr":"172.21.15.118:6800/296472230","cluster_addr":"172.21.15.118:6801/296472230","heartbeat_back_addr":"172.21.15.118:6803/296472230","heartbeat_front_addr":"172.21.15.118:6802/296472230","state":["exists","up"]},{"osd":5,"uuid":"bff24605-38ce-47ce-9142-4ced3766476e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":326385780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":326385780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":326385780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":326385780}]},"public_addr":"172.21.15.118:6804/326385780","cluster_addr":"172.21.15.118:6805/326385780","heartbeat_back_addr":"172.21.15.118:6807/326385780","heartbeat_front_addr":"172.21.15.118:6806/326385780","state":["exists","up"]},{"osd":6,"uuid":"73a781e7-8660-4885-b976-2aeb8515109a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1322208109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":1322208109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":1322208109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1322208109}]},"public_addr":"172.21.15.118:6808/1322208109","cluster_addr":"172.21.15.118:6809/1322208109","heartbeat_back_addr":"172.21.15.118:6811/1322208109","heartbeat_front_addr":"172.21.15.118:6810/1322208109","state":["exists","up"]},{"osd":7,"uuid":"3e95559b-79f6-40ba-9d98-d350bcdd60e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2128678671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":2128678671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":2128678671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2128678671}]},"public_addr":"172.21.15.118:6812/2128678671","cluster_addr":"172.21.15.118:6813/2128678671","heartbeat_back_addr":"172.21.15.118:6815/2128678671","heartbeat_front_addr":"172.21.15.118:6814/2128678671","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:03:05.182913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:21.152955+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:38.100201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:06.539457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:19.486841+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:38.038171+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:01:51.130228+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:17.436393+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3884352348":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2319227492":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2154949954":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/4162970769":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/2292819669":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/520312891":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2267426691":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/3211426112":"2024-03-22T13:57:56.576834+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T14:03:05.804 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:05 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:03:05.804 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:05 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:03:05.804 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:05 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:05.804 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:03:05 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:03:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:05 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:03:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:05 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:03:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:05 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:05 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:03:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:05 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:03:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:05 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:06.570 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T14:03:06.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T14:03:06.823 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:06 smithi118 ceph-mon[33862]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T14:03:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:06 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1263990836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:03:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:06 smithi118 ceph-mon[33862]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71 B/s, 2 objects/s recovering 2024-03-21T14:03:06.931 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:03:06 smithi118 systemd[1]: Started Ceph prometheus.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:03:06.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:06 smithi003 ceph-mon[29576]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T14:03:06.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:06 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1263990836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:03:06.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:06 smithi003 ceph-mon[29576]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71 B/s, 2 objects/s recovering 2024-03-21T14:03:06.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:06 smithi003 ceph-mon[40510]: osdmap e76: 8 total, 8 up, 8 in 2024-03-21T14:03:06.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:06 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1263990836' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:03:06.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:06 smithi003 ceph-mon[40510]: pgmap v221: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 71 B/s, 2 objects/s recovering 2024-03-21T14:03:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ignoring --setuser ceph since I am not root 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ignoring --setgroup ceph since I am not root 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:07 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: ignoring --setuser ceph since I am not root 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: ignoring --setgroup ceph since I am not root 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-mgr[35715]: -- 172.21.15.118:0/830569079 <== mon.1 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ae094a65a0 con 0x55ae0947b000 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' 2024-03-21T14:03:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:07 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-21T14:03:08.681 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:08 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:08.347+0000 7f338038c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T14:03:08.681 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:08 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:08.439+0000 7f338038c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T14:03:08.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:08.386+0000 7fa2054b1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-21T14:03:08.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:08.483+0000 7fa2054b1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-21T14:03:08.766 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:03:08.766 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":76,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","created":"2024-03-21T13:56:54.530125+0000","modified":"2024-03-21T14:03:05.699216+0000","last_up_change":"2024-03-21T14:02:19.321454+0000","last_in_change":"2024-03-21T14:02:01.236118+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T14:00:12.925535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T14:02:31.373659+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T14:02:31.794831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T14:02:33.750394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T14:02:35.711498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T14:02:37.763124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"daf9bc3c-4e73-4379-afb6-7ee181b686fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":930096433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":930096433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":930096433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":930096433}]},"public_addr":"172.21.15.3:6801/930096433","cluster_addr":"172.21.15.3:6802/930096433","heartbeat_back_addr":"172.21.15.3:6804/930096433","heartbeat_front_addr":"172.21.15.3:6803/930096433","state":["exists","up"]},{"osd":1,"uuid":"1652f24d-aa0e-471c-bf54-03814b1ee11d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1526485482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1526485482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1526485482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1526485482}]},"public_addr":"172.21.15.3:6805/1526485482","cluster_addr":"172.21.15.3:6806/1526485482","heartbeat_back_addr":"172.21.15.3:6808/1526485482","heartbeat_front_addr":"172.21.15.3:6807/1526485482","state":["exists","up"]},{"osd":2,"uuid":"5e67d907-f8d1-4ecc-85d3-df88da5c364c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3751378277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3751378277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3751378277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3751378277}]},"public_addr":"172.21.15.3:6809/3751378277","cluster_addr":"172.21.15.3:6810/3751378277","heartbeat_back_addr":"172.21.15.3:6812/3751378277","heartbeat_front_addr":"172.21.15.3:6811/3751378277","state":["exists","up"]},{"osd":3,"uuid":"ad8dabdf-fce5-4312-9e80-dbbb6ba2947c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":1918209225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1918209225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1918209225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":1918209225}]},"public_addr":"172.21.15.3:6813/1918209225","cluster_addr":"172.21.15.3:6814/1918209225","heartbeat_back_addr":"172.21.15.3:6816/1918209225","heartbeat_front_addr":"172.21.15.3:6815/1918209225","state":["exists","up"]},{"osd":4,"uuid":"b5d486b1-2cb6-46c7-8664-994af07ff7b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":296472230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":296472230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":296472230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":296472230}]},"public_addr":"172.21.15.118:6800/296472230","cluster_addr":"172.21.15.118:6801/296472230","heartbeat_back_addr":"172.21.15.118:6803/296472230","heartbeat_front_addr":"172.21.15.118:6802/296472230","state":["exists","up"]},{"osd":5,"uuid":"bff24605-38ce-47ce-9142-4ced3766476e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":326385780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":326385780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":326385780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":326385780}]},"public_addr":"172.21.15.118:6804/326385780","cluster_addr":"172.21.15.118:6805/326385780","heartbeat_back_addr":"172.21.15.118:6807/326385780","heartbeat_front_addr":"172.21.15.118:6806/326385780","state":["exists","up"]},{"osd":6,"uuid":"73a781e7-8660-4885-b976-2aeb8515109a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1322208109}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":1322208109}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":1322208109}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1322208109}]},"public_addr":"172.21.15.118:6808/1322208109","cluster_addr":"172.21.15.118:6809/1322208109","heartbeat_back_addr":"172.21.15.118:6811/1322208109","heartbeat_front_addr":"172.21.15.118:6810/1322208109","state":["exists","up"]},{"osd":7,"uuid":"3e95559b-79f6-40ba-9d98-d350bcdd60e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2128678671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":2128678671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":2128678671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2128678671}]},"public_addr":"172.21.15.118:6812/2128678671","cluster_addr":"172.21.15.118:6813/2128678671","heartbeat_back_addr":"172.21.15.118:6815/2128678671","heartbeat_front_addr":"172.21.15.118:6814/2128678671","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:03:05.182913+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:21.152955+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:38.100201+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:06.539457+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:19.486841+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:38.038171+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:01:51.130228+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:02:17.436393+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/3884352348":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2319227492":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2154949954":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/4162970769":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/2292819669":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/520312891":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2267426691":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/3211426112":"2024-03-22T13:57:56.576834+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T14:03:09.072 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:08 smithi118 ceph-mon[33862]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T14:03:09.072 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:08 smithi118 ceph-mon[33862]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T14:03:09.073 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:08 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3308782880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:03:09.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-mon[29576]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T14:03:09.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-mon[29576]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T14:03:09.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3308782880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:03:09.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-mon[40510]: from='mgr.14152 172.21.15.3:0/472829056' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-21T14:03:09.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-mon[40510]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-21T14:03:09.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:08 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3308782880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:03:09.338 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.071+0000 7f338038c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T14:03:09.338 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.167+0000 7f338038c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T14:03:09.339 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.311+0000 7f338038c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T14:03:09.456 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:09.161+0000 7fa2054b1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-21T14:03:09.456 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:09.257+0000 7fa2054b1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-21T14:03:09.456 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:09.402+0000 7fa2054b1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-21T14:03:09.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T14:03:09.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T14:03:09.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T14:03:09.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T14:03:09.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T14:03:09.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T14:03:09.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T14:03:09.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T14:03:09.681 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.411+0000 7f338038c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T14:03:09.681 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.502+0000 7f338038c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T14:03:09.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:09.516+0000 7fa2054b1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-21T14:03:09.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:09.672+0000 7fa2054b1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-21T14:03:10.029 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:10.030 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:10.031 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:10.071 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.775+0000 7f338038c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T14:03:10.071 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.886+0000 7f338038c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T14:03:10.071 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:09 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:09.974+0000 7f338038c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T14:03:10.071 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:10.069+0000 7f338038c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T14:03:10.079 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:10.080 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:10.388 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:10.106+0000 7fa2054b1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-21T14:03:10.388 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:10.222+0000 7fa2054b1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-21T14:03:10.388 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:10.318+0000 7fa2054b1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-21T14:03:10.431 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:10.246+0000 7f338038c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T14:03:10.645 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:10.420+0000 7fa2054b1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-21T14:03:10.645 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:10.607+0000 7fa2054b1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-21T14:03:11.214 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:11.653 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:11 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:11.194+0000 7f338038c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T14:03:11.654 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:11 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:11.321+0000 7f338038c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T14:03:11.654 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:11 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:11.652+0000 7f338038c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T14:03:11.931 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:11 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:11.743+0000 7f338038c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T14:03:12.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:11 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:11.658+0000 7fa2054b1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-21T14:03:12.004 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:11 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:11.792+0000 7fa2054b1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-21T14:03:12.413 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:11 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:11.937+0000 7f338038c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T14:03:12.413 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:12.124+0000 7f338038c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T14:03:12.431 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:12.134+0000 7fa2054b1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-21T14:03:12.431 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:12.228+0000 7fa2054b1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-21T14:03:12.431 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:12.430+0000 7fa2054b1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-21T14:03:12.670 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:12.413+0000 7f338038c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T14:03:12.690 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:12.621+0000 7fa2054b1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-21T14:03:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:12.934 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:12.670+0000 7f338038c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T14:03:12.934 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:12.910+0000 7f338038c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T14:03:12.974 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:12.926+0000 7fa2054b1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-21T14:03:13.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:13 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:13.190+0000 7fa2054b1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-21T14:03:13.633 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:13 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:13.444+0000 7fa2054b1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-21T14:03:14.181 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:13.716+0000 7f338038c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T14:03:14.625 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:14.310+0000 7f338038c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T14:03:14.625 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:14.409+0000 7f338038c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T14:03:14.625 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:14.625+0000 7f338038c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T14:03:14.698 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:14.292+0000 7fa2054b1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-mon[33862]: Standby manager daemon x restarted 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-mon[33862]: Standby manager daemon x started 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-mon[33862]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T14:03:14.931 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: [21/Mar/2024:14:03:14] ENGINE Bus STARTING 2024-03-21T14:03:14.932 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: CherryPy Checker: 2024-03-21T14:03:14.932 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: The Application mounted at '' has an empty config. 2024-03-21T14:03:14.932 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: 2024-03-21T14:03:14.932 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: [21/Mar/2024:14:03:14] ENGINE Serving on http://:::9283 2024-03-21T14:03:14.932 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:03:14 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x[35711]: [21/Mar/2024:14:03:14] ENGINE Bus STARTED 2024-03-21T14:03:14.992 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:14.890+0000 7fa2054b1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-21T14:03:14.992 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:14.992+0000 7fa2054b1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[29576]: Standby manager daemon x restarted 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[29576]: Standby manager daemon x started 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[29576]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[40510]: Standby manager daemon x restarted 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[40510]: Standby manager daemon x started 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-21T14:03:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:14 smithi003 ceph-mon[40510]: from='mgr.? 172.21.15.118:0/3790309492' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-21T14:03:15.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:15.225+0000 7fa2054b1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:15] ENGINE Bus STARTING 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: CherryPy Checker: 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: The Application mounted at '' has an empty config. 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:15] ENGINE Serving on http://:::9283 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:15] ENGINE Bus STARTED 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T14:03:15.700 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: Active manager daemon y restarted 2024-03-21T14:03:15.701 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: Activating manager daemon y 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: Active manager daemon y restarted 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: Activating manager daemon y 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: mgrmap e19: y(active, starting, since 0.00919227s), standbys: x 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:03:15.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T14:03:15.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: Manager daemon y is now available 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: mgrmap e19: y(active, starting, since 0.00919227s), standbys: x 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:03:15.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: Manager daemon y is now available 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:15.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:15.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T14:03:15.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T14:03:15.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T14:03:15.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T14:03:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-21T14:03:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: Active manager daemon y restarted 2024-03-21T14:03:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: Activating manager daemon y 2024-03-21T14:03:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: osdmap e77: 8 total, 8 up, 8 in 2024-03-21T14:03:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: mgrmap e19: y(active, starting, since 0.00919227s), standbys: x 2024-03-21T14:03:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-21T14:03:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: Manager daemon y is now available 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T14:03:16.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-21T14:03:16.561 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:03:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:03:16.744 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-mon[29576]: [21/Mar/2024:14:03:15] ENGINE Bus STARTING 2024-03-21T14:03:16.744 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-mon[29576]: [21/Mar/2024:14:03:15] ENGINE Serving on https://172.21.15.3:7150 2024-03-21T14:03:16.744 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-mon[29576]: mgrmap e20: y(active, since 1.01259s), standbys: x 2024-03-21T14:03:16.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-mon[40510]: [21/Mar/2024:14:03:15] ENGINE Bus STARTING 2024-03-21T14:03:16.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-mon[40510]: [21/Mar/2024:14:03:15] ENGINE Serving on https://172.21.15.3:7150 2024-03-21T14:03:16.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-mon[40510]: mgrmap e20: y(active, since 1.01259s), standbys: x 2024-03-21T14:03:16.745 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:03:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:03:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:16 smithi118 ceph-mon[33862]: [21/Mar/2024:14:03:15] ENGINE Bus STARTING 2024-03-21T14:03:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:16 smithi118 ceph-mon[33862]: [21/Mar/2024:14:03:15] ENGINE Serving on https://172.21.15.3:7150 2024-03-21T14:03:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:16 smithi118 ceph-mon[33862]: mgrmap e20: y(active, since 1.01259s), standbys: x 2024-03-21T14:03:17.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[29576]: [21/Mar/2024:14:03:15] ENGINE Serving on http://172.21.15.3:8765 2024-03-21T14:03:17.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[29576]: [21/Mar/2024:14:03:15] ENGINE Bus STARTED 2024-03-21T14:03:17.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:17.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[29576]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:03:17.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[40510]: [21/Mar/2024:14:03:15] ENGINE Serving on http://172.21.15.3:8765 2024-03-21T14:03:17.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[40510]: [21/Mar/2024:14:03:15] ENGINE Bus STARTED 2024-03-21T14:03:17.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:17.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:17 smithi003 ceph-mon[40510]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:03:18.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:17 smithi118 ceph-mon[33862]: [21/Mar/2024:14:03:15] ENGINE Serving on http://172.21.15.3:8765 2024-03-21T14:03:18.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:17 smithi118 ceph-mon[33862]: [21/Mar/2024:14:03:15] ENGINE Bus STARTED 2024-03-21T14:03:18.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:18.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:17 smithi118 ceph-mon[33862]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:03:18.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:18 smithi003 ceph-mon[29576]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:03:18.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:18 smithi003 ceph-mon[29576]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-21T14:03:18.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:18 smithi003 ceph-mon[40510]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:03:18.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:18 smithi003 ceph-mon[40510]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-21T14:03:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:18 smithi118 ceph-mon[33862]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-21T14:03:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:18 smithi118 ceph-mon[33862]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-21T14:03:19.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:19.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:19.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:19 smithi003 ceph-mon[29576]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T14:03:19.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:19.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:19.842 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:19 smithi003 ceph-mon[40510]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T14:03:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:19 smithi118 ceph-mon[33862]: osdmap e78: 8 total, 8 up, 8 in 2024-03-21T14:03:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-21T14:03:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi118 to 3760M 2024-03-21T14:03:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi118 to 3760M 2024-03-21T14:03:21.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 0 B/s wr, 39 op/s 2024-03-21T14:03:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-21T14:03:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi118 to 3760M 2024-03-21T14:03:21.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:22.143 INFO:teuthology.orchestra.run.smithi003.stdout:34359738418 2024-03-21T14:03:22.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T14:03:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:03:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:03:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[29576]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-21T14:03:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[29576]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T14:03:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:03:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:03:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[40510]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-21T14:03:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:22 smithi003 ceph-mon[40510]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T14:03:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:22 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:03:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:22 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:03:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:22 smithi118 ceph-mon[33862]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-21T14:03:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:22 smithi118 ceph-mon[33862]: osdmap e79: 8 total, 8 up, 8 in 2024-03-21T14:03:23.885 INFO:teuthology.orchestra.run.smithi003.stdout:77309411368 2024-03-21T14:03:23.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T14:03:24.380 INFO:teuthology.orchestra.run.smithi003.stdout:206158430226 2024-03-21T14:03:24.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T14:03:24.771 INFO:teuthology.orchestra.run.smithi003.stdout:249108103181 2024-03-21T14:03:24.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T14:03:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:24 smithi003 ceph-mon[29576]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T14:03:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:24 smithi003 ceph-mon[29576]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T14:03:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:24 smithi003 ceph-mon[40510]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T14:03:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:24 smithi003 ceph-mon[40510]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T14:03:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:24 smithi118 ceph-mon[33862]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-21T14:03:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:24 smithi118 ceph-mon[33862]: osdmap e80: 8 total, 8 up, 8 in 2024-03-21T14:03:25.393 INFO:teuthology.orchestra.run.smithi003.stdout:128849018910 2024-03-21T14:03:25.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T14:03:26.378 INFO:teuthology.orchestra.run.smithi003.stdout:55834574894 2024-03-21T14:03:26.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T14:03:26.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:03:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:03:26.744 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:03:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:03:26.829 INFO:teuthology.orchestra.run.smithi003.stdout:107374182433 2024-03-21T14:03:26.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T14:03:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:26 smithi003 ceph-mon[29576]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T14:03:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:26 smithi003 ceph-mon[40510]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T14:03:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:26 smithi118 ceph-mon[33862]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T14:03:27.205 INFO:teuthology.orchestra.run.smithi003.stdout:171798691865 2024-03-21T14:03:27.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T14:03:27.229 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:27.260 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:28.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:28 smithi003 ceph-mon[29576]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:03:29.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:28 smithi003 ceph-mon[40510]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:03:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:28 smithi118 ceph-mon[33862]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:03:29.611 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:30 smithi118 ceph-mon[33862]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:03:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:30 smithi118 ceph-mon[33862]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T14:03:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:30 smithi003 ceph-mon[29576]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:03:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:30 smithi003 ceph-mon[29576]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T14:03:30.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:30 smithi003 ceph-mon[40510]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:03:30.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:30 smithi003 ceph-mon[40510]: osdmap e81: 8 total, 8 up, 8 in 2024-03-21T14:03:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:31 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:03:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:31 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:03:31.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:31.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:31 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:03:31.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:31 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:03:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:31 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:03:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:31 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:32 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:32 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:32 smithi118 ceph-mon[33862]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:32.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:32 smithi118 ceph-mon[33862]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T14:03:32.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[29576]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[29576]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[40510]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:32.702 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:32 smithi003 ceph-mon[40510]: osdmap e82: 8 total, 8 up, 8 in 2024-03-21T14:03:33.481 INFO:teuthology.orchestra.run.smithi003.stdout:206158430229 2024-03-21T14:03:34.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:34 smithi118 ceph-mon[33862]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:34.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/867414018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T14:03:34.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:34 smithi118 ceph-mon[33862]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T14:03:34.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:34 smithi003 ceph-mon[29576]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:34.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/867414018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T14:03:34.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:34 smithi003 ceph-mon[29576]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T14:03:34.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:34 smithi003 ceph-mon[40510]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:34.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/867414018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T14:03:34.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:34 smithi003 ceph-mon[40510]: osdmap e83: 8 total, 8 up, 8 in 2024-03-21T14:03:35.331 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430226 got 206158430229 for osd.6 2024-03-21T14:03:35.332 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:35 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:03:35.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:35 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:03:35.686 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:35 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:03:35.687 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:35 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:03:35.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:35 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:03:35.687 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:35 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:03:36.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:03:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:03:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:36 smithi118 ceph-mon[33862]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:03:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:36 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:03:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:36 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:03:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:36 smithi118 ceph-mon[33862]: 4.19 deep-scrub starts 2024-03-21T14:03:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:36 smithi118 ceph-mon[33862]: 4.19 deep-scrub ok 2024-03-21T14:03:36.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[29576]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:03:36.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:03:36.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:03:36.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[29576]: 4.19 deep-scrub starts 2024-03-21T14:03:36.696 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[29576]: 4.19 deep-scrub ok 2024-03-21T14:03:36.697 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:03:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:03:36.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[40510]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:03:36.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:03:36.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:03:36.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[40510]: 4.19 deep-scrub starts 2024-03-21T14:03:36.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:36 smithi003 ceph-mon[40510]: 4.19 deep-scrub ok 2024-03-21T14:03:37.018 INFO:teuthology.orchestra.run.smithi003.stdout:55834574896 2024-03-21T14:03:37.238 INFO:teuthology.orchestra.run.smithi003.stdout:34359738421 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[29576]: 3.7 deep-scrub starts 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[29576]: 3.7 deep-scrub ok 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/913662329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2757154968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[40510]: 3.7 deep-scrub starts 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[40510]: 3.7 deep-scrub ok 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/913662329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T14:03:37.645 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2757154968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T14:03:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:37 smithi118 ceph-mon[33862]: 3.7 deep-scrub starts 2024-03-21T14:03:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:37 smithi118 ceph-mon[33862]: 3.7 deep-scrub ok 2024-03-21T14:03:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/913662329' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T14:03:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2757154968' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T14:03:38.371 INFO:teuthology.orchestra.run.smithi003.stdout:77309411371 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[29576]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[40510]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:03:38.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:03:38.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:03:38.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:03:38.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:38 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:03:38.567 INFO:teuthology.orchestra.run.smithi003.stdout:171798691868 2024-03-21T14:03:38.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:38 smithi118 ceph-mon[33862]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:03:38.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:38 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:03:38.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:38 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:03:38.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:38 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:03:38.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:38 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:03:38.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574894 got 55834574896 for osd.1 2024-03-21T14:03:38.905 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2358771865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2272982485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:03:39.576 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2358771865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T14:03:39.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2272982485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T14:03:39.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:03:39.577 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:39 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:03:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:39 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:03:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:39 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:03:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2358771865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T14:03:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2272982485' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T14:03:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:39 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:03:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:39 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:03:39.681 INFO:teuthology.orchestra.run.smithi003.stdout:249108103185 2024-03-21T14:03:40.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:03:40.431 INFO:teuthology.orchestra.run.smithi003.stdout:128849018913 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:03:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:40.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T14:03:40.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 3.18 deep-scrub starts 2024-03-21T14:03:40.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: 3.18 deep-scrub ok 2024-03-21T14:03:40.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:40 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2605595181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T14:03:40.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:03:40.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:03:40.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:03:40.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:03:40.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 3.18 deep-scrub starts 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: 3.18 deep-scrub ok 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2605595181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:03:40.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:03:40.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:03:40.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:40.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: osdmap e84: 8 total, 8 up, 8 in 2024-03-21T14:03:40.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 3.18 deep-scrub starts 2024-03-21T14:03:40.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: 3.18 deep-scrub ok 2024-03-21T14:03:40.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2605595181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T14:03:41.162 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:41.190 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-03-21T14:03:41.190 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:41 smithi003 ceph-mon[29576]: 4.7 deep-scrub starts 2024-03-21T14:03:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:41 smithi003 ceph-mon[29576]: 4.7 deep-scrub ok 2024-03-21T14:03:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1726298222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T14:03:41.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:41 smithi003 ceph-mon[40510]: 4.7 deep-scrub starts 2024-03-21T14:03:41.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:41 smithi003 ceph-mon[40510]: 4.7 deep-scrub ok 2024-03-21T14:03:41.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1726298222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T14:03:41.617 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738421 for osd.0 2024-03-21T14:03:41.617 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:41 smithi118 ceph-mon[33862]: 4.7 deep-scrub starts 2024-03-21T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:41 smithi118 ceph-mon[33862]: 4.7 deep-scrub ok 2024-03-21T14:03:41.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1726298222' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T14:03:42.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018913 for osd.4 2024-03-21T14:03:42.256 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[29576]: 5.a deep-scrub starts 2024-03-21T14:03:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[29576]: 5.a deep-scrub ok 2024-03-21T14:03:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[29576]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:03:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:03:42.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[40510]: 5.a deep-scrub starts 2024-03-21T14:03:42.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[40510]: 5.a deep-scrub ok 2024-03-21T14:03:42.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[40510]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:42.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:03:42.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:42 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:03:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:42 smithi118 ceph-mon[33862]: 5.a deep-scrub starts 2024-03-21T14:03:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:42 smithi118 ceph-mon[33862]: 5.a deep-scrub ok 2024-03-21T14:03:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:42 smithi118 ceph-mon[33862]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:42 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:03:42.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:42 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:03:43.307 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691865 got 171798691868 for osd.5 2024-03-21T14:03:43.307 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:43.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:43.426 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:43 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: osdmap e85: 8 total, 8 up, 8 in 2024-03-21T14:03:43.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:43.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:43 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:44.162 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103181 got 249108103185 for osd.7 2024-03-21T14:03:44.162 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:44.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:44 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:03:44.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:44 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:03:44.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:44 smithi118 ceph-mon[33862]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:44.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:44 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:03:44.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:44 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[29576]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[40510]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:03:44.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:44 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:03:44.943 INFO:teuthology.orchestra.run.smithi003.stdout:107374182438 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.698 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi003 to 3761M 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/46097429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.699 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi003 to 3761M 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/46097429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.701 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.732 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182433 got 107374182438 for osd.3 2024-03-21T14:03:45.732 DEBUG:teuthology.parallel:result is None 2024-03-21T14:03:45.732 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T14:03:45.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi003 to 3761M 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: Updating smithi003:/etc/ceph/ceph.conf 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/46097429' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:45.985 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:46.592 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:03:46.737 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:03:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:03:46.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:03:46.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:03:46.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-21T14:03:46.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-21T14:03:46.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: Deploying daemon alertmanager.a on smithi003 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:03:46.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: Deploying daemon alertmanager.a on smithi003 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:03:46.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:46 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: Updating smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: Updating smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:03:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:03:46.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:03:46.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: Deploying daemon alertmanager.a on smithi003 2024-03-21T14:03:46.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:03:46.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:46 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:03:47.598 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:47.598 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:03:47.598 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:03:47.598 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:03:47.598 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:03:47.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:03:47.795 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:03:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:47 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:03:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:47 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:03:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:47 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:03:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:47 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:03:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:03:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:03:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:03:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:47 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:03:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:48 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:03:48.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:03:48.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:03:48.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:03:48.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:03:48.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:03:48.956 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:03:48.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:03:48.957 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:48 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:03:48.989 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-03-21T14:03:47.248409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1000,"num_read_kb":846,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":613,"ondisk_log_size":613,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253076,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708140,"statfs":{"total":767960285184,"available":767701135360,"internally_reserved":0,"allocated":7991296,"data_stored":4460448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001531"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389060+0000","last_change":"2024-03-21T14:02:39.463103+0000","last_active":"2024-03-21T14:03:42.389060+0000","last_peered":"2024-03-21T14:03:42.389060+0000","last_clean":"2024-03-21T14:03:42.389060+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:03:42.389060+0000","last_undegraded":"2024-03-21T14:03:42.389060+0000","last_fullsized":"2024-03-21T14:03:42.389060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.364725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756817+0000","last_change":"2024-03-21T14:02:37.454690+0000","last_active":"2024-03-21T14:03:43.756817+0000","last_peered":"2024-03-21T14:03:43.756817+0000","last_clean":"2024-03-21T14:03:43.756817+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:03:43.756817+0000","last_undegraded":"2024-03-21T14:03:43.756817+0000","last_fullsized":"2024-03-21T14:03:43.756817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.175742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996444+0000","last_change":"2024-03-21T14:02:33.442705+0000","last_active":"2024-03-21T14:03:42.996444+0000","last_peered":"2024-03-21T14:03:42.996444+0000","last_clean":"2024-03-21T14:03:42.996444+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:03:42.996444+0000","last_undegraded":"2024-03-21T14:03:42.996444+0000","last_fullsized":"2024-03-21T14:03:42.996444+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:44.437990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388929+0000","last_change":"2024-03-21T14:02:35.450207+0000","last_active":"2024-03-21T14:03:42.388929+0000","last_peered":"2024-03-21T14:03:42.388929+0000","last_clean":"2024-03-21T14:03:42.388929+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:03:42.388929+0000","last_undegraded":"2024-03-21T14:03:42.388929+0000","last_fullsized":"2024-03-21T14:03:42.388929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.595411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395241+0000","last_change":"2024-03-21T14:02:39.517587+0000","last_active":"2024-03-21T14:03:42.395241+0000","last_peered":"2024-03-21T14:03:42.395241+0000","last_clean":"2024-03-21T14:03:42.395241+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:03:42.395241+0000","last_undegraded":"2024-03-21T14:03:42.395241+0000","last_fullsized":"2024-03-21T14:03:42.395241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.320905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388940+0000","last_change":"2024-03-21T14:02:37.458374+0000","last_active":"2024-03-21T14:03:42.388940+0000","last_peered":"2024-03-21T14:03:42.388940+0000","last_clean":"2024-03-21T14:03:42.388940+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:03:42.388940+0000","last_undegraded":"2024-03-21T14:03:42.388940+0000","last_fullsized":"2024-03-21T14:03:42.388940+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.406467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396378+0000","last_change":"2024-03-21T14:03:37.567249+0000","last_active":"2024-03-21T14:03:42.396378+0000","last_peered":"2024-03-21T14:03:42.396378+0000","last_clean":"2024-03-21T14:03:42.396378+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:03:42.396378+0000","last_undegraded":"2024-03-21T14:03:42.396378+0000","last_fullsized":"2024-03-21T14:03:42.396378+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:37.567167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.567167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.628456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394378+0000","last_change":"2024-03-21T14:03:35.571620+0000","last_active":"2024-03-21T14:03:42.394378+0000","last_peered":"2024-03-21T14:03:42.394378+0000","last_clean":"2024-03-21T14:03:42.394378+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:03:42.394378+0000","last_undegraded":"2024-03-21T14:03:42.394378+0000","last_fullsized":"2024-03-21T14:03:42.394378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.571531+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:41.701221+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995806+0000","last_change":"2024-03-21T14:02:39.469561+0000","last_active":"2024-03-21T14:03:42.995806+0000","last_peered":"2024-03-21T14:03:42.995806+0000","last_clean":"2024-03-21T14:03:42.995806+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:03:42.995806+0000","last_undegraded":"2024-03-21T14:03:42.995806+0000","last_fullsized":"2024-03-21T14:03:42.995806+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.240000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999035+0000","last_change":"2024-03-21T14:02:37.455588+0000","last_active":"2024-03-21T14:03:42.999035+0000","last_peered":"2024-03-21T14:03:42.999035+0000","last_clean":"2024-03-21T14:03:42.999035+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:03:42.999035+0000","last_undegraded":"2024-03-21T14:03:42.999035+0000","last_fullsized":"2024-03-21T14:03:42.999035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.535187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999090+0000","last_change":"2024-03-21T14:02:33.441047+0000","last_active":"2024-03-21T14:03:42.999090+0000","last_peered":"2024-03-21T14:03:42.999090+0000","last_clean":"2024-03-21T14:03:42.999090+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:03:42.999090+0000","last_undegraded":"2024-03-21T14:03:42.999090+0000","last_fullsized":"2024-03-21T14:03:42.999090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.089986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389241+0000","last_change":"2024-03-21T14:02:35.445770+0000","last_active":"2024-03-21T14:03:42.389241+0000","last_peered":"2024-03-21T14:03:42.389241+0000","last_clean":"2024-03-21T14:03:42.389241+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:03:42.389241+0000","last_undegraded":"2024-03-21T14:03:42.389241+0000","last_fullsized":"2024-03-21T14:03:42.389241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.203589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998655+0000","last_change":"2024-03-21T14:02:39.466860+0000","last_active":"2024-03-21T14:03:42.998655+0000","last_peered":"2024-03-21T14:03:42.998655+0000","last_clean":"2024-03-21T14:03:42.998655+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:03:42.998655+0000","last_undegraded":"2024-03-21T14:03:42.998655+0000","last_fullsized":"2024-03-21T14:03:42.998655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.470502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532890+0000","last_change":"2024-03-21T14:02:37.457212+0000","last_active":"2024-03-21T14:03:43.532890+0000","last_peered":"2024-03-21T14:03:43.532890+0000","last_clean":"2024-03-21T14:03:43.532890+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:03:43.532890+0000","last_undegraded":"2024-03-21T14:03:43.532890+0000","last_fullsized":"2024-03-21T14:03:43.532890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999073+0000","last_change":"2024-03-21T14:02:33.441980+0000","last_active":"2024-03-21T14:03:42.999073+0000","last_peered":"2024-03-21T14:03:42.999073+0000","last_clean":"2024-03-21T14:03:42.999073+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:03:42.999073+0000","last_undegraded":"2024-03-21T14:03:42.999073+0000","last_fullsized":"2024-03-21T14:03:42.999073+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.536834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389583+0000","last_change":"2024-03-21T14:02:35.453933+0000","last_active":"2024-03-21T14:03:42.389583+0000","last_peered":"2024-03-21T14:03:42.389583+0000","last_clean":"2024-03-21T14:03:42.389583+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:03:42.389583+0000","last_undegraded":"2024-03-21T14:03:42.389583+0000","last_fullsized":"2024-03-21T14:03:42.389583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.127706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389793+0000","last_change":"2024-03-21T14:02:39.516305+0000","last_active":"2024-03-21T14:03:42.389793+0000","last_peered":"2024-03-21T14:03:42.389793+0000","last_clean":"2024-03-21T14:03:42.389793+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:03:42.389793+0000","last_undegraded":"2024-03-21T14:03:42.389793+0000","last_fullsized":"2024-03-21T14:03:42.389793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.252625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757380+0000","last_change":"2024-03-21T14:02:37.459191+0000","last_active":"2024-03-21T14:03:43.757380+0000","last_peered":"2024-03-21T14:03:43.757380+0000","last_clean":"2024-03-21T14:03:43.757380+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:03:43.757380+0000","last_undegraded":"2024-03-21T14:03:43.757380+0000","last_fullsized":"2024-03-21T14:03:43.757380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.913816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996252+0000","last_change":"2024-03-21T14:02:33.447014+0000","last_active":"2024-03-21T14:03:42.996252+0000","last_peered":"2024-03-21T14:03:42.996252+0000","last_clean":"2024-03-21T14:03:42.996252+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:03:42.996252+0000","last_undegraded":"2024-03-21T14:03:42.996252+0000","last_fullsized":"2024-03-21T14:03:42.996252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.197179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395670+0000","last_change":"2024-03-21T14:02:35.454150+0000","last_active":"2024-03-21T14:03:42.395670+0000","last_peered":"2024-03-21T14:03:42.395670+0000","last_clean":"2024-03-21T14:03:42.395670+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:03:42.395670+0000","last_undegraded":"2024-03-21T14:03:42.395670+0000","last_fullsized":"2024-03-21T14:03:42.395670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.304290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395547+0000","last_change":"2024-03-21T14:02:39.517386+0000","last_active":"2024-03-21T14:03:42.395547+0000","last_peered":"2024-03-21T14:03:42.395547+0000","last_clean":"2024-03-21T14:03:42.395547+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:03:42.395547+0000","last_undegraded":"2024-03-21T14:03:42.395547+0000","last_fullsized":"2024-03-21T14:03:42.395547+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.220059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389360+0000","last_change":"2024-03-21T14:02:37.458298+0000","last_active":"2024-03-21T14:03:42.389360+0000","last_peered":"2024-03-21T14:03:42.389360+0000","last_clean":"2024-03-21T14:03:42.389360+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:03:42.389360+0000","last_undegraded":"2024-03-21T14:03:42.389360+0000","last_fullsized":"2024-03-21T14:03:42.389360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.474405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389477+0000","last_change":"2024-03-21T14:02:33.434594+0000","last_active":"2024-03-21T14:03:42.389477+0000","last_peered":"2024-03-21T14:03:42.389477+0000","last_clean":"2024-03-21T14:03:42.389477+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:03:42.389477+0000","last_undegraded":"2024-03-21T14:03:42.389477+0000","last_fullsized":"2024-03-21T14:03:42.389477+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:43.240294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395701+0000","last_change":"2024-03-21T14:02:35.451688+0000","last_active":"2024-03-21T14:03:42.395701+0000","last_peered":"2024-03-21T14:03:42.395701+0000","last_clean":"2024-03-21T14:03:42.395701+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:03:42.395701+0000","last_undegraded":"2024-03-21T14:03:42.395701+0000","last_fullsized":"2024-03-21T14:03:42.395701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.784537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532313+0000","last_change":"2024-03-21T14:02:39.466190+0000","last_active":"2024-03-21T14:03:43.532313+0000","last_peered":"2024-03-21T14:03:43.532313+0000","last_clean":"2024-03-21T14:03:43.532313+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:03:43.532313+0000","last_undegraded":"2024-03-21T14:03:43.532313+0000","last_fullsized":"2024-03-21T14:03:43.532313+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.968518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735515+0000","last_change":"2024-03-21T14:02:37.455355+0000","last_active":"2024-03-21T14:03:42.735515+0000","last_peered":"2024-03-21T14:03:42.735515+0000","last_clean":"2024-03-21T14:03:42.735515+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:03:42.735515+0000","last_undegraded":"2024-03-21T14:03:42.735515+0000","last_fullsized":"2024-03-21T14:03:42.735515+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.196417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'2","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756941+0000","last_change":"2024-03-21T14:02:33.441483+0000","last_active":"2024-03-21T14:03:43.756941+0000","last_peered":"2024-03-21T14:03:43.756941+0000","last_clean":"2024-03-21T14:03:43.756941+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:03:43.756941+0000","last_undegraded":"2024-03-21T14:03:43.756941+0000","last_fullsized":"2024-03-21T14:03:43.756941+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.043328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:44.904151+0000","last_change":"2024-03-21T14:03:44.904204+0000","last_active":"2024-03-21T14:03:44.904151+0000","last_peered":"2024-03-21T14:03:44.904151+0000","last_clean":"2024-03-21T14:03:44.904151+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:03:44.904151+0000","last_undegraded":"2024-03-21T14:03:44.904151+0000","last_fullsized":"2024-03-21T14:03:44.904151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:44.904071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:44.904071+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.807993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756221+0000","last_change":"2024-03-21T14:02:39.465886+0000","last_active":"2024-03-21T14:03:43.756221+0000","last_peered":"2024-03-21T14:03:43.756221+0000","last_clean":"2024-03-21T14:03:43.756221+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:03:43.756221+0000","last_undegraded":"2024-03-21T14:03:43.756221+0000","last_fullsized":"2024-03-21T14:03:43.756221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.933267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996574+0000","last_change":"2024-03-21T14:02:37.463347+0000","last_active":"2024-03-21T14:03:42.996574+0000","last_peered":"2024-03-21T14:03:42.996574+0000","last_clean":"2024-03-21T14:03:42.996574+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:03:42.996574+0000","last_undegraded":"2024-03-21T14:03:42.996574+0000","last_fullsized":"2024-03-21T14:03:42.996574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.394323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394109+0000","last_change":"2024-03-21T14:03:39.522391+0000","last_active":"2024-03-21T14:03:42.394109+0000","last_peered":"2024-03-21T14:03:42.394109+0000","last_clean":"2024-03-21T14:03:42.394109+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:03:42.394109+0000","last_undegraded":"2024-03-21T14:03:42.394109+0000","last_fullsized":"2024-03-21T14:03:42.394109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.522309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.085948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735454+0000","last_change":"2024-03-21T14:02:35.449454+0000","last_active":"2024-03-21T14:03:42.735454+0000","last_peered":"2024-03-21T14:03:42.735454+0000","last_clean":"2024-03-21T14:03:42.735454+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:03:42.735454+0000","last_undegraded":"2024-03-21T14:03:42.735454+0000","last_fullsized":"2024-03-21T14:03:42.735454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:43.505518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532253+0000","last_change":"2024-03-21T14:02:39.466841+0000","last_active":"2024-03-21T14:03:43.532253+0000","last_peered":"2024-03-21T14:03:43.532253+0000","last_clean":"2024-03-21T14:03:43.532253+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:03:43.532253+0000","last_undegraded":"2024-03-21T14:03:43.532253+0000","last_fullsized":"2024-03-21T14:03:43.532253+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.901031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734740+0000","last_change":"2024-03-21T14:02:37.456532+0000","last_active":"2024-03-21T14:03:42.734740+0000","last_peered":"2024-03-21T14:03:42.734740+0000","last_clean":"2024-03-21T14:03:42.734740+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:03:42.734740+0000","last_undegraded":"2024-03-21T14:03:42.734740+0000","last_fullsized":"2024-03-21T14:03:42.734740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.215037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996436+0000","last_change":"2024-03-21T14:02:33.442910+0000","last_active":"2024-03-21T14:03:42.996436+0000","last_peered":"2024-03-21T14:03:42.996436+0000","last_clean":"2024-03-21T14:03:42.996436+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:03:42.996436+0000","last_undegraded":"2024-03-21T14:03:42.996436+0000","last_fullsized":"2024-03-21T14:03:42.996436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:44.782621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.509555+0000","last_change":"2024-03-21T14:03:45.509584+0000","last_active":"2024-03-21T14:03:45.509555+0000","last_peered":"2024-03-21T14:03:45.509555+0000","last_clean":"2024-03-21T14:03:45.509555+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:03:45.509555+0000","last_undegraded":"2024-03-21T14:03:45.509555+0000","last_fullsized":"2024-03-21T14:03:45.509555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:03:45.509492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.509492+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.086854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395138+0000","last_change":"2024-03-21T14:02:39.517579+0000","last_active":"2024-03-21T14:03:42.395138+0000","last_peered":"2024-03-21T14:03:42.395138+0000","last_clean":"2024-03-21T14:03:42.395138+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:03:42.395138+0000","last_undegraded":"2024-03-21T14:03:42.395138+0000","last_fullsized":"2024-03-21T14:03:42.395138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.492381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533442+0000","last_change":"2024-03-21T14:02:37.455946+0000","last_active":"2024-03-21T14:03:43.533442+0000","last_peered":"2024-03-21T14:03:43.533442+0000","last_clean":"2024-03-21T14:03:43.533442+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:03:43.533442+0000","last_undegraded":"2024-03-21T14:03:43.533442+0000","last_fullsized":"2024-03-21T14:03:43.533442+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:44.058453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:03:41.362147+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:41.362005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.362005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:10.804533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394356+0000","last_change":"2024-03-21T14:02:35.773947+0000","last_active":"2024-03-21T14:03:42.394356+0000","last_peered":"2024-03-21T14:03:42.394356+0000","last_clean":"2024-03-21T14:03:42.394356+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:03:42.394356+0000","last_undegraded":"2024-03-21T14:03:42.394356+0000","last_fullsized":"2024-03-21T14:03:42.394356+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995540+0000","last_change":"2024-03-21T14:02:39.471220+0000","last_active":"2024-03-21T14:03:42.995540+0000","last_peered":"2024-03-21T14:03:42.995540+0000","last_clean":"2024-03-21T14:03:42.995540+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:03:42.995540+0000","last_undegraded":"2024-03-21T14:03:42.995540+0000","last_fullsized":"2024-03-21T14:03:42.995540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.470185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395282+0000","last_change":"2024-03-21T14:02:37.458457+0000","last_active":"2024-03-21T14:03:42.395282+0000","last_peered":"2024-03-21T14:03:42.395282+0000","last_clean":"2024-03-21T14:03:42.395282+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:03:42.395282+0000","last_undegraded":"2024-03-21T14:03:42.395282+0000","last_fullsized":"2024-03-21T14:03:42.395282+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533251+0000","last_change":"2024-03-21T14:02:33.449232+0000","last_active":"2024-03-21T14:03:43.533251+0000","last_peered":"2024-03-21T14:03:43.533251+0000","last_clean":"2024-03-21T14:03:43.533251+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:03:43.533251+0000","last_undegraded":"2024-03-21T14:03:43.533251+0000","last_fullsized":"2024-03-21T14:03:43.533251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.141490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.759191+0000","last_change":"2024-03-21T14:03:43.759222+0000","last_active":"2024-03-21T14:03:43.759191+0000","last_peered":"2024-03-21T14:03:43.759191+0000","last_clean":"2024-03-21T14:03:43.759191+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:03:43.759191+0000","last_undegraded":"2024-03-21T14:03:43.759191+0000","last_fullsized":"2024-03-21T14:03:43.759191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:03:43.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.759108+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.080054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394090+0000","last_change":"2024-03-21T14:02:39.471886+0000","last_active":"2024-03-21T14:03:42.394090+0000","last_peered":"2024-03-21T14:03:42.394090+0000","last_clean":"2024-03-21T14:03:42.394090+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:03:42.394090+0000","last_undegraded":"2024-03-21T14:03:42.394090+0000","last_fullsized":"2024-03-21T14:03:42.394090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.715774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757272+0000","last_change":"2024-03-21T14:02:37.455324+0000","last_active":"2024-03-21T14:03:43.757272+0000","last_peered":"2024-03-21T14:03:43.757272+0000","last_clean":"2024-03-21T14:03:43.757272+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:03:43.757272+0000","last_undegraded":"2024-03-21T14:03:43.757272+0000","last_fullsized":"2024-03-21T14:03:43.757272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.837893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.390039+0000","last_change":"2024-03-21T14:02:33.444875+0000","last_active":"2024-03-21T14:03:42.390039+0000","last_peered":"2024-03-21T14:03:42.390039+0000","last_clean":"2024-03-21T14:03:42.390039+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:03:42.390039+0000","last_undegraded":"2024-03-21T14:03:42.390039+0000","last_fullsized":"2024-03-21T14:03:42.390039+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389901+0000","last_change":"2024-03-21T14:02:58.514129+0000","last_active":"2024-03-21T14:03:42.389901+0000","last_peered":"2024-03-21T14:03:42.389901+0000","last_clean":"2024-03-21T14:03:42.389901+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:03:42.389901+0000","last_undegraded":"2024-03-21T14:03:42.389901+0000","last_fullsized":"2024-03-21T14:03:42.389901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.398980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996214+0000","last_change":"2024-03-21T14:02:39.473392+0000","last_active":"2024-03-21T14:03:42.996214+0000","last_peered":"2024-03-21T14:03:42.996214+0000","last_clean":"2024-03-21T14:03:42.996214+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:03:42.996214+0000","last_undegraded":"2024-03-21T14:03:42.996214+0000","last_fullsized":"2024-03-21T14:03:42.996214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.586523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999296+0000","last_change":"2024-03-21T14:03:37.352147+0000","last_active":"2024-03-21T14:03:42.999296+0000","last_peered":"2024-03-21T14:03:42.999296+0000","last_clean":"2024-03-21T14:03:42.999296+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:03:42.999296+0000","last_undegraded":"2024-03-21T14:03:42.999296+0000","last_fullsized":"2024-03-21T14:03:42.999296+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:37.352012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.352012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.843950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533005+0000","last_change":"2024-03-21T14:02:33.446863+0000","last_active":"2024-03-21T14:03:43.533005+0000","last_peered":"2024-03-21T14:03:43.533005+0000","last_clean":"2024-03-21T14:03:43.533005+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:03:43.533005+0000","last_undegraded":"2024-03-21T14:03:43.533005+0000","last_fullsized":"2024-03-21T14:03:43.533005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.983498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395982+0000","last_change":"2024-03-21T14:02:35.775119+0000","last_active":"2024-03-21T14:03:42.395982+0000","last_peered":"2024-03-21T14:03:42.395982+0000","last_clean":"2024-03-21T14:03:42.395982+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:03:42.395982+0000","last_undegraded":"2024-03-21T14:03:42.395982+0000","last_fullsized":"2024-03-21T14:03:42.395982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.079104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.390025+0000","last_change":"2024-03-21T14:02:39.467603+0000","last_active":"2024-03-21T14:03:42.390025+0000","last_peered":"2024-03-21T14:03:42.390025+0000","last_clean":"2024-03-21T14:03:42.390025+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:03:42.390025+0000","last_undegraded":"2024-03-21T14:03:42.390025+0000","last_fullsized":"2024-03-21T14:03:42.390025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.883243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396211+0000","last_change":"2024-03-21T14:02:37.461808+0000","last_active":"2024-03-21T14:03:42.396211+0000","last_peered":"2024-03-21T14:03:42.396211+0000","last_clean":"2024-03-21T14:03:42.396211+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:03:42.396211+0000","last_undegraded":"2024-03-21T14:03:42.396211+0000","last_fullsized":"2024-03-21T14:03:42.396211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995810+0000","last_change":"2024-03-21T14:02:33.439871+0000","last_active":"2024-03-21T14:03:42.995810+0000","last_peered":"2024-03-21T14:03:42.995810+0000","last_clean":"2024-03-21T14:03:42.995810+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:03:42.995810+0000","last_undegraded":"2024-03-21T14:03:42.995810+0000","last_fullsized":"2024-03-21T14:03:42.995810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.912588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:02:35.775497+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:03.620820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395973+0000","last_change":"2024-03-21T14:02:39.464551+0000","last_active":"2024-03-21T14:03:42.395973+0000","last_peered":"2024-03-21T14:03:42.395973+0000","last_clean":"2024-03-21T14:03:42.395973+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:03:42.395973+0000","last_undegraded":"2024-03-21T14:03:42.395973+0000","last_fullsized":"2024-03-21T14:03:42.395973+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.769258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396096+0000","last_change":"2024-03-21T14:02:37.456859+0000","last_active":"2024-03-21T14:03:42.396096+0000","last_peered":"2024-03-21T14:03:42.396096+0000","last_clean":"2024-03-21T14:03:42.396096+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:03:42.396096+0000","last_undegraded":"2024-03-21T14:03:42.396096+0000","last_fullsized":"2024-03-21T14:03:42.396096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.881127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532903+0000","last_change":"2024-03-21T14:03:38.848893+0000","last_active":"2024-03-21T14:03:43.532903+0000","last_peered":"2024-03-21T14:03:43.532903+0000","last_clean":"2024-03-21T14:03:43.532903+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:03:43.532903+0000","last_undegraded":"2024-03-21T14:03:43.532903+0000","last_fullsized":"2024-03-21T14:03:43.532903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.848793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.848793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.951970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996491+0000","last_change":"2024-03-21T14:02:35.774922+0000","last_active":"2024-03-21T14:03:42.996491+0000","last_peered":"2024-03-21T14:03:42.996491+0000","last_clean":"2024-03-21T14:03:42.996491+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:03:42.996491+0000","last_undegraded":"2024-03-21T14:03:42.996491+0000","last_fullsized":"2024-03-21T14:03:42.996491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.077976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532613+0000","last_change":"2024-03-21T14:03:41.906118+0000","last_active":"2024-03-21T14:03:43.532613+0000","last_peered":"2024-03-21T14:03:43.532613+0000","last_clean":"2024-03-21T14:03:43.532613+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:03:43.532613+0000","last_undegraded":"2024-03-21T14:03:43.532613+0000","last_fullsized":"2024-03-21T14:03:43.532613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.906016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.906016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:46.096482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999520+0000","last_change":"2024-03-21T14:02:37.454805+0000","last_active":"2024-03-21T14:03:42.999520+0000","last_peered":"2024-03-21T14:03:42.999520+0000","last_clean":"2024-03-21T14:03:42.999520+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:03:42.999520+0000","last_undegraded":"2024-03-21T14:03:42.999520+0000","last_fullsized":"2024-03-21T14:03:42.999520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.374515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734655+0000","last_change":"2024-03-21T14:03:38.199089+0000","last_active":"2024-03-21T14:03:42.734655+0000","last_peered":"2024-03-21T14:03:42.734655+0000","last_clean":"2024-03-21T14:03:42.734655+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:03:42.734655+0000","last_undegraded":"2024-03-21T14:03:42.734655+0000","last_fullsized":"2024-03-21T14:03:42.734655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.198986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.198986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.188712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395952+0000","last_change":"2024-03-21T14:02:35.774132+0000","last_active":"2024-03-21T14:03:42.395952+0000","last_peered":"2024-03-21T14:03:42.395952+0000","last_clean":"2024-03-21T14:03:42.395952+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:03:42.395952+0000","last_undegraded":"2024-03-21T14:03:42.395952+0000","last_fullsized":"2024-03-21T14:03:42.395952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.386896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999034+0000","last_change":"2024-03-21T14:02:39.517249+0000","last_active":"2024-03-21T14:03:42.999034+0000","last_peered":"2024-03-21T14:03:42.999034+0000","last_clean":"2024-03-21T14:03:42.999034+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:03:42.999034+0000","last_undegraded":"2024-03-21T14:03:42.999034+0000","last_fullsized":"2024-03-21T14:03:42.999034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.493578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533577+0000","last_change":"2024-03-21T14:02:33.448412+0000","last_active":"2024-03-21T14:03:43.533577+0000","last_peered":"2024-03-21T14:03:43.533577+0000","last_clean":"2024-03-21T14:03:43.533577+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:03:43.533577+0000","last_undegraded":"2024-03-21T14:03:43.533577+0000","last_fullsized":"2024-03-21T14:03:43.533577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.124819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998643+0000","last_change":"2024-03-21T14:02:35.775475+0000","last_active":"2024-03-21T14:03:42.998643+0000","last_peered":"2024-03-21T14:03:42.998643+0000","last_clean":"2024-03-21T14:03:42.998643+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:03:42.998643+0000","last_undegraded":"2024-03-21T14:03:42.998643+0000","last_fullsized":"2024-03-21T14:03:42.998643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.820694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533090+0000","last_change":"2024-03-21T14:02:37.454898+0000","last_active":"2024-03-21T14:03:43.533090+0000","last_peered":"2024-03-21T14:03:43.533090+0000","last_clean":"2024-03-21T14:03:43.533090+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:03:43.533090+0000","last_undegraded":"2024-03-21T14:03:43.533090+0000","last_fullsized":"2024-03-21T14:03:43.533090+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.360256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395170+0000","last_change":"2024-03-21T14:02:39.472019+0000","last_active":"2024-03-21T14:03:42.395170+0000","last_peered":"2024-03-21T14:03:42.395170+0000","last_clean":"2024-03-21T14:03:42.395170+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:03:42.395170+0000","last_undegraded":"2024-03-21T14:03:42.395170+0000","last_fullsized":"2024-03-21T14:03:42.395170+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.761931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.909142+0000","last_change":"2024-03-21T14:03:45.909165+0000","last_active":"2024-03-21T14:03:45.909142+0000","last_peered":"2024-03-21T14:03:45.909142+0000","last_clean":"2024-03-21T14:03:45.909142+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:03:45.909142+0000","last_undegraded":"2024-03-21T14:03:45.909142+0000","last_fullsized":"2024-03-21T14:03:45.909142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:45.909064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.909064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.828690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389532+0000","last_change":"2024-03-21T14:02:35.450101+0000","last_active":"2024-03-21T14:03:42.389532+0000","last_peered":"2024-03-21T14:03:42.389532+0000","last_clean":"2024-03-21T14:03:42.389532+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:03:42.389532+0000","last_undegraded":"2024-03-21T14:03:42.389532+0000","last_fullsized":"2024-03-21T14:03:42.389532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.268875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394123+0000","last_change":"2024-03-21T14:02:37.457375+0000","last_active":"2024-03-21T14:03:42.394123+0000","last_peered":"2024-03-21T14:03:42.394123+0000","last_clean":"2024-03-21T14:03:42.394123+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:03:42.394123+0000","last_undegraded":"2024-03-21T14:03:42.394123+0000","last_fullsized":"2024-03-21T14:03:42.394123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.887492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532480+0000","last_change":"2024-03-21T14:02:39.470278+0000","last_active":"2024-03-21T14:03:43.532480+0000","last_peered":"2024-03-21T14:03:43.532480+0000","last_clean":"2024-03-21T14:03:43.532480+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:03:43.532480+0000","last_undegraded":"2024-03-21T14:03:43.532480+0000","last_fullsized":"2024-03-21T14:03:43.532480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.685616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395196+0000","last_change":"2024-03-21T14:03:39.021466+0000","last_active":"2024-03-21T14:03:42.395196+0000","last_peered":"2024-03-21T14:03:42.395196+0000","last_clean":"2024-03-21T14:03:42.395196+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:03:42.395196+0000","last_undegraded":"2024-03-21T14:03:42.395196+0000","last_fullsized":"2024-03-21T14:03:42.395196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.021385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.021385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.898145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533423+0000","last_change":"2024-03-21T14:02:33.445582+0000","last_active":"2024-03-21T14:03:43.533423+0000","last_peered":"2024-03-21T14:03:43.533423+0000","last_clean":"2024-03-21T14:03:43.533423+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:03:43.533423+0000","last_undegraded":"2024-03-21T14:03:43.533423+0000","last_fullsized":"2024-03-21T14:03:43.533423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.597456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734954+0000","last_change":"2024-03-21T14:02:35.773869+0000","last_active":"2024-03-21T14:03:42.734954+0000","last_peered":"2024-03-21T14:03:42.734954+0000","last_clean":"2024-03-21T14:03:42.734954+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:03:42.734954+0000","last_undegraded":"2024-03-21T14:03:42.734954+0000","last_fullsized":"2024-03-21T14:03:42.734954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.067670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995644+0000","last_change":"2024-03-21T14:03:38.925041+0000","last_active":"2024-03-21T14:03:42.995644+0000","last_peered":"2024-03-21T14:03:42.995644+0000","last_clean":"2024-03-21T14:03:42.995644+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:03:42.995644+0000","last_undegraded":"2024-03-21T14:03:42.995644+0000","last_fullsized":"2024-03-21T14:03:42.995644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.924957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.924957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.656539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394809+0000","last_change":"2024-03-21T14:03:41.007728+0000","last_active":"2024-03-21T14:03:42.394809+0000","last_peered":"2024-03-21T14:03:42.394809+0000","last_clean":"2024-03-21T14:03:42.394809+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:03:42.394809+0000","last_undegraded":"2024-03-21T14:03:42.394809+0000","last_fullsized":"2024-03-21T14:03:42.394809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.007598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.075924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999121+0000","last_change":"2024-03-21T14:02:33.442629+0000","last_active":"2024-03-21T14:03:42.999121+0000","last_peered":"2024-03-21T14:03:42.999121+0000","last_clean":"2024-03-21T14:03:42.999121+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:03:42.999121+0000","last_undegraded":"2024-03-21T14:03:42.999121+0000","last_fullsized":"2024-03-21T14:03:42.999121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.962101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":79,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996413+0000","last_change":"2024-03-21T14:02:58.514359+0000","last_active":"2024-03-21T14:03:42.996413+0000","last_peered":"2024-03-21T14:03:42.996413+0000","last_clean":"2024-03-21T14:03:42.996413+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:03:42.996413+0000","last_undegraded":"2024-03-21T14:03:42.996413+0000","last_fullsized":"2024-03-21T14:03:42.996413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.552419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396180+0000","last_change":"2024-03-21T14:02:39.472519+0000","last_active":"2024-03-21T14:03:42.396180+0000","last_peered":"2024-03-21T14:03:42.396180+0000","last_clean":"2024-03-21T14:03:42.396180+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:03:42.396180+0000","last_undegraded":"2024-03-21T14:03:42.396180+0000","last_fullsized":"2024-03-21T14:03:42.396180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.779767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998816+0000","last_change":"2024-03-21T14:03:35.349742+0000","last_active":"2024-03-21T14:03:42.998816+0000","last_peered":"2024-03-21T14:03:42.998816+0000","last_clean":"2024-03-21T14:03:42.998816+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:03:42.998816+0000","last_undegraded":"2024-03-21T14:03:42.998816+0000","last_fullsized":"2024-03-21T14:03:42.998816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:03:35.349636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.349636+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.186069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389279+0000","last_change":"2024-03-21T14:02:33.441265+0000","last_active":"2024-03-21T14:03:42.389279+0000","last_peered":"2024-03-21T14:03:42.389279+0000","last_clean":"2024-03-21T14:03:42.389279+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:03:42.389279+0000","last_undegraded":"2024-03-21T14:03:42.389279+0000","last_fullsized":"2024-03-21T14:03:42.389279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.975558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":64,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998984+0000","last_change":"2024-03-21T14:02:58.530633+0000","last_active":"2024-03-21T14:03:42.998984+0000","last_peered":"2024-03-21T14:03:42.998984+0000","last_clean":"2024-03-21T14:03:42.998984+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:03:42.998984+0000","last_undegraded":"2024-03-21T14:03:42.998984+0000","last_fullsized":"2024-03-21T14:03:42.998984+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.120051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996233+0000","last_change":"2024-03-21T14:02:37.456968+0000","last_active":"2024-03-21T14:03:42.996233+0000","last_peered":"2024-03-21T14:03:42.996233+0000","last_clean":"2024-03-21T14:03:42.996233+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:03:42.996233+0000","last_undegraded":"2024-03-21T14:03:42.996233+0000","last_fullsized":"2024-03-21T14:03:42.996233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.729850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532410+0000","last_change":"2024-03-21T14:02:39.516083+0000","last_active":"2024-03-21T14:03:43.532410+0000","last_peered":"2024-03-21T14:03:43.532410+0000","last_clean":"2024-03-21T14:03:43.532410+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:03:43.532410+0000","last_undegraded":"2024-03-21T14:03:43.532410+0000","last_fullsized":"2024-03-21T14:03:43.532410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756711+0000","last_change":"2024-03-21T14:02:33.442413+0000","last_active":"2024-03-21T14:03:43.756711+0000","last_peered":"2024-03-21T14:03:43.756711+0000","last_clean":"2024-03-21T14:03:43.756711+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:03:43.756711+0000","last_undegraded":"2024-03-21T14:03:43.756711+0000","last_fullsized":"2024-03-21T14:03:43.756711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.621090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394635+0000","last_change":"2024-03-21T14:02:36.440450+0000","last_active":"2024-03-21T14:03:42.394635+0000","last_peered":"2024-03-21T14:03:42.394635+0000","last_clean":"2024-03-21T14:03:42.394635+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:03:42.394635+0000","last_undegraded":"2024-03-21T14:03:42.394635+0000","last_fullsized":"2024-03-21T14:03:42.394635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:02:31.423682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.085797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757698+0000","last_change":"2024-03-21T14:03:39.830694+0000","last_active":"2024-03-21T14:03:43.757698+0000","last_peered":"2024-03-21T14:03:43.757698+0000","last_clean":"2024-03-21T14:03:43.757698+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:03:43.757698+0000","last_undegraded":"2024-03-21T14:03:43.757698+0000","last_fullsized":"2024-03-21T14:03:43.757698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.830542+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.877216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395396+0000","last_change":"2024-03-21T14:02:37.459280+0000","last_active":"2024-03-21T14:03:42.395396+0000","last_peered":"2024-03-21T14:03:42.395396+0000","last_clean":"2024-03-21T14:03:42.395396+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:03:42.395396+0000","last_undegraded":"2024-03-21T14:03:42.395396+0000","last_fullsized":"2024-03-21T14:03:42.395396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.298610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999465+0000","last_change":"2024-03-21T14:02:39.470869+0000","last_active":"2024-03-21T14:03:42.999465+0000","last_peered":"2024-03-21T14:03:42.999465+0000","last_clean":"2024-03-21T14:03:42.999465+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:03:42.999465+0000","last_undegraded":"2024-03-21T14:03:42.999465+0000","last_fullsized":"2024-03-21T14:03:42.999465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.994358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":304,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532237+0000","last_change":"2024-03-21T14:02:45.449726+0000","last_active":"2024-03-21T14:03:43.532237+0000","last_peered":"2024-03-21T14:03:43.532237+0000","last_clean":"2024-03-21T14:03:43.532237+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:03:43.532237+0000","last_undegraded":"2024-03-21T14:03:43.532237+0000","last_fullsized":"2024-03-21T14:03:43.532237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-21T14:02:45.449583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:02:45.449583+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.492847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395005+0000","last_change":"2024-03-21T14:02:33.444001+0000","last_active":"2024-03-21T14:03:42.395005+0000","last_peered":"2024-03-21T14:03:42.395005+0000","last_clean":"2024-03-21T14:03:42.395005+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:03:42.395005+0000","last_undegraded":"2024-03-21T14:03:42.395005+0000","last_fullsized":"2024-03-21T14:03:42.395005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.524470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735446+0000","last_change":"2024-03-21T14:02:35.774062+0000","last_active":"2024-03-21T14:03:42.735446+0000","last_peered":"2024-03-21T14:03:42.735446+0000","last_clean":"2024-03-21T14:03:42.735446+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:03:42.735446+0000","last_undegraded":"2024-03-21T14:03:42.735446+0000","last_fullsized":"2024-03-21T14:03:42.735446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.985882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533239+0000","last_change":"2024-03-21T14:02:37.459722+0000","last_active":"2024-03-21T14:03:43.533239+0000","last_peered":"2024-03-21T14:03:43.533239+0000","last_clean":"2024-03-21T14:03:43.533239+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:03:43.533239+0000","last_undegraded":"2024-03-21T14:03:43.533239+0000","last_fullsized":"2024-03-21T14:03:43.533239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.571697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756191+0000","last_change":"2024-03-21T14:02:39.471788+0000","last_active":"2024-03-21T14:03:43.756191+0000","last_peered":"2024-03-21T14:03:43.756191+0000","last_clean":"2024-03-21T14:03:43.756191+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:03:43.756191+0000","last_undegraded":"2024-03-21T14:03:43.756191+0000","last_fullsized":"2024-03-21T14:03:43.756191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.372670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995815+0000","last_change":"2024-03-21T14:02:33.447469+0000","last_active":"2024-03-21T14:03:42.995815+0000","last_peered":"2024-03-21T14:03:42.995815+0000","last_clean":"2024-03-21T14:03:42.995815+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:03:42.995815+0000","last_undegraded":"2024-03-21T14:03:42.995815+0000","last_fullsized":"2024-03-21T14:03:42.995815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.063604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":129,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:44.551350+0000","last_change":"2024-03-21T14:02:36.444864+0000","last_active":"2024-03-21T14:03:44.551350+0000","last_peered":"2024-03-21T14:03:44.551350+0000","last_clean":"2024-03-21T14:03:44.551350+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:03:44.551350+0000","last_undegraded":"2024-03-21T14:03:44.551350+0000","last_fullsized":"2024-03-21T14:03:44.551350+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:02:31.423682+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.753790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":66,"num_read_kb":61,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996292+0000","last_change":"2024-03-21T14:02:35.449792+0000","last_active":"2024-03-21T14:03:42.996292+0000","last_peered":"2024-03-21T14:03:42.996292+0000","last_clean":"2024-03-21T14:03:42.996292+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:03:42.996292+0000","last_undegraded":"2024-03-21T14:03:42.996292+0000","last_fullsized":"2024-03-21T14:03:42.996292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.553541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999655+0000","last_change":"2024-03-21T14:02:37.455752+0000","last_active":"2024-03-21T14:03:42.999655+0000","last_peered":"2024-03-21T14:03:42.999655+0000","last_clean":"2024-03-21T14:03:42.999655+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:03:42.999655+0000","last_undegraded":"2024-03-21T14:03:42.999655+0000","last_fullsized":"2024-03-21T14:03:42.999655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.531727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756153+0000","last_change":"2024-03-21T14:02:39.516952+0000","last_active":"2024-03-21T14:03:43.756153+0000","last_peered":"2024-03-21T14:03:43.756153+0000","last_clean":"2024-03-21T14:03:43.756153+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:03:43.756153+0000","last_undegraded":"2024-03-21T14:03:43.756153+0000","last_fullsized":"2024-03-21T14:03:43.756153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.885390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757538+0000","last_change":"2024-03-21T14:02:33.444253+0000","last_active":"2024-03-21T14:03:43.757538+0000","last_peered":"2024-03-21T14:03:43.757538+0000","last_clean":"2024-03-21T14:03:43.757538+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:03:43.757538+0000","last_undegraded":"2024-03-21T14:03:43.757538+0000","last_fullsized":"2024-03-21T14:03:43.757538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.260994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":91,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996230+0000","last_change":"2024-03-21T14:03:41.967449+0000","last_active":"2024-03-21T14:03:42.996230+0000","last_peered":"2024-03-21T14:03:42.996230+0000","last_clean":"2024-03-21T14:03:42.996230+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:03:42.996230+0000","last_undegraded":"2024-03-21T14:03:42.996230+0000","last_fullsized":"2024-03-21T14:03:42.996230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:03:41.967343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.967343+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.280117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734983+0000","last_change":"2024-03-21T14:02:37.454582+0000","last_active":"2024-03-21T14:03:42.734983+0000","last_peered":"2024-03-21T14:03:42.734983+0000","last_clean":"2024-03-21T14:03:42.734983+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:03:42.734983+0000","last_undegraded":"2024-03-21T14:03:42.734983+0000","last_fullsized":"2024-03-21T14:03:42.734983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.341912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995555+0000","last_change":"2024-03-21T14:02:39.473493+0000","last_active":"2024-03-21T14:03:42.995555+0000","last_peered":"2024-03-21T14:03:42.995555+0000","last_clean":"2024-03-21T14:03:42.995555+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:03:42.995555+0000","last_undegraded":"2024-03-21T14:03:42.995555+0000","last_fullsized":"2024-03-21T14:03:42.995555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.101298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999221+0000","last_change":"2024-03-21T14:02:33.443262+0000","last_active":"2024-03-21T14:03:42.999221+0000","last_peered":"2024-03-21T14:03:42.999221+0000","last_clean":"2024-03-21T14:03:42.999221+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:03:42.999221+0000","last_undegraded":"2024-03-21T14:03:42.999221+0000","last_fullsized":"2024-03-21T14:03:42.999221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.529943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757322+0000","last_change":"2024-03-21T14:03:38.812824+0000","last_active":"2024-03-21T14:03:43.757322+0000","last_peered":"2024-03-21T14:03:43.757322+0000","last_clean":"2024-03-21T14:03:43.757322+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:03:43.757322+0000","last_undegraded":"2024-03-21T14:03:43.757322+0000","last_fullsized":"2024-03-21T14:03:43.757322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:38.812732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.812732+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.836722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996697+0000","last_change":"2024-03-21T14:02:37.459903+0000","last_active":"2024-03-21T14:03:42.996697+0000","last_peered":"2024-03-21T14:03:42.996697+0000","last_clean":"2024-03-21T14:03:42.996697+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:03:42.996697+0000","last_undegraded":"2024-03-21T14:03:42.996697+0000","last_fullsized":"2024-03-21T14:03:42.996697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.203232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532331+0000","last_change":"2024-03-21T14:02:39.517270+0000","last_active":"2024-03-21T14:03:43.532331+0000","last_peered":"2024-03-21T14:03:43.532331+0000","last_clean":"2024-03-21T14:03:43.532331+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:03:43.532331+0000","last_undegraded":"2024-03-21T14:03:43.532331+0000","last_fullsized":"2024-03-21T14:03:43.532331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.528174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.935772+0000","last_change":"2024-03-21T14:03:43.935798+0000","last_active":"2024-03-21T14:03:43.935772+0000","last_peered":"2024-03-21T14:03:43.935772+0000","last_clean":"2024-03-21T14:03:43.935772+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:03:43.935772+0000","last_undegraded":"2024-03-21T14:03:43.935772+0000","last_fullsized":"2024-03-21T14:03:43.935772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:43.935711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.935711+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.752006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389164+0000","last_change":"2024-03-21T14:02:35.444623+0000","last_active":"2024-03-21T14:03:42.389164+0000","last_peered":"2024-03-21T14:03:42.389164+0000","last_clean":"2024-03-21T14:03:42.389164+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:03:42.389164+0000","last_undegraded":"2024-03-21T14:03:42.389164+0000","last_fullsized":"2024-03-21T14:03:42.389164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.184966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395071+0000","last_change":"2024-03-21T14:02:37.457785+0000","last_active":"2024-03-21T14:03:42.395071+0000","last_peered":"2024-03-21T14:03:42.395071+0000","last_clean":"2024-03-21T14:03:42.395071+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:03:42.395071+0000","last_undegraded":"2024-03-21T14:03:42.395071+0000","last_fullsized":"2024-03-21T14:03:42.395071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.398614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389071+0000","last_change":"2024-03-21T14:02:39.469558+0000","last_active":"2024-03-21T14:03:42.389071+0000","last_peered":"2024-03-21T14:03:42.389071+0000","last_clean":"2024-03-21T14:03:42.389071+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:03:42.389071+0000","last_undegraded":"2024-03-21T14:03:42.389071+0000","last_fullsized":"2024-03-21T14:03:42.389071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.811931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395781+0000","last_change":"2024-03-21T14:03:36.591924+0000","last_active":"2024-03-21T14:03:42.395781+0000","last_peered":"2024-03-21T14:03:42.395781+0000","last_clean":"2024-03-21T14:03:42.395781+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:03:42.395781+0000","last_undegraded":"2024-03-21T14:03:42.395781+0000","last_fullsized":"2024-03-21T14:03:42.395781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:03:36.591844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.332405+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":79,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394241+0000","last_change":"2024-03-21T14:02:35.773836+0000","last_active":"2024-03-21T14:03:42.394241+0000","last_peered":"2024-03-21T14:03:42.394241+0000","last_clean":"2024-03-21T14:03:42.394241+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:03:42.394241+0000","last_undegraded":"2024-03-21T14:03:42.394241+0000","last_fullsized":"2024-03-21T14:03:42.394241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388989+0000","last_change":"2024-03-21T14:02:37.456640+0000","last_active":"2024-03-21T14:03:42.388989+0000","last_peered":"2024-03-21T14:03:42.388989+0000","last_clean":"2024-03-21T14:03:42.388989+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:03:42.388989+0000","last_undegraded":"2024-03-21T14:03:42.388989+0000","last_fullsized":"2024-03-21T14:03:42.388989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.154776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999089+0000","last_change":"2024-03-21T14:02:39.471265+0000","last_active":"2024-03-21T14:03:42.999089+0000","last_peered":"2024-03-21T14:03:42.999089+0000","last_clean":"2024-03-21T14:03:42.999089+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:03:42.999089+0000","last_undegraded":"2024-03-21T14:03:42.999089+0000","last_fullsized":"2024-03-21T14:03:42.999089+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.256674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.001482+0000","last_change":"2024-03-21T14:03:43.001482+0000","last_active":"2024-03-21T14:03:43.001482+0000","last_peered":"2024-03-21T14:03:43.001482+0000","last_clean":"2024-03-21T14:03:43.001482+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:03:43.001482+0000","last_undegraded":"2024-03-21T14:03:43.001482+0000","last_fullsized":"2024-03-21T14:03:43.001482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:43.001363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.001363+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.127322+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394227+0000","last_change":"2024-03-21T14:02:33.444152+0000","last_active":"2024-03-21T14:03:42.394227+0000","last_peered":"2024-03-21T14:03:42.394227+0000","last_clean":"2024-03-21T14:03:42.394227+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:03:42.394227+0000","last_undegraded":"2024-03-21T14:03:42.394227+0000","last_fullsized":"2024-03-21T14:03:42.394227+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.592513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757597+0000","last_change":"2024-03-21T14:02:35.453150+0000","last_active":"2024-03-21T14:03:43.757597+0000","last_peered":"2024-03-21T14:03:43.757597+0000","last_clean":"2024-03-21T14:03:43.757597+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:03:43.757597+0000","last_undegraded":"2024-03-21T14:03:43.757597+0000","last_fullsized":"2024-03-21T14:03:43.757597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.011104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395610+0000","last_change":"2024-03-21T14:02:39.516061+0000","last_active":"2024-03-21T14:03:42.395610+0000","last_peered":"2024-03-21T14:03:42.395610+0000","last_clean":"2024-03-21T14:03:42.395610+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:03:42.395610+0000","last_undegraded":"2024-03-21T14:03:42.395610+0000","last_fullsized":"2024-03-21T14:03:42.395610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.869674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999626+0000","last_change":"2024-03-21T14:02:37.454216+0000","last_active":"2024-03-21T14:03:42.999626+0000","last_peered":"2024-03-21T14:03:42.999626+0000","last_clean":"2024-03-21T14:03:42.999626+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:03:42.999626+0000","last_undegraded":"2024-03-21T14:03:42.999626+0000","last_fullsized":"2024-03-21T14:03:42.999626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.938463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389026+0000","last_change":"2024-03-21T14:03:34.139285+0000","last_active":"2024-03-21T14:03:42.389026+0000","last_peered":"2024-03-21T14:03:42.389026+0000","last_clean":"2024-03-21T14:03:42.389026+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:03:42.389026+0000","last_undegraded":"2024-03-21T14:03:42.389026+0000","last_fullsized":"2024-03-21T14:03:42.389026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:34.139127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:34.139127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.352287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389671+0000","last_change":"2024-03-21T14:02:35.450014+0000","last_active":"2024-03-21T14:03:42.389671+0000","last_peered":"2024-03-21T14:03:42.389671+0000","last_clean":"2024-03-21T14:03:42.389671+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:03:42.389671+0000","last_undegraded":"2024-03-21T14:03:42.389671+0000","last_fullsized":"2024-03-21T14:03:42.389671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.855239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396108+0000","last_change":"2024-03-21T14:02:39.468574+0000","last_active":"2024-03-21T14:03:42.396108+0000","last_peered":"2024-03-21T14:03:42.396108+0000","last_clean":"2024-03-21T14:03:42.396108+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:03:42.396108+0000","last_undegraded":"2024-03-21T14:03:42.396108+0000","last_fullsized":"2024-03-21T14:03:42.396108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.471894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757692+0000","last_change":"2024-03-21T14:02:37.456947+0000","last_active":"2024-03-21T14:03:43.757692+0000","last_peered":"2024-03-21T14:03:43.757692+0000","last_clean":"2024-03-21T14:03:43.757692+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:03:43.757692+0000","last_undegraded":"2024-03-21T14:03:43.757692+0000","last_fullsized":"2024-03-21T14:03:43.757692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.581511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735065+0000","last_change":"2024-03-21T14:02:33.442462+0000","last_active":"2024-03-21T14:03:42.735065+0000","last_peered":"2024-03-21T14:03:42.735065+0000","last_clean":"2024-03-21T14:03:42.735065+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:03:42.735065+0000","last_undegraded":"2024-03-21T14:03:42.735065+0000","last_fullsized":"2024-03-21T14:03:42.735065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.441774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389894+0000","last_change":"2024-03-21T14:02:35.453988+0000","last_active":"2024-03-21T14:03:42.389894+0000","last_peered":"2024-03-21T14:03:42.389894+0000","last_clean":"2024-03-21T14:03:42.389894+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:03:42.389894+0000","last_undegraded":"2024-03-21T14:03:42.389894+0000","last_fullsized":"2024-03-21T14:03:42.389894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:43.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389971+0000","last_change":"2024-03-21T14:02:39.459328+0000","last_active":"2024-03-21T14:03:42.389971+0000","last_peered":"2024-03-21T14:03:42.389971+0000","last_clean":"2024-03-21T14:03:42.389971+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:03:42.389971+0000","last_undegraded":"2024-03-21T14:03:42.389971+0000","last_fullsized":"2024-03-21T14:03:42.389971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.833817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395780+0000","last_change":"2024-03-21T14:02:37.460196+0000","last_active":"2024-03-21T14:03:42.395780+0000","last_peered":"2024-03-21T14:03:42.395780+0000","last_clean":"2024-03-21T14:03:42.395780+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:03:42.395780+0000","last_undegraded":"2024-03-21T14:03:42.395780+0000","last_fullsized":"2024-03-21T14:03:42.395780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395341+0000","last_change":"2024-03-21T14:02:33.442529+0000","last_active":"2024-03-21T14:03:42.395341+0000","last_peered":"2024-03-21T14:03:42.395341+0000","last_clean":"2024-03-21T14:03:42.395341+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:03:42.395341+0000","last_undegraded":"2024-03-21T14:03:42.395341+0000","last_fullsized":"2024-03-21T14:03:42.395341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.804413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389402+0000","last_change":"2024-03-21T14:02:35.450511+0000","last_active":"2024-03-21T14:03:42.389402+0000","last_peered":"2024-03-21T14:03:42.389402+0000","last_clean":"2024-03-21T14:03:42.389402+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:03:42.389402+0000","last_undegraded":"2024-03-21T14:03:42.389402+0000","last_fullsized":"2024-03-21T14:03:42.389402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.494861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":61,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103187,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32516,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712636,"statfs":{"total":95995035648,"available":95961739264,"internally_reserved":0,"allocated":1368064,"data_stored":927293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430232,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28332,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716820,"statfs":{"total":95995035648,"available":95966023680,"internally_reserved":0,"allocated":1343488,"data_stored":925330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31960,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713192,"statfs":{"total":95995035648,"available":95962308608,"internally_reserved":0,"allocated":798720,"data_stored":337516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":926124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":335310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":794624,"data_stored":336302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":786432,"data_stored":336016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691870,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31936,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713216,"statfs":{"total":95995035648,"available":95962333184,"internally_reserved":0,"allocated":774144,"data_stored":336557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:03:48.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:03:49.257 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:49.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:03:49.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:03:49.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 3.1a deep-scrub starts 2024-03-21T14:03:49.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 3.1a deep-scrub ok 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 4.4 scrub starts 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[29576]: 4.4 scrub ok 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 3.1a deep-scrub starts 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 3.1a deep-scrub ok 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 4.4 scrub starts 2024-03-21T14:03:49.632 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:49 smithi003 ceph-mon[40510]: 4.4 scrub ok 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 3.1a deep-scrub starts 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 3.1a deep-scrub ok 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 4.4 scrub starts 2024-03-21T14:03:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:49 smithi118 ceph-mon[33862]: 4.4 scrub ok 2024-03-21T14:03:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:50 smithi003 ceph-mon[29576]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:03:50.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:50 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:03:50.787 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:50 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:03:50.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:50 smithi003 ceph-mon[40510]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:03:50.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:50 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:03:50.787 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:50 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:03:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:50 smithi118 ceph-mon[33862]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:03:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:50 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:03:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:50 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:03:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:51 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:03:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:51 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:03:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:51 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:03:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:51 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:03:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:03:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:03:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:03:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:03:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:03:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:03:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:03:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:51 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:03:52.720 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:52 smithi003 ceph-mon[29576]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:03:52.721 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:52 smithi003 ceph-mon[40510]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:03:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:52 smithi118 ceph-mon[33862]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:03:53.235 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:03:53.237 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:03:53.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[29576]: 6.1c scrub starts 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[29576]: 6.1c scrub ok 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[40510]: 6.1c scrub starts 2024-03-21T14:03:53.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:53 smithi003 ceph-mon[40510]: 6.1c scrub ok 2024-03-21T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:53 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:53 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:53 smithi118 ceph-mon[33862]: 6.1c scrub starts 2024-03-21T14:03:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:53 smithi118 ceph-mon[33862]: 6.1c scrub ok 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:03:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:54 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:03:54.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:03:54.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:03:54.938 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 943 B/s rd, 0 op/s 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:03:54.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:54 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 5.18 scrub starts 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 5.18 scrub ok 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 6.d deep-scrub starts 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 6.d deep-scrub ok 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 3.19 deep-scrub starts 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 3.19 deep-scrub ok 2024-03-21T14:03:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:03:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:55 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 5.18 scrub starts 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 5.18 scrub ok 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 6.d deep-scrub starts 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 6.d deep-scrub ok 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:03:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 3.19 deep-scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 3.19 deep-scrub ok 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 5.18 scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 5.18 scrub ok 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 6.d deep-scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 6.d deep-scrub ok 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 3.19 deep-scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 3.19 deep-scrub ok 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:03:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:55 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:03:56.264 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-21T14:03:51.249387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1005,"num_read_kb":851,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":613,"ondisk_log_size":613,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253056,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708160,"statfs":{"total":767960285184,"available":767701155840,"internally_reserved":0,"allocated":7991296,"data_stored":4460448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002215"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389060+0000","last_change":"2024-03-21T14:02:39.463103+0000","last_active":"2024-03-21T14:03:42.389060+0000","last_peered":"2024-03-21T14:03:42.389060+0000","last_clean":"2024-03-21T14:03:42.389060+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:03:42.389060+0000","last_undegraded":"2024-03-21T14:03:42.389060+0000","last_fullsized":"2024-03-21T14:03:42.389060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.364725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756817+0000","last_change":"2024-03-21T14:02:37.454690+0000","last_active":"2024-03-21T14:03:43.756817+0000","last_peered":"2024-03-21T14:03:43.756817+0000","last_clean":"2024-03-21T14:03:43.756817+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:03:43.756817+0000","last_undegraded":"2024-03-21T14:03:43.756817+0000","last_fullsized":"2024-03-21T14:03:43.756817+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.175742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.957706+0000","last_change":"2024-03-21T14:03:46.957726+0000","last_active":"2024-03-21T14:03:46.957706+0000","last_peered":"2024-03-21T14:03:46.957706+0000","last_clean":"2024-03-21T14:03:46.957706+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:03:46.957706+0000","last_undegraded":"2024-03-21T14:03:46.957706+0000","last_fullsized":"2024-03-21T14:03:46.957706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.957648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.957648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.716204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388929+0000","last_change":"2024-03-21T14:02:35.450207+0000","last_active":"2024-03-21T14:03:42.388929+0000","last_peered":"2024-03-21T14:03:42.388929+0000","last_clean":"2024-03-21T14:03:42.388929+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:03:42.388929+0000","last_undegraded":"2024-03-21T14:03:42.388929+0000","last_fullsized":"2024-03-21T14:03:42.388929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.595411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395241+0000","last_change":"2024-03-21T14:02:39.517587+0000","last_active":"2024-03-21T14:03:42.395241+0000","last_peered":"2024-03-21T14:03:42.395241+0000","last_clean":"2024-03-21T14:03:42.395241+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:03:42.395241+0000","last_undegraded":"2024-03-21T14:03:42.395241+0000","last_fullsized":"2024-03-21T14:03:42.395241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.320905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388940+0000","last_change":"2024-03-21T14:02:37.458374+0000","last_active":"2024-03-21T14:03:42.388940+0000","last_peered":"2024-03-21T14:03:42.388940+0000","last_clean":"2024-03-21T14:03:42.388940+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:03:42.388940+0000","last_undegraded":"2024-03-21T14:03:42.388940+0000","last_fullsized":"2024-03-21T14:03:42.388940+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.406467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396378+0000","last_change":"2024-03-21T14:03:37.567249+0000","last_active":"2024-03-21T14:03:42.396378+0000","last_peered":"2024-03-21T14:03:42.396378+0000","last_clean":"2024-03-21T14:03:42.396378+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:03:42.396378+0000","last_undegraded":"2024-03-21T14:03:42.396378+0000","last_fullsized":"2024-03-21T14:03:42.396378+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:37.567167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.567167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.628456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394378+0000","last_change":"2024-03-21T14:03:35.571620+0000","last_active":"2024-03-21T14:03:42.394378+0000","last_peered":"2024-03-21T14:03:42.394378+0000","last_clean":"2024-03-21T14:03:42.394378+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:03:42.394378+0000","last_undegraded":"2024-03-21T14:03:42.394378+0000","last_fullsized":"2024-03-21T14:03:42.394378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.571531+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:41.701221+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995806+0000","last_change":"2024-03-21T14:02:39.469561+0000","last_active":"2024-03-21T14:03:42.995806+0000","last_peered":"2024-03-21T14:03:42.995806+0000","last_clean":"2024-03-21T14:03:42.995806+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:03:42.995806+0000","last_undegraded":"2024-03-21T14:03:42.995806+0000","last_fullsized":"2024-03-21T14:03:42.995806+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.240000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999035+0000","last_change":"2024-03-21T14:02:37.455588+0000","last_active":"2024-03-21T14:03:42.999035+0000","last_peered":"2024-03-21T14:03:42.999035+0000","last_clean":"2024-03-21T14:03:42.999035+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:03:42.999035+0000","last_undegraded":"2024-03-21T14:03:42.999035+0000","last_fullsized":"2024-03-21T14:03:42.999035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.535187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999090+0000","last_change":"2024-03-21T14:02:33.441047+0000","last_active":"2024-03-21T14:03:42.999090+0000","last_peered":"2024-03-21T14:03:42.999090+0000","last_clean":"2024-03-21T14:03:42.999090+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:03:42.999090+0000","last_undegraded":"2024-03-21T14:03:42.999090+0000","last_fullsized":"2024-03-21T14:03:42.999090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.089986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389241+0000","last_change":"2024-03-21T14:02:35.445770+0000","last_active":"2024-03-21T14:03:42.389241+0000","last_peered":"2024-03-21T14:03:42.389241+0000","last_clean":"2024-03-21T14:03:42.389241+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:03:42.389241+0000","last_undegraded":"2024-03-21T14:03:42.389241+0000","last_fullsized":"2024-03-21T14:03:42.389241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.203589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998655+0000","last_change":"2024-03-21T14:02:39.466860+0000","last_active":"2024-03-21T14:03:42.998655+0000","last_peered":"2024-03-21T14:03:42.998655+0000","last_clean":"2024-03-21T14:03:42.998655+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:03:42.998655+0000","last_undegraded":"2024-03-21T14:03:42.998655+0000","last_fullsized":"2024-03-21T14:03:42.998655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.470502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532890+0000","last_change":"2024-03-21T14:02:37.457212+0000","last_active":"2024-03-21T14:03:43.532890+0000","last_peered":"2024-03-21T14:03:43.532890+0000","last_clean":"2024-03-21T14:03:43.532890+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:03:43.532890+0000","last_undegraded":"2024-03-21T14:03:43.532890+0000","last_fullsized":"2024-03-21T14:03:43.532890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.357086+0000","last_change":"2024-03-21T14:03:46.357111+0000","last_active":"2024-03-21T14:03:46.357086+0000","last_peered":"2024-03-21T14:03:46.357086+0000","last_clean":"2024-03-21T14:03:46.357086+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:03:46.357086+0000","last_undegraded":"2024-03-21T14:03:46.357086+0000","last_fullsized":"2024-03-21T14:03:46.357086+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.357008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.357008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.267704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389583+0000","last_change":"2024-03-21T14:02:35.453933+0000","last_active":"2024-03-21T14:03:42.389583+0000","last_peered":"2024-03-21T14:03:42.389583+0000","last_clean":"2024-03-21T14:03:42.389583+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:03:42.389583+0000","last_undegraded":"2024-03-21T14:03:42.389583+0000","last_fullsized":"2024-03-21T14:03:42.389583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.127706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389793+0000","last_change":"2024-03-21T14:02:39.516305+0000","last_active":"2024-03-21T14:03:42.389793+0000","last_peered":"2024-03-21T14:03:42.389793+0000","last_clean":"2024-03-21T14:03:42.389793+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:03:42.389793+0000","last_undegraded":"2024-03-21T14:03:42.389793+0000","last_fullsized":"2024-03-21T14:03:42.389793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.252625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757380+0000","last_change":"2024-03-21T14:02:37.459191+0000","last_active":"2024-03-21T14:03:43.757380+0000","last_peered":"2024-03-21T14:03:43.757380+0000","last_clean":"2024-03-21T14:03:43.757380+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:03:43.757380+0000","last_undegraded":"2024-03-21T14:03:43.757380+0000","last_fullsized":"2024-03-21T14:03:43.757380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.913816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996252+0000","last_change":"2024-03-21T14:02:33.447014+0000","last_active":"2024-03-21T14:03:42.996252+0000","last_peered":"2024-03-21T14:03:42.996252+0000","last_clean":"2024-03-21T14:03:42.996252+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:03:42.996252+0000","last_undegraded":"2024-03-21T14:03:42.996252+0000","last_fullsized":"2024-03-21T14:03:42.996252+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.197179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395670+0000","last_change":"2024-03-21T14:02:35.454150+0000","last_active":"2024-03-21T14:03:42.395670+0000","last_peered":"2024-03-21T14:03:42.395670+0000","last_clean":"2024-03-21T14:03:42.395670+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:03:42.395670+0000","last_undegraded":"2024-03-21T14:03:42.395670+0000","last_fullsized":"2024-03-21T14:03:42.395670+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.304290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395547+0000","last_change":"2024-03-21T14:02:39.517386+0000","last_active":"2024-03-21T14:03:42.395547+0000","last_peered":"2024-03-21T14:03:42.395547+0000","last_clean":"2024-03-21T14:03:42.395547+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:03:42.395547+0000","last_undegraded":"2024-03-21T14:03:42.395547+0000","last_fullsized":"2024-03-21T14:03:42.395547+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.220059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389360+0000","last_change":"2024-03-21T14:02:37.458298+0000","last_active":"2024-03-21T14:03:42.389360+0000","last_peered":"2024-03-21T14:03:42.389360+0000","last_clean":"2024-03-21T14:03:42.389360+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:03:42.389360+0000","last_undegraded":"2024-03-21T14:03:42.389360+0000","last_fullsized":"2024-03-21T14:03:42.389360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.474405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.949455+0000","last_change":"2024-03-21T14:03:47.949482+0000","last_active":"2024-03-21T14:03:47.949455+0000","last_peered":"2024-03-21T14:03:47.949455+0000","last_clean":"2024-03-21T14:03:47.949455+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:03:47.949455+0000","last_undegraded":"2024-03-21T14:03:47.949455+0000","last_fullsized":"2024-03-21T14:03:47.949455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.949388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.443300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395701+0000","last_change":"2024-03-21T14:02:35.451688+0000","last_active":"2024-03-21T14:03:42.395701+0000","last_peered":"2024-03-21T14:03:42.395701+0000","last_clean":"2024-03-21T14:03:42.395701+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:03:42.395701+0000","last_undegraded":"2024-03-21T14:03:42.395701+0000","last_fullsized":"2024-03-21T14:03:42.395701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.784537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532313+0000","last_change":"2024-03-21T14:02:39.466190+0000","last_active":"2024-03-21T14:03:43.532313+0000","last_peered":"2024-03-21T14:03:43.532313+0000","last_clean":"2024-03-21T14:03:43.532313+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:03:43.532313+0000","last_undegraded":"2024-03-21T14:03:43.532313+0000","last_fullsized":"2024-03-21T14:03:43.532313+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.968518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.499680+0000","last_change":"2024-03-21T14:02:37.455355+0000","last_active":"2024-03-21T14:03:43.499680+0000","last_peered":"2024-03-21T14:03:43.499680+0000","last_clean":"2024-03-21T14:03:43.499680+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:03:43.499680+0000","last_undegraded":"2024-03-21T14:03:43.499680+0000","last_fullsized":"2024-03-21T14:03:43.499680+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"65'2","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756941+0000","last_change":"2024-03-21T14:02:33.441483+0000","last_active":"2024-03-21T14:03:43.756941+0000","last_peered":"2024-03-21T14:03:43.756941+0000","last_clean":"2024-03-21T14:03:43.756941+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:03:43.756941+0000","last_undegraded":"2024-03-21T14:03:43.756941+0000","last_fullsized":"2024-03-21T14:03:43.756941+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.043328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:44.904151+0000","last_change":"2024-03-21T14:03:44.904204+0000","last_active":"2024-03-21T14:03:44.904151+0000","last_peered":"2024-03-21T14:03:44.904151+0000","last_clean":"2024-03-21T14:03:44.904151+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:03:44.904151+0000","last_undegraded":"2024-03-21T14:03:44.904151+0000","last_fullsized":"2024-03-21T14:03:44.904151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:44.904071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:44.904071+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.807993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756221+0000","last_change":"2024-03-21T14:02:39.465886+0000","last_active":"2024-03-21T14:03:43.756221+0000","last_peered":"2024-03-21T14:03:43.756221+0000","last_clean":"2024-03-21T14:03:43.756221+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:03:43.756221+0000","last_undegraded":"2024-03-21T14:03:43.756221+0000","last_fullsized":"2024-03-21T14:03:43.756221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.933267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996574+0000","last_change":"2024-03-21T14:02:37.463347+0000","last_active":"2024-03-21T14:03:42.996574+0000","last_peered":"2024-03-21T14:03:42.996574+0000","last_clean":"2024-03-21T14:03:42.996574+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:03:42.996574+0000","last_undegraded":"2024-03-21T14:03:42.996574+0000","last_fullsized":"2024-03-21T14:03:42.996574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.394323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394109+0000","last_change":"2024-03-21T14:03:39.522391+0000","last_active":"2024-03-21T14:03:42.394109+0000","last_peered":"2024-03-21T14:03:42.394109+0000","last_clean":"2024-03-21T14:03:42.394109+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:03:42.394109+0000","last_undegraded":"2024-03-21T14:03:42.394109+0000","last_fullsized":"2024-03-21T14:03:42.394109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.522309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.085948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735454+0000","last_change":"2024-03-21T14:02:35.449454+0000","last_active":"2024-03-21T14:03:42.735454+0000","last_peered":"2024-03-21T14:03:42.735454+0000","last_clean":"2024-03-21T14:03:42.735454+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:03:42.735454+0000","last_undegraded":"2024-03-21T14:03:42.735454+0000","last_fullsized":"2024-03-21T14:03:42.735454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:43.505518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532253+0000","last_change":"2024-03-21T14:02:39.466841+0000","last_active":"2024-03-21T14:03:43.532253+0000","last_peered":"2024-03-21T14:03:43.532253+0000","last_clean":"2024-03-21T14:03:43.532253+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:03:43.532253+0000","last_undegraded":"2024-03-21T14:03:43.532253+0000","last_fullsized":"2024-03-21T14:03:43.532253+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.901031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734740+0000","last_change":"2024-03-21T14:02:37.456532+0000","last_active":"2024-03-21T14:03:42.734740+0000","last_peered":"2024-03-21T14:03:42.734740+0000","last_clean":"2024-03-21T14:03:42.734740+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:03:42.734740+0000","last_undegraded":"2024-03-21T14:03:42.734740+0000","last_fullsized":"2024-03-21T14:03:42.734740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.215037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.916303+0000","last_change":"2024-03-21T14:03:47.916324+0000","last_active":"2024-03-21T14:03:47.916303+0000","last_peered":"2024-03-21T14:03:47.916303+0000","last_clean":"2024-03-21T14:03:47.916303+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:03:47.916303+0000","last_undegraded":"2024-03-21T14:03:47.916303+0000","last_fullsized":"2024-03-21T14:03:47.916303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.916245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.916245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.354491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.509555+0000","last_change":"2024-03-21T14:03:45.509584+0000","last_active":"2024-03-21T14:03:45.509555+0000","last_peered":"2024-03-21T14:03:45.509555+0000","last_clean":"2024-03-21T14:03:45.509555+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:03:45.509555+0000","last_undegraded":"2024-03-21T14:03:45.509555+0000","last_fullsized":"2024-03-21T14:03:45.509555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:03:45.509492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.509492+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.086854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395138+0000","last_change":"2024-03-21T14:02:39.517579+0000","last_active":"2024-03-21T14:03:42.395138+0000","last_peered":"2024-03-21T14:03:42.395138+0000","last_clean":"2024-03-21T14:03:42.395138+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:03:42.395138+0000","last_undegraded":"2024-03-21T14:03:42.395138+0000","last_fullsized":"2024-03-21T14:03:42.395138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.492381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.836823+0000","last_change":"2024-03-21T14:03:47.836847+0000","last_active":"2024-03-21T14:03:47.836823+0000","last_peered":"2024-03-21T14:03:47.836823+0000","last_clean":"2024-03-21T14:03:47.836823+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:03:47.836823+0000","last_undegraded":"2024-03-21T14:03:47.836823+0000","last_fullsized":"2024-03-21T14:03:47.836823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.836751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.836751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:07.563311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:03:41.362147+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:41.362005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.362005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:10.804533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394356+0000","last_change":"2024-03-21T14:02:35.773947+0000","last_active":"2024-03-21T14:03:42.394356+0000","last_peered":"2024-03-21T14:03:42.394356+0000","last_clean":"2024-03-21T14:03:42.394356+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:03:42.394356+0000","last_undegraded":"2024-03-21T14:03:42.394356+0000","last_fullsized":"2024-03-21T14:03:42.394356+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995540+0000","last_change":"2024-03-21T14:02:39.471220+0000","last_active":"2024-03-21T14:03:42.995540+0000","last_peered":"2024-03-21T14:03:42.995540+0000","last_clean":"2024-03-21T14:03:42.995540+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:03:42.995540+0000","last_undegraded":"2024-03-21T14:03:42.995540+0000","last_fullsized":"2024-03-21T14:03:42.995540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.470185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395282+0000","last_change":"2024-03-21T14:02:37.458457+0000","last_active":"2024-03-21T14:03:42.395282+0000","last_peered":"2024-03-21T14:03:42.395282+0000","last_clean":"2024-03-21T14:03:42.395282+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:03:42.395282+0000","last_undegraded":"2024-03-21T14:03:42.395282+0000","last_fullsized":"2024-03-21T14:03:42.395282+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533251+0000","last_change":"2024-03-21T14:02:33.449232+0000","last_active":"2024-03-21T14:03:43.533251+0000","last_peered":"2024-03-21T14:03:43.533251+0000","last_clean":"2024-03-21T14:03:43.533251+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:03:43.533251+0000","last_undegraded":"2024-03-21T14:03:43.533251+0000","last_fullsized":"2024-03-21T14:03:43.533251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.141490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.759191+0000","last_change":"2024-03-21T14:03:43.759222+0000","last_active":"2024-03-21T14:03:43.759191+0000","last_peered":"2024-03-21T14:03:43.759191+0000","last_clean":"2024-03-21T14:03:43.759191+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:03:43.759191+0000","last_undegraded":"2024-03-21T14:03:43.759191+0000","last_fullsized":"2024-03-21T14:03:43.759191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:03:43.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.759108+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.080054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394090+0000","last_change":"2024-03-21T14:02:39.471886+0000","last_active":"2024-03-21T14:03:42.394090+0000","last_peered":"2024-03-21T14:03:42.394090+0000","last_clean":"2024-03-21T14:03:42.394090+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:03:42.394090+0000","last_undegraded":"2024-03-21T14:03:42.394090+0000","last_fullsized":"2024-03-21T14:03:42.394090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.715774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757272+0000","last_change":"2024-03-21T14:02:37.455324+0000","last_active":"2024-03-21T14:03:43.757272+0000","last_peered":"2024-03-21T14:03:43.757272+0000","last_clean":"2024-03-21T14:03:43.757272+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:03:43.757272+0000","last_undegraded":"2024-03-21T14:03:43.757272+0000","last_fullsized":"2024-03-21T14:03:43.757272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.837893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.994403+0000","last_change":"2024-03-21T14:03:46.994427+0000","last_active":"2024-03-21T14:03:46.994403+0000","last_peered":"2024-03-21T14:03:46.994403+0000","last_clean":"2024-03-21T14:03:46.994403+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:03:46.994403+0000","last_undegraded":"2024-03-21T14:03:46.994403+0000","last_fullsized":"2024-03-21T14:03:46.994403+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.994338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.994338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.043818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389901+0000","last_change":"2024-03-21T14:02:58.514129+0000","last_active":"2024-03-21T14:03:42.389901+0000","last_peered":"2024-03-21T14:03:42.389901+0000","last_clean":"2024-03-21T14:03:42.389901+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:03:42.389901+0000","last_undegraded":"2024-03-21T14:03:42.389901+0000","last_fullsized":"2024-03-21T14:03:42.389901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.398980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996214+0000","last_change":"2024-03-21T14:02:39.473392+0000","last_active":"2024-03-21T14:03:42.996214+0000","last_peered":"2024-03-21T14:03:42.996214+0000","last_clean":"2024-03-21T14:03:42.996214+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:03:42.996214+0000","last_undegraded":"2024-03-21T14:03:42.996214+0000","last_fullsized":"2024-03-21T14:03:42.996214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.586523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.499622+0000","last_change":"2024-03-21T14:03:37.352147+0000","last_active":"2024-03-21T14:03:43.499622+0000","last_peered":"2024-03-21T14:03:43.499622+0000","last_clean":"2024-03-21T14:03:43.499622+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:03:43.499622+0000","last_undegraded":"2024-03-21T14:03:43.499622+0000","last_fullsized":"2024-03-21T14:03:43.499622+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:37.352012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.352012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.843950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533005+0000","last_change":"2024-03-21T14:02:33.446863+0000","last_active":"2024-03-21T14:03:43.533005+0000","last_peered":"2024-03-21T14:03:43.533005+0000","last_clean":"2024-03-21T14:03:43.533005+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:03:43.533005+0000","last_undegraded":"2024-03-21T14:03:43.533005+0000","last_fullsized":"2024-03-21T14:03:43.533005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.983498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395982+0000","last_change":"2024-03-21T14:02:35.775119+0000","last_active":"2024-03-21T14:03:42.395982+0000","last_peered":"2024-03-21T14:03:42.395982+0000","last_clean":"2024-03-21T14:03:42.395982+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:03:42.395982+0000","last_undegraded":"2024-03-21T14:03:42.395982+0000","last_fullsized":"2024-03-21T14:03:42.395982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.079104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.390025+0000","last_change":"2024-03-21T14:02:39.467603+0000","last_active":"2024-03-21T14:03:42.390025+0000","last_peered":"2024-03-21T14:03:42.390025+0000","last_clean":"2024-03-21T14:03:42.390025+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:03:42.390025+0000","last_undegraded":"2024-03-21T14:03:42.390025+0000","last_fullsized":"2024-03-21T14:03:42.390025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.883243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.500013+0000","last_change":"2024-03-21T14:03:47.500013+0000","last_active":"2024-03-21T14:03:47.500013+0000","last_peered":"2024-03-21T14:03:47.500013+0000","last_clean":"2024-03-21T14:03:47.500013+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:03:47.500013+0000","last_undegraded":"2024-03-21T14:03:47.500013+0000","last_fullsized":"2024-03-21T14:03:47.500013+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:47.499917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.499917+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.440649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995810+0000","last_change":"2024-03-21T14:02:33.439871+0000","last_active":"2024-03-21T14:03:42.995810+0000","last_peered":"2024-03-21T14:03:42.995810+0000","last_clean":"2024-03-21T14:03:42.995810+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:03:42.995810+0000","last_undegraded":"2024-03-21T14:03:42.995810+0000","last_fullsized":"2024-03-21T14:03:42.995810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.912588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:02:35.775497+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:03.620820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395973+0000","last_change":"2024-03-21T14:02:39.464551+0000","last_active":"2024-03-21T14:03:42.395973+0000","last_peered":"2024-03-21T14:03:42.395973+0000","last_clean":"2024-03-21T14:03:42.395973+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:03:42.395973+0000","last_undegraded":"2024-03-21T14:03:42.395973+0000","last_fullsized":"2024-03-21T14:03:42.395973+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.769258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396096+0000","last_change":"2024-03-21T14:02:37.456859+0000","last_active":"2024-03-21T14:03:42.396096+0000","last_peered":"2024-03-21T14:03:42.396096+0000","last_clean":"2024-03-21T14:03:42.396096+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:03:42.396096+0000","last_undegraded":"2024-03-21T14:03:42.396096+0000","last_fullsized":"2024-03-21T14:03:42.396096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.881127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532903+0000","last_change":"2024-03-21T14:03:38.848893+0000","last_active":"2024-03-21T14:03:43.532903+0000","last_peered":"2024-03-21T14:03:43.532903+0000","last_clean":"2024-03-21T14:03:43.532903+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:03:43.532903+0000","last_undegraded":"2024-03-21T14:03:43.532903+0000","last_fullsized":"2024-03-21T14:03:43.532903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.848793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.848793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.951970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996491+0000","last_change":"2024-03-21T14:02:35.774922+0000","last_active":"2024-03-21T14:03:42.996491+0000","last_peered":"2024-03-21T14:03:42.996491+0000","last_clean":"2024-03-21T14:03:42.996491+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:03:42.996491+0000","last_undegraded":"2024-03-21T14:03:42.996491+0000","last_fullsized":"2024-03-21T14:03:42.996491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.077976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532613+0000","last_change":"2024-03-21T14:03:41.906118+0000","last_active":"2024-03-21T14:03:43.532613+0000","last_peered":"2024-03-21T14:03:43.532613+0000","last_clean":"2024-03-21T14:03:43.532613+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:03:43.532613+0000","last_undegraded":"2024-03-21T14:03:43.532613+0000","last_fullsized":"2024-03-21T14:03:43.532613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.906016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.906016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:46.096482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999520+0000","last_change":"2024-03-21T14:02:37.454805+0000","last_active":"2024-03-21T14:03:42.999520+0000","last_peered":"2024-03-21T14:03:42.999520+0000","last_clean":"2024-03-21T14:03:42.999520+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:03:42.999520+0000","last_undegraded":"2024-03-21T14:03:42.999520+0000","last_fullsized":"2024-03-21T14:03:42.999520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.374515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734655+0000","last_change":"2024-03-21T14:03:38.199089+0000","last_active":"2024-03-21T14:03:42.734655+0000","last_peered":"2024-03-21T14:03:42.734655+0000","last_clean":"2024-03-21T14:03:42.734655+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:03:42.734655+0000","last_undegraded":"2024-03-21T14:03:42.734655+0000","last_fullsized":"2024-03-21T14:03:42.734655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.198986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.198986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.188712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395952+0000","last_change":"2024-03-21T14:02:35.774132+0000","last_active":"2024-03-21T14:03:42.395952+0000","last_peered":"2024-03-21T14:03:42.395952+0000","last_clean":"2024-03-21T14:03:42.395952+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:03:42.395952+0000","last_undegraded":"2024-03-21T14:03:42.395952+0000","last_fullsized":"2024-03-21T14:03:42.395952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.386896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999034+0000","last_change":"2024-03-21T14:02:39.517249+0000","last_active":"2024-03-21T14:03:42.999034+0000","last_peered":"2024-03-21T14:03:42.999034+0000","last_clean":"2024-03-21T14:03:42.999034+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:03:42.999034+0000","last_undegraded":"2024-03-21T14:03:42.999034+0000","last_fullsized":"2024-03-21T14:03:42.999034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.493578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533577+0000","last_change":"2024-03-21T14:02:33.448412+0000","last_active":"2024-03-21T14:03:43.533577+0000","last_peered":"2024-03-21T14:03:43.533577+0000","last_clean":"2024-03-21T14:03:43.533577+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:03:43.533577+0000","last_undegraded":"2024-03-21T14:03:43.533577+0000","last_fullsized":"2024-03-21T14:03:43.533577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.124819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998643+0000","last_change":"2024-03-21T14:02:35.775475+0000","last_active":"2024-03-21T14:03:42.998643+0000","last_peered":"2024-03-21T14:03:42.998643+0000","last_clean":"2024-03-21T14:03:42.998643+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:03:42.998643+0000","last_undegraded":"2024-03-21T14:03:42.998643+0000","last_fullsized":"2024-03-21T14:03:42.998643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.820694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533184+0000","last_change":"2024-03-21T14:02:37.454898+0000","last_active":"2024-03-21T14:03:43.533184+0000","last_peered":"2024-03-21T14:03:43.533184+0000","last_clean":"2024-03-21T14:03:43.533184+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:03:43.533184+0000","last_undegraded":"2024-03-21T14:03:43.533184+0000","last_fullsized":"2024-03-21T14:03:43.533184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.360256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395170+0000","last_change":"2024-03-21T14:02:39.472019+0000","last_active":"2024-03-21T14:03:42.395170+0000","last_peered":"2024-03-21T14:03:42.395170+0000","last_clean":"2024-03-21T14:03:42.395170+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:03:42.395170+0000","last_undegraded":"2024-03-21T14:03:42.395170+0000","last_fullsized":"2024-03-21T14:03:42.395170+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.761931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.909142+0000","last_change":"2024-03-21T14:03:45.909165+0000","last_active":"2024-03-21T14:03:45.909142+0000","last_peered":"2024-03-21T14:03:45.909142+0000","last_clean":"2024-03-21T14:03:45.909142+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:03:45.909142+0000","last_undegraded":"2024-03-21T14:03:45.909142+0000","last_fullsized":"2024-03-21T14:03:45.909142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:45.909064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.909064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.828690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389532+0000","last_change":"2024-03-21T14:02:35.450101+0000","last_active":"2024-03-21T14:03:42.389532+0000","last_peered":"2024-03-21T14:03:42.389532+0000","last_clean":"2024-03-21T14:03:42.389532+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:03:42.389532+0000","last_undegraded":"2024-03-21T14:03:42.389532+0000","last_fullsized":"2024-03-21T14:03:42.389532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.268875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394123+0000","last_change":"2024-03-21T14:02:37.457375+0000","last_active":"2024-03-21T14:03:42.394123+0000","last_peered":"2024-03-21T14:03:42.394123+0000","last_clean":"2024-03-21T14:03:42.394123+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:03:42.394123+0000","last_undegraded":"2024-03-21T14:03:42.394123+0000","last_fullsized":"2024-03-21T14:03:42.394123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.887492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532480+0000","last_change":"2024-03-21T14:02:39.470278+0000","last_active":"2024-03-21T14:03:43.532480+0000","last_peered":"2024-03-21T14:03:43.532480+0000","last_clean":"2024-03-21T14:03:43.532480+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:03:43.532480+0000","last_undegraded":"2024-03-21T14:03:43.532480+0000","last_fullsized":"2024-03-21T14:03:43.532480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.685616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395196+0000","last_change":"2024-03-21T14:03:39.021466+0000","last_active":"2024-03-21T14:03:42.395196+0000","last_peered":"2024-03-21T14:03:42.395196+0000","last_clean":"2024-03-21T14:03:42.395196+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:03:42.395196+0000","last_undegraded":"2024-03-21T14:03:42.395196+0000","last_fullsized":"2024-03-21T14:03:42.395196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.021385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.021385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.898145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533423+0000","last_change":"2024-03-21T14:02:33.445582+0000","last_active":"2024-03-21T14:03:43.533423+0000","last_peered":"2024-03-21T14:03:43.533423+0000","last_clean":"2024-03-21T14:03:43.533423+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:03:43.533423+0000","last_undegraded":"2024-03-21T14:03:43.533423+0000","last_fullsized":"2024-03-21T14:03:43.533423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.597456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734954+0000","last_change":"2024-03-21T14:02:35.773869+0000","last_active":"2024-03-21T14:03:42.734954+0000","last_peered":"2024-03-21T14:03:42.734954+0000","last_clean":"2024-03-21T14:03:42.734954+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:03:42.734954+0000","last_undegraded":"2024-03-21T14:03:42.734954+0000","last_fullsized":"2024-03-21T14:03:42.734954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.067670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995644+0000","last_change":"2024-03-21T14:03:38.925041+0000","last_active":"2024-03-21T14:03:42.995644+0000","last_peered":"2024-03-21T14:03:42.995644+0000","last_clean":"2024-03-21T14:03:42.995644+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:03:42.995644+0000","last_undegraded":"2024-03-21T14:03:42.995644+0000","last_fullsized":"2024-03-21T14:03:42.995644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.924957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.924957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.656539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394809+0000","last_change":"2024-03-21T14:03:41.007728+0000","last_active":"2024-03-21T14:03:42.394809+0000","last_peered":"2024-03-21T14:03:42.394809+0000","last_clean":"2024-03-21T14:03:42.394809+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:03:42.394809+0000","last_undegraded":"2024-03-21T14:03:42.394809+0000","last_fullsized":"2024-03-21T14:03:42.394809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.007598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.075924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999121+0000","last_change":"2024-03-21T14:02:33.442629+0000","last_active":"2024-03-21T14:03:42.999121+0000","last_peered":"2024-03-21T14:03:42.999121+0000","last_clean":"2024-03-21T14:03:42.999121+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:03:42.999121+0000","last_undegraded":"2024-03-21T14:03:42.999121+0000","last_fullsized":"2024-03-21T14:03:42.999121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.962101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":79,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996413+0000","last_change":"2024-03-21T14:02:58.514359+0000","last_active":"2024-03-21T14:03:42.996413+0000","last_peered":"2024-03-21T14:03:42.996413+0000","last_clean":"2024-03-21T14:03:42.996413+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:03:42.996413+0000","last_undegraded":"2024-03-21T14:03:42.996413+0000","last_fullsized":"2024-03-21T14:03:42.996413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.552419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396180+0000","last_change":"2024-03-21T14:02:39.472519+0000","last_active":"2024-03-21T14:03:42.396180+0000","last_peered":"2024-03-21T14:03:42.396180+0000","last_clean":"2024-03-21T14:03:42.396180+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:03:42.396180+0000","last_undegraded":"2024-03-21T14:03:42.396180+0000","last_fullsized":"2024-03-21T14:03:42.396180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.779767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998816+0000","last_change":"2024-03-21T14:03:35.349742+0000","last_active":"2024-03-21T14:03:42.998816+0000","last_peered":"2024-03-21T14:03:42.998816+0000","last_clean":"2024-03-21T14:03:42.998816+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:03:42.998816+0000","last_undegraded":"2024-03-21T14:03:42.998816+0000","last_fullsized":"2024-03-21T14:03:42.998816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:03:35.349636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.349636+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.186069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389279+0000","last_change":"2024-03-21T14:02:33.441265+0000","last_active":"2024-03-21T14:03:42.389279+0000","last_peered":"2024-03-21T14:03:42.389279+0000","last_clean":"2024-03-21T14:03:42.389279+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:03:42.389279+0000","last_undegraded":"2024-03-21T14:03:42.389279+0000","last_fullsized":"2024-03-21T14:03:42.389279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.975558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":73,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:48.369792+0000","last_change":"2024-03-21T14:03:48.369792+0000","last_active":"2024-03-21T14:03:48.369792+0000","last_peered":"2024-03-21T14:03:48.369792+0000","last_clean":"2024-03-21T14:03:48.369792+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:03:48.369792+0000","last_undegraded":"2024-03-21T14:03:48.369792+0000","last_fullsized":"2024-03-21T14:03:48.369792+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:03:48.369672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:48.369672+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.178152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996233+0000","last_change":"2024-03-21T14:02:37.456968+0000","last_active":"2024-03-21T14:03:42.996233+0000","last_peered":"2024-03-21T14:03:42.996233+0000","last_clean":"2024-03-21T14:03:42.996233+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:03:42.996233+0000","last_undegraded":"2024-03-21T14:03:42.996233+0000","last_fullsized":"2024-03-21T14:03:42.996233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.729850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.872553+0000","last_change":"2024-03-21T14:03:46.872577+0000","last_active":"2024-03-21T14:03:46.872553+0000","last_peered":"2024-03-21T14:03:46.872553+0000","last_clean":"2024-03-21T14:03:46.872553+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:03:46.872553+0000","last_undegraded":"2024-03-21T14:03:46.872553+0000","last_fullsized":"2024-03-21T14:03:46.872553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.872477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.872477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.050084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756711+0000","last_change":"2024-03-21T14:02:33.442413+0000","last_active":"2024-03-21T14:03:43.756711+0000","last_peered":"2024-03-21T14:03:43.756711+0000","last_clean":"2024-03-21T14:03:43.756711+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:03:43.756711+0000","last_undegraded":"2024-03-21T14:03:43.756711+0000","last_fullsized":"2024-03-21T14:03:43.756711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.621090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394635+0000","last_change":"2024-03-21T14:02:36.440450+0000","last_active":"2024-03-21T14:03:42.394635+0000","last_peered":"2024-03-21T14:03:42.394635+0000","last_clean":"2024-03-21T14:03:42.394635+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:03:42.394635+0000","last_undegraded":"2024-03-21T14:03:42.394635+0000","last_fullsized":"2024-03-21T14:03:42.394635+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:02:31.423682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.085797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757698+0000","last_change":"2024-03-21T14:03:39.830694+0000","last_active":"2024-03-21T14:03:43.757698+0000","last_peered":"2024-03-21T14:03:43.757698+0000","last_clean":"2024-03-21T14:03:43.757698+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:03:43.757698+0000","last_undegraded":"2024-03-21T14:03:43.757698+0000","last_fullsized":"2024-03-21T14:03:43.757698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.830542+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.877216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395396+0000","last_change":"2024-03-21T14:02:37.459280+0000","last_active":"2024-03-21T14:03:42.395396+0000","last_peered":"2024-03-21T14:03:42.395396+0000","last_clean":"2024-03-21T14:03:42.395396+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:03:42.395396+0000","last_undegraded":"2024-03-21T14:03:42.395396+0000","last_fullsized":"2024-03-21T14:03:42.395396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.298610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999465+0000","last_change":"2024-03-21T14:02:39.470869+0000","last_active":"2024-03-21T14:03:42.999465+0000","last_peered":"2024-03-21T14:03:42.999465+0000","last_clean":"2024-03-21T14:03:42.999465+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:03:42.999465+0000","last_undegraded":"2024-03-21T14:03:42.999465+0000","last_fullsized":"2024-03-21T14:03:42.999465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.994358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":304,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532237+0000","last_change":"2024-03-21T14:02:45.449726+0000","last_active":"2024-03-21T14:03:43.532237+0000","last_peered":"2024-03-21T14:03:43.532237+0000","last_clean":"2024-03-21T14:03:43.532237+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:03:43.532237+0000","last_undegraded":"2024-03-21T14:03:43.532237+0000","last_fullsized":"2024-03-21T14:03:43.532237+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"21'117","last_scrub_stamp":"2024-03-21T14:02:45.449583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:02:45.449583+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.492847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395005+0000","last_change":"2024-03-21T14:02:33.444001+0000","last_active":"2024-03-21T14:03:42.395005+0000","last_peered":"2024-03-21T14:03:42.395005+0000","last_clean":"2024-03-21T14:03:42.395005+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:03:42.395005+0000","last_undegraded":"2024-03-21T14:03:42.395005+0000","last_fullsized":"2024-03-21T14:03:42.395005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.524470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":85,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.499690+0000","last_change":"2024-03-21T14:02:35.774062+0000","last_active":"2024-03-21T14:03:43.499690+0000","last_peered":"2024-03-21T14:03:43.499690+0000","last_clean":"2024-03-21T14:03:43.499690+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:03:43.499690+0000","last_undegraded":"2024-03-21T14:03:43.499690+0000","last_fullsized":"2024-03-21T14:03:43.499690+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.985882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533239+0000","last_change":"2024-03-21T14:02:37.459722+0000","last_active":"2024-03-21T14:03:43.533239+0000","last_peered":"2024-03-21T14:03:43.533239+0000","last_clean":"2024-03-21T14:03:43.533239+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:03:43.533239+0000","last_undegraded":"2024-03-21T14:03:43.533239+0000","last_fullsized":"2024-03-21T14:03:43.533239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.571697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756191+0000","last_change":"2024-03-21T14:02:39.471788+0000","last_active":"2024-03-21T14:03:43.756191+0000","last_peered":"2024-03-21T14:03:43.756191+0000","last_clean":"2024-03-21T14:03:43.756191+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:03:43.756191+0000","last_undegraded":"2024-03-21T14:03:43.756191+0000","last_fullsized":"2024-03-21T14:03:43.756191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.372670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995815+0000","last_change":"2024-03-21T14:02:33.447469+0000","last_active":"2024-03-21T14:03:42.995815+0000","last_peered":"2024-03-21T14:03:42.995815+0000","last_clean":"2024-03-21T14:03:42.995815+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:03:42.995815+0000","last_undegraded":"2024-03-21T14:03:42.995815+0000","last_fullsized":"2024-03-21T14:03:42.995815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.063604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":142,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:49.845488+0000","last_change":"2024-03-21T14:03:49.845488+0000","last_active":"2024-03-21T14:03:49.845488+0000","last_peered":"2024-03-21T14:03:49.845488+0000","last_clean":"2024-03-21T14:03:49.845488+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:03:49.845488+0000","last_undegraded":"2024-03-21T14:03:49.845488+0000","last_fullsized":"2024-03-21T14:03:49.845488+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:03:49.845376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:49.845376+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:12.461975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996292+0000","last_change":"2024-03-21T14:02:35.449792+0000","last_active":"2024-03-21T14:03:42.996292+0000","last_peered":"2024-03-21T14:03:42.996292+0000","last_clean":"2024-03-21T14:03:42.996292+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:03:42.996292+0000","last_undegraded":"2024-03-21T14:03:42.996292+0000","last_fullsized":"2024-03-21T14:03:42.996292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.553541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999655+0000","last_change":"2024-03-21T14:02:37.455752+0000","last_active":"2024-03-21T14:03:42.999655+0000","last_peered":"2024-03-21T14:03:42.999655+0000","last_clean":"2024-03-21T14:03:42.999655+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:03:42.999655+0000","last_undegraded":"2024-03-21T14:03:42.999655+0000","last_fullsized":"2024-03-21T14:03:42.999655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.531727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756153+0000","last_change":"2024-03-21T14:02:39.516952+0000","last_active":"2024-03-21T14:03:43.756153+0000","last_peered":"2024-03-21T14:03:43.756153+0000","last_clean":"2024-03-21T14:03:43.756153+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:03:43.756153+0000","last_undegraded":"2024-03-21T14:03:43.756153+0000","last_fullsized":"2024-03-21T14:03:43.756153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.885390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757538+0000","last_change":"2024-03-21T14:02:33.444253+0000","last_active":"2024-03-21T14:03:43.757538+0000","last_peered":"2024-03-21T14:03:43.757538+0000","last_clean":"2024-03-21T14:03:43.757538+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:03:43.757538+0000","last_undegraded":"2024-03-21T14:03:43.757538+0000","last_fullsized":"2024-03-21T14:03:43.757538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.260994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":91,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996230+0000","last_change":"2024-03-21T14:03:41.967449+0000","last_active":"2024-03-21T14:03:42.996230+0000","last_peered":"2024-03-21T14:03:42.996230+0000","last_clean":"2024-03-21T14:03:42.996230+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:03:42.996230+0000","last_undegraded":"2024-03-21T14:03:42.996230+0000","last_fullsized":"2024-03-21T14:03:42.996230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:03:41.967343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.967343+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.280117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734983+0000","last_change":"2024-03-21T14:02:37.454582+0000","last_active":"2024-03-21T14:03:42.734983+0000","last_peered":"2024-03-21T14:03:42.734983+0000","last_clean":"2024-03-21T14:03:42.734983+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:03:42.734983+0000","last_undegraded":"2024-03-21T14:03:42.734983+0000","last_fullsized":"2024-03-21T14:03:42.734983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.341912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995555+0000","last_change":"2024-03-21T14:02:39.473493+0000","last_active":"2024-03-21T14:03:42.995555+0000","last_peered":"2024-03-21T14:03:42.995555+0000","last_clean":"2024-03-21T14:03:42.995555+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:03:42.995555+0000","last_undegraded":"2024-03-21T14:03:42.995555+0000","last_fullsized":"2024-03-21T14:03:42.995555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.101298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999221+0000","last_change":"2024-03-21T14:02:33.443262+0000","last_active":"2024-03-21T14:03:42.999221+0000","last_peered":"2024-03-21T14:03:42.999221+0000","last_clean":"2024-03-21T14:03:42.999221+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:03:42.999221+0000","last_undegraded":"2024-03-21T14:03:42.999221+0000","last_fullsized":"2024-03-21T14:03:42.999221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.529943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757322+0000","last_change":"2024-03-21T14:03:38.812824+0000","last_active":"2024-03-21T14:03:43.757322+0000","last_peered":"2024-03-21T14:03:43.757322+0000","last_clean":"2024-03-21T14:03:43.757322+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:03:43.757322+0000","last_undegraded":"2024-03-21T14:03:43.757322+0000","last_fullsized":"2024-03-21T14:03:43.757322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:38.812732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.812732+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.836722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.499362+0000","last_change":"2024-03-21T14:02:37.459903+0000","last_active":"2024-03-21T14:03:43.499362+0000","last_peered":"2024-03-21T14:03:43.499362+0000","last_clean":"2024-03-21T14:03:43.499362+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:03:43.499362+0000","last_undegraded":"2024-03-21T14:03:43.499362+0000","last_fullsized":"2024-03-21T14:03:43.499362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532331+0000","last_change":"2024-03-21T14:02:39.517270+0000","last_active":"2024-03-21T14:03:43.532331+0000","last_peered":"2024-03-21T14:03:43.532331+0000","last_clean":"2024-03-21T14:03:43.532331+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:03:43.532331+0000","last_undegraded":"2024-03-21T14:03:43.532331+0000","last_fullsized":"2024-03-21T14:03:43.532331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.528174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.935772+0000","last_change":"2024-03-21T14:03:43.935798+0000","last_active":"2024-03-21T14:03:43.935772+0000","last_peered":"2024-03-21T14:03:43.935772+0000","last_clean":"2024-03-21T14:03:43.935772+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:03:43.935772+0000","last_undegraded":"2024-03-21T14:03:43.935772+0000","last_fullsized":"2024-03-21T14:03:43.935772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:43.935711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.935711+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.752006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389164+0000","last_change":"2024-03-21T14:02:35.444623+0000","last_active":"2024-03-21T14:03:42.389164+0000","last_peered":"2024-03-21T14:03:42.389164+0000","last_clean":"2024-03-21T14:03:42.389164+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:03:42.389164+0000","last_undegraded":"2024-03-21T14:03:42.389164+0000","last_fullsized":"2024-03-21T14:03:42.389164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.184966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":67,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.499360+0000","last_change":"2024-03-21T14:02:37.457785+0000","last_active":"2024-03-21T14:03:43.499360+0000","last_peered":"2024-03-21T14:03:43.499360+0000","last_clean":"2024-03-21T14:03:43.499360+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:03:43.499360+0000","last_undegraded":"2024-03-21T14:03:43.499360+0000","last_fullsized":"2024-03-21T14:03:43.499360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.398614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389071+0000","last_change":"2024-03-21T14:02:39.469558+0000","last_active":"2024-03-21T14:03:42.389071+0000","last_peered":"2024-03-21T14:03:42.389071+0000","last_clean":"2024-03-21T14:03:42.389071+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:03:42.389071+0000","last_undegraded":"2024-03-21T14:03:42.389071+0000","last_fullsized":"2024-03-21T14:03:42.389071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.811931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395781+0000","last_change":"2024-03-21T14:03:36.591924+0000","last_active":"2024-03-21T14:03:42.395781+0000","last_peered":"2024-03-21T14:03:42.395781+0000","last_clean":"2024-03-21T14:03:42.395781+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:03:42.395781+0000","last_undegraded":"2024-03-21T14:03:42.395781+0000","last_fullsized":"2024-03-21T14:03:42.395781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:03:36.591844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.332405+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:50.545354+0000","last_change":"2024-03-21T14:03:50.545394+0000","last_active":"2024-03-21T14:03:50.545354+0000","last_peered":"2024-03-21T14:03:50.545354+0000","last_clean":"2024-03-21T14:03:50.545354+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:03:50.545354+0000","last_undegraded":"2024-03-21T14:03:50.545354+0000","last_fullsized":"2024-03-21T14:03:50.545354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:03:50.545283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:50.545283+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.508308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388989+0000","last_change":"2024-03-21T14:02:37.456640+0000","last_active":"2024-03-21T14:03:42.388989+0000","last_peered":"2024-03-21T14:03:42.388989+0000","last_clean":"2024-03-21T14:03:42.388989+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:03:42.388989+0000","last_undegraded":"2024-03-21T14:03:42.388989+0000","last_fullsized":"2024-03-21T14:03:42.388989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.154776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999089+0000","last_change":"2024-03-21T14:02:39.471265+0000","last_active":"2024-03-21T14:03:42.999089+0000","last_peered":"2024-03-21T14:03:42.999089+0000","last_clean":"2024-03-21T14:03:42.999089+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:03:42.999089+0000","last_undegraded":"2024-03-21T14:03:42.999089+0000","last_fullsized":"2024-03-21T14:03:42.999089+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.256674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.499623+0000","last_change":"2024-03-21T14:03:43.001482+0000","last_active":"2024-03-21T14:03:43.499623+0000","last_peered":"2024-03-21T14:03:43.499623+0000","last_clean":"2024-03-21T14:03:43.499623+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:03:43.499623+0000","last_undegraded":"2024-03-21T14:03:43.499623+0000","last_fullsized":"2024-03-21T14:03:43.499623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:43.001363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.001363+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.127322+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394227+0000","last_change":"2024-03-21T14:02:33.444152+0000","last_active":"2024-03-21T14:03:42.394227+0000","last_peered":"2024-03-21T14:03:42.394227+0000","last_clean":"2024-03-21T14:03:42.394227+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:03:42.394227+0000","last_undegraded":"2024-03-21T14:03:42.394227+0000","last_fullsized":"2024-03-21T14:03:42.394227+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.592513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757597+0000","last_change":"2024-03-21T14:02:35.453150+0000","last_active":"2024-03-21T14:03:43.757597+0000","last_peered":"2024-03-21T14:03:43.757597+0000","last_clean":"2024-03-21T14:03:43.757597+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:03:43.757597+0000","last_undegraded":"2024-03-21T14:03:43.757597+0000","last_fullsized":"2024-03-21T14:03:43.757597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:45.011104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395610+0000","last_change":"2024-03-21T14:02:39.516061+0000","last_active":"2024-03-21T14:03:42.395610+0000","last_peered":"2024-03-21T14:03:42.395610+0000","last_clean":"2024-03-21T14:03:42.395610+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:03:42.395610+0000","last_undegraded":"2024-03-21T14:03:42.395610+0000","last_fullsized":"2024-03-21T14:03:42.395610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.869674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999626+0000","last_change":"2024-03-21T14:02:37.454216+0000","last_active":"2024-03-21T14:03:42.999626+0000","last_peered":"2024-03-21T14:03:42.999626+0000","last_clean":"2024-03-21T14:03:42.999626+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:03:42.999626+0000","last_undegraded":"2024-03-21T14:03:42.999626+0000","last_fullsized":"2024-03-21T14:03:42.999626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.938463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389026+0000","last_change":"2024-03-21T14:03:34.139285+0000","last_active":"2024-03-21T14:03:42.389026+0000","last_peered":"2024-03-21T14:03:42.389026+0000","last_clean":"2024-03-21T14:03:42.389026+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:03:42.389026+0000","last_undegraded":"2024-03-21T14:03:42.389026+0000","last_fullsized":"2024-03-21T14:03:42.389026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:34.139127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:34.139127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.352287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389671+0000","last_change":"2024-03-21T14:02:35.450014+0000","last_active":"2024-03-21T14:03:42.389671+0000","last_peered":"2024-03-21T14:03:42.389671+0000","last_clean":"2024-03-21T14:03:42.389671+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:03:42.389671+0000","last_undegraded":"2024-03-21T14:03:42.389671+0000","last_fullsized":"2024-03-21T14:03:42.389671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.855239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396108+0000","last_change":"2024-03-21T14:02:39.468574+0000","last_active":"2024-03-21T14:03:42.396108+0000","last_peered":"2024-03-21T14:03:42.396108+0000","last_clean":"2024-03-21T14:03:42.396108+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:03:42.396108+0000","last_undegraded":"2024-03-21T14:03:42.396108+0000","last_fullsized":"2024-03-21T14:03:42.396108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.471894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757692+0000","last_change":"2024-03-21T14:02:37.456947+0000","last_active":"2024-03-21T14:03:43.757692+0000","last_peered":"2024-03-21T14:03:43.757692+0000","last_clean":"2024-03-21T14:03:43.757692+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:03:43.757692+0000","last_undegraded":"2024-03-21T14:03:43.757692+0000","last_fullsized":"2024-03-21T14:03:43.757692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.581511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735065+0000","last_change":"2024-03-21T14:02:33.442462+0000","last_active":"2024-03-21T14:03:42.735065+0000","last_peered":"2024-03-21T14:03:42.735065+0000","last_clean":"2024-03-21T14:03:42.735065+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:03:42.735065+0000","last_undegraded":"2024-03-21T14:03:42.735065+0000","last_fullsized":"2024-03-21T14:03:42.735065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.441774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389894+0000","last_change":"2024-03-21T14:02:35.453988+0000","last_active":"2024-03-21T14:03:42.389894+0000","last_peered":"2024-03-21T14:03:42.389894+0000","last_clean":"2024-03-21T14:03:42.389894+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:03:42.389894+0000","last_undegraded":"2024-03-21T14:03:42.389894+0000","last_fullsized":"2024-03-21T14:03:42.389894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:43.858976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389971+0000","last_change":"2024-03-21T14:02:39.459328+0000","last_active":"2024-03-21T14:03:42.389971+0000","last_peered":"2024-03-21T14:03:42.389971+0000","last_clean":"2024-03-21T14:03:42.389971+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:03:42.389971+0000","last_undegraded":"2024-03-21T14:03:42.389971+0000","last_fullsized":"2024-03-21T14:03:42.389971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.833817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395780+0000","last_change":"2024-03-21T14:02:37.460196+0000","last_active":"2024-03-21T14:03:42.395780+0000","last_peered":"2024-03-21T14:03:42.395780+0000","last_clean":"2024-03-21T14:03:42.395780+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:03:42.395780+0000","last_undegraded":"2024-03-21T14:03:42.395780+0000","last_fullsized":"2024-03-21T14:03:42.395780+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.085035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395341+0000","last_change":"2024-03-21T14:02:33.442529+0000","last_active":"2024-03-21T14:03:42.395341+0000","last_peered":"2024-03-21T14:03:42.395341+0000","last_clean":"2024-03-21T14:03:42.395341+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:03:42.395341+0000","last_undegraded":"2024-03-21T14:03:42.395341+0000","last_fullsized":"2024-03-21T14:03:42.395341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.804413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389402+0000","last_change":"2024-03-21T14:02:35.450511+0000","last_active":"2024-03-21T14:03:42.389402+0000","last_peered":"2024-03-21T14:03:42.389402+0000","last_clean":"2024-03-21T14:03:42.389402+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:03:42.389402+0000","last_undegraded":"2024-03-21T14:03:42.389402+0000","last_fullsized":"2024-03-21T14:03:42.389402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.494861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103188,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":927293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430233,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28328,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716824,"statfs":{"total":95995035648,"available":95966027776,"internally_reserved":0,"allocated":1343488,"data_stored":925330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":337516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":926124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713232,"statfs":{"total":95995035648,"available":95962349568,"internally_reserved":0,"allocated":757760,"data_stored":335310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":794624,"data_stored":336302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":786432,"data_stored":336016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691871,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":336557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:03:56.269 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T14:03:56.269 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-21T14:03:56.269 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-21T14:03:56.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-21T14:03:56.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:03:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:03:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:03:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:03:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:03:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:03:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[29576]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:03:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:03:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:03:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:03:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:03:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[40510]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:03:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:03:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:56 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:03:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:56 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:03:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:56 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:03:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:56 smithi118 ceph-mon[33862]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:03:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:56 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:03:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:56 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:03:56.962 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:03:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:03:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:57 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:03:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:57 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[29576]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:03:58.828 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[40510]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:58.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:03:58.829 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:58 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:58 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:58 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:58 smithi118 ceph-mon[33862]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:58 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:03:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:58 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:03:59.104 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:03:59.104 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-21T14:03:59.104 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 podman[75577]: 2024-03-21 14:03:59.029413366 +0000 UTC m=+0.444972937 container create 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:03:59.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:03:59.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:03:59.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4064813465' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T14:03:59.777 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:59] ENGINE Bus STOPPING 2024-03-21T14:03:59.778 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:59] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:03:59.778 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:59] ENGINE Bus STOPPED 2024-03-21T14:03:59.778 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:59] ENGINE Bus STARTING 2024-03-21T14:03:59.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:03:59.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:03:59.778 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4064813465' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T14:03:59.778 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 podman[75577]: 2024-03-21 14:03:59.508175848 +0000 UTC m=+0.923735425 container init 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 podman[75577]: 2024-03-21 14:03:59.510819916 +0000 UTC m=+0.926379478 container start 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.521Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.521Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.522Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.523Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.545Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.546Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.547Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:03:59.547Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 bash[75577]: 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 2024-03-21T14:03:59.779 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:03:59 smithi003 systemd[1]: Started Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:03:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:59 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:03:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:59 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:03:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:03:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4064813465' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-21T14:04:00.185 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:59] ENGINE Serving on http://:::9283 2024-03-21T14:04:00.186 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:03:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:03:59] ENGINE Bus STARTED 2024-03-21T14:04:00.215 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-21T14:04:00.215 INFO:tasks.cephadm:Setup complete, yielding 2024-03-21T14:04:00.216 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-21T14:04:00.226 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-21T14:04:00.226 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-adk-testing-2024-03-20-2326'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'OSDMAP_FLAGS', 'PG_DEGRADED', 'OSD_DOWN', 'POOL_APP_NOT_ENABLED', 'CACHE_POOL_NO_HIT_SET', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-03-20-2326', 'sha1': 'ba760091cd7bd2b0d23f4825ac856ba66450e988'}} 2024-03-21T14:04:00.227 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-21T14:04:00.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-21T14:04:00.482 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: Deploying daemon grafana.a on smithi118 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: 5.19 deep-scrub starts 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: 5.19 deep-scrub ok 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:00 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:00.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: Deploying daemon grafana.a on smithi118 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: 5.19 deep-scrub starts 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: 5.19 deep-scrub ok 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:04:00.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-21T14:04:00.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: Deploying daemon grafana.a on smithi118 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: 5.19 deep-scrub starts 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: 5.19 deep-scrub ok 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:00.935 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:00 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:01.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:01 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:04:01.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:01 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:04:01.811 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:01 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:04:01.811 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:01 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:04:01.811 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:04:01 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:04:01.524Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000959563s 2024-03-21T14:04:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:01 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:04:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:01 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:04:02.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:04:02.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:04:02.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[29576]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:02.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:04:02] ENGINE Bus STOPPING 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:04:02] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:04:02] ENGINE Bus STOPPED 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:04:02] ENGINE Bus STARTING 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:04:02] ENGINE Serving on http://:::9283 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:04:02] ENGINE Bus STARTED 2024-03-21T14:04:02.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:04:02.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:04:02.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[40510]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:02.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:04:02.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:04:02.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:02 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' 2024-03-21T14:04:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:02 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:04:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:02 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:04:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:02 smithi118 ceph-mon[33862]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:02 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:04:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:02 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:04:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:02 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' 2024-03-21T14:04:03.118 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-21T14:04:03.119 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-21T14:04:03.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:04:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:04:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:04:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:04:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:04:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:04:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:04:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:04:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:04 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:04:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:04:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:04:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:04:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:04:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:04:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:04 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:04:05.139 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:04:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:04:05.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[29576]: 6.2 deep-scrub starts 2024-03-21T14:04:05.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[29576]: 6.2 deep-scrub ok 2024-03-21T14:04:05.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:04:05.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:04:05.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[40510]: 6.2 deep-scrub starts 2024-03-21T14:04:05.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[40510]: 6.2 deep-scrub ok 2024-03-21T14:04:05.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:04:05.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:05 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:04:05.821 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-21T14:04:03.252473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1015,"num_read_kb":861,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":615,"ondisk_log_size":615,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253052,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708164,"statfs":{"total":767960285184,"available":767701159936,"internally_reserved":0,"allocated":7991296,"data_stored":4460448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003085"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389060+0000","last_change":"2024-03-21T14:02:39.463103+0000","last_active":"2024-03-21T14:03:42.389060+0000","last_peered":"2024-03-21T14:03:42.389060+0000","last_clean":"2024-03-21T14:03:42.389060+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:03:42.389060+0000","last_undegraded":"2024-03-21T14:03:42.389060+0000","last_fullsized":"2024-03-21T14:03:42.389060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.364725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.788738+0000","last_change":"2024-03-21T14:03:59.788760+0000","last_active":"2024-03-21T14:03:59.788738+0000","last_peered":"2024-03-21T14:03:59.788738+0000","last_clean":"2024-03-21T14:03:59.788738+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:03:59.788738+0000","last_undegraded":"2024-03-21T14:03:59.788738+0000","last_fullsized":"2024-03-21T14:03:59.788738+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:59.788675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:59.788675+0000","last_clean_scrub_stamp":"2024-03-21T14:03:59.788675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:03.612884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.957706+0000","last_change":"2024-03-21T14:03:46.957726+0000","last_active":"2024-03-21T14:03:46.957706+0000","last_peered":"2024-03-21T14:03:46.957706+0000","last_clean":"2024-03-21T14:03:46.957706+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:03:46.957706+0000","last_undegraded":"2024-03-21T14:03:46.957706+0000","last_fullsized":"2024-03-21T14:03:46.957706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.957648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.957648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.716204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388929+0000","last_change":"2024-03-21T14:02:35.450207+0000","last_active":"2024-03-21T14:03:42.388929+0000","last_peered":"2024-03-21T14:03:42.388929+0000","last_clean":"2024-03-21T14:03:42.388929+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:03:42.388929+0000","last_undegraded":"2024-03-21T14:03:42.388929+0000","last_fullsized":"2024-03-21T14:03:42.388929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.595411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395241+0000","last_change":"2024-03-21T14:02:39.517587+0000","last_active":"2024-03-21T14:03:42.395241+0000","last_peered":"2024-03-21T14:03:42.395241+0000","last_clean":"2024-03-21T14:03:42.395241+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:03:42.395241+0000","last_undegraded":"2024-03-21T14:03:42.395241+0000","last_fullsized":"2024-03-21T14:03:42.395241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.320905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.967003+0000","last_change":"2024-03-21T14:03:53.967027+0000","last_active":"2024-03-21T14:03:53.967003+0000","last_peered":"2024-03-21T14:03:53.967003+0000","last_clean":"2024-03-21T14:03:53.967003+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:03:53.967003+0000","last_undegraded":"2024-03-21T14:03:53.967003+0000","last_fullsized":"2024-03-21T14:03:53.967003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.966920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.966920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.338422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396378+0000","last_change":"2024-03-21T14:03:37.567249+0000","last_active":"2024-03-21T14:03:42.396378+0000","last_peered":"2024-03-21T14:03:42.396378+0000","last_clean":"2024-03-21T14:03:42.396378+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:03:42.396378+0000","last_undegraded":"2024-03-21T14:03:42.396378+0000","last_fullsized":"2024-03-21T14:03:42.396378+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:37.567167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.567167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.628456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394378+0000","last_change":"2024-03-21T14:03:35.571620+0000","last_active":"2024-03-21T14:03:42.394378+0000","last_peered":"2024-03-21T14:03:42.394378+0000","last_clean":"2024-03-21T14:03:42.394378+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:03:42.394378+0000","last_undegraded":"2024-03-21T14:03:42.394378+0000","last_fullsized":"2024-03-21T14:03:42.394378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.571531+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:41.701221+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995806+0000","last_change":"2024-03-21T14:02:39.469561+0000","last_active":"2024-03-21T14:03:42.995806+0000","last_peered":"2024-03-21T14:03:42.995806+0000","last_clean":"2024-03-21T14:03:42.995806+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:03:42.995806+0000","last_undegraded":"2024-03-21T14:03:42.995806+0000","last_fullsized":"2024-03-21T14:03:42.995806+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.240000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999035+0000","last_change":"2024-03-21T14:02:37.455588+0000","last_active":"2024-03-21T14:03:42.999035+0000","last_peered":"2024-03-21T14:03:42.999035+0000","last_clean":"2024-03-21T14:03:42.999035+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:03:42.999035+0000","last_undegraded":"2024-03-21T14:03:42.999035+0000","last_fullsized":"2024-03-21T14:03:42.999035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.535187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999090+0000","last_change":"2024-03-21T14:02:33.441047+0000","last_active":"2024-03-21T14:03:42.999090+0000","last_peered":"2024-03-21T14:03:42.999090+0000","last_clean":"2024-03-21T14:03:42.999090+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:03:42.999090+0000","last_undegraded":"2024-03-21T14:03:42.999090+0000","last_fullsized":"2024-03-21T14:03:42.999090+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.089986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389241+0000","last_change":"2024-03-21T14:02:35.445770+0000","last_active":"2024-03-21T14:03:42.389241+0000","last_peered":"2024-03-21T14:03:42.389241+0000","last_clean":"2024-03-21T14:03:42.389241+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:03:42.389241+0000","last_undegraded":"2024-03-21T14:03:42.389241+0000","last_fullsized":"2024-03-21T14:03:42.389241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.203589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.410037+0000","last_change":"2024-03-21T14:03:56.410063+0000","last_active":"2024-03-21T14:03:56.410037+0000","last_peered":"2024-03-21T14:03:56.410037+0000","last_clean":"2024-03-21T14:03:56.410037+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:03:56.410037+0000","last_undegraded":"2024-03-21T14:03:56.410037+0000","last_fullsized":"2024-03-21T14:03:56.410037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.409946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.409946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:17.565720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532890+0000","last_change":"2024-03-21T14:02:37.457212+0000","last_active":"2024-03-21T14:03:43.532890+0000","last_peered":"2024-03-21T14:03:43.532890+0000","last_clean":"2024-03-21T14:03:43.532890+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:03:43.532890+0000","last_undegraded":"2024-03-21T14:03:43.532890+0000","last_fullsized":"2024-03-21T14:03:43.532890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.357086+0000","last_change":"2024-03-21T14:03:46.357111+0000","last_active":"2024-03-21T14:03:46.357086+0000","last_peered":"2024-03-21T14:03:46.357086+0000","last_clean":"2024-03-21T14:03:46.357086+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:03:46.357086+0000","last_undegraded":"2024-03-21T14:03:46.357086+0000","last_fullsized":"2024-03-21T14:03:46.357086+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.357008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.357008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.267704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389583+0000","last_change":"2024-03-21T14:02:35.453933+0000","last_active":"2024-03-21T14:03:42.389583+0000","last_peered":"2024-03-21T14:03:42.389583+0000","last_clean":"2024-03-21T14:03:42.389583+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:03:42.389583+0000","last_undegraded":"2024-03-21T14:03:42.389583+0000","last_fullsized":"2024-03-21T14:03:42.389583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.127706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389793+0000","last_change":"2024-03-21T14:02:39.516305+0000","last_active":"2024-03-21T14:03:42.389793+0000","last_peered":"2024-03-21T14:03:42.389793+0000","last_clean":"2024-03-21T14:03:42.389793+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:03:42.389793+0000","last_undegraded":"2024-03-21T14:03:42.389793+0000","last_fullsized":"2024-03-21T14:03:42.389793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.252625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757380+0000","last_change":"2024-03-21T14:02:37.459191+0000","last_active":"2024-03-21T14:03:43.757380+0000","last_peered":"2024-03-21T14:03:43.757380+0000","last_clean":"2024-03-21T14:03:43.757380+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:03:43.757380+0000","last_undegraded":"2024-03-21T14:03:43.757380+0000","last_fullsized":"2024-03-21T14:03:43.757380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.913816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.004769+0000","last_change":"2024-03-21T14:03:53.004791+0000","last_active":"2024-03-21T14:03:53.004769+0000","last_peered":"2024-03-21T14:03:53.004769+0000","last_clean":"2024-03-21T14:03:53.004769+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:03:53.004769+0000","last_undegraded":"2024-03-21T14:03:53.004769+0000","last_fullsized":"2024-03-21T14:03:53.004769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.004692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.004692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.564997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:48.070675+0000","last_change":"2024-03-21T14:03:48.070734+0000","last_active":"2024-03-21T14:03:48.070675+0000","last_peered":"2024-03-21T14:03:48.070675+0000","last_clean":"2024-03-21T14:03:48.070675+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:03:48.070675+0000","last_undegraded":"2024-03-21T14:03:48.070675+0000","last_fullsized":"2024-03-21T14:03:48.070675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:03:48.070599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:48.070599+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.334848+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395547+0000","last_change":"2024-03-21T14:02:39.517386+0000","last_active":"2024-03-21T14:03:42.395547+0000","last_peered":"2024-03-21T14:03:42.395547+0000","last_clean":"2024-03-21T14:03:42.395547+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:03:42.395547+0000","last_undegraded":"2024-03-21T14:03:42.395547+0000","last_fullsized":"2024-03-21T14:03:42.395547+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.220059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389360+0000","last_change":"2024-03-21T14:02:37.458298+0000","last_active":"2024-03-21T14:03:42.389360+0000","last_peered":"2024-03-21T14:03:42.389360+0000","last_clean":"2024-03-21T14:03:42.389360+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:03:42.389360+0000","last_undegraded":"2024-03-21T14:03:42.389360+0000","last_fullsized":"2024-03-21T14:03:42.389360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.474405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.949455+0000","last_change":"2024-03-21T14:03:47.949482+0000","last_active":"2024-03-21T14:03:47.949455+0000","last_peered":"2024-03-21T14:03:47.949455+0000","last_clean":"2024-03-21T14:03:47.949455+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:03:47.949455+0000","last_undegraded":"2024-03-21T14:03:47.949455+0000","last_fullsized":"2024-03-21T14:03:47.949455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.949388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.443300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395701+0000","last_change":"2024-03-21T14:02:35.451688+0000","last_active":"2024-03-21T14:03:42.395701+0000","last_peered":"2024-03-21T14:03:42.395701+0000","last_clean":"2024-03-21T14:03:42.395701+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:03:42.395701+0000","last_undegraded":"2024-03-21T14:03:42.395701+0000","last_fullsized":"2024-03-21T14:03:42.395701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.784537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:52.884156+0000","last_change":"2024-03-21T14:03:52.884209+0000","last_active":"2024-03-21T14:03:52.884156+0000","last_peered":"2024-03-21T14:03:52.884156+0000","last_clean":"2024-03-21T14:03:52.884156+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:03:52.884156+0000","last_undegraded":"2024-03-21T14:03:52.884156+0000","last_fullsized":"2024-03-21T14:03:52.884156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:03:52.884054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:52.884054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:00.091462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501685+0000","last_change":"2024-03-21T14:02:37.455355+0000","last_active":"2024-03-21T14:03:53.501685+0000","last_peered":"2024-03-21T14:03:53.501685+0000","last_clean":"2024-03-21T14:03:53.501685+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:03:53.501685+0000","last_undegraded":"2024-03-21T14:03:53.501685+0000","last_fullsized":"2024-03-21T14:03:53.501685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.710291+0000","last_change":"2024-03-21T14:03:54.710330+0000","last_active":"2024-03-21T14:03:54.710291+0000","last_peered":"2024-03-21T14:03:54.710291+0000","last_clean":"2024-03-21T14:03:54.710291+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:03:54.710291+0000","last_undegraded":"2024-03-21T14:03:54.710291+0000","last_fullsized":"2024-03-21T14:03:54.710291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.710220+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.710102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:44.904151+0000","last_change":"2024-03-21T14:03:44.904204+0000","last_active":"2024-03-21T14:03:44.904151+0000","last_peered":"2024-03-21T14:03:44.904151+0000","last_clean":"2024-03-21T14:03:44.904151+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:03:44.904151+0000","last_undegraded":"2024-03-21T14:03:44.904151+0000","last_fullsized":"2024-03-21T14:03:44.904151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:44.904071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:44.904071+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.807993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756221+0000","last_change":"2024-03-21T14:02:39.465886+0000","last_active":"2024-03-21T14:03:43.756221+0000","last_peered":"2024-03-21T14:03:43.756221+0000","last_clean":"2024-03-21T14:03:43.756221+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:03:43.756221+0000","last_undegraded":"2024-03-21T14:03:43.756221+0000","last_fullsized":"2024-03-21T14:03:43.756221+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.933267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996574+0000","last_change":"2024-03-21T14:02:37.463347+0000","last_active":"2024-03-21T14:03:42.996574+0000","last_peered":"2024-03-21T14:03:42.996574+0000","last_clean":"2024-03-21T14:03:42.996574+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:03:42.996574+0000","last_undegraded":"2024-03-21T14:03:42.996574+0000","last_fullsized":"2024-03-21T14:03:42.996574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.394323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394109+0000","last_change":"2024-03-21T14:03:39.522391+0000","last_active":"2024-03-21T14:03:42.394109+0000","last_peered":"2024-03-21T14:03:42.394109+0000","last_clean":"2024-03-21T14:03:42.394109+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:03:42.394109+0000","last_undegraded":"2024-03-21T14:03:42.394109+0000","last_fullsized":"2024-03-21T14:03:42.394109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.522309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.085948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.279816+0000","last_change":"2024-03-21T14:03:53.279859+0000","last_active":"2024-03-21T14:03:53.279816+0000","last_peered":"2024-03-21T14:03:53.279816+0000","last_clean":"2024-03-21T14:03:53.279816+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:03:53.279816+0000","last_undegraded":"2024-03-21T14:03:53.279816+0000","last_fullsized":"2024-03-21T14:03:53.279816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:03:53.279738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.279738+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:57.232403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.862928+0000","last_change":"2024-03-21T14:03:59.862965+0000","last_active":"2024-03-21T14:03:59.862928+0000","last_peered":"2024-03-21T14:03:59.862928+0000","last_clean":"2024-03-21T14:03:59.862928+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:03:59.862928+0000","last_undegraded":"2024-03-21T14:03:59.862928+0000","last_fullsized":"2024-03-21T14:03:59.862928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:59.862840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:59.862840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.546256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734740+0000","last_change":"2024-03-21T14:02:37.456532+0000","last_active":"2024-03-21T14:03:42.734740+0000","last_peered":"2024-03-21T14:03:42.734740+0000","last_clean":"2024-03-21T14:03:42.734740+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:03:42.734740+0000","last_undegraded":"2024-03-21T14:03:42.734740+0000","last_fullsized":"2024-03-21T14:03:42.734740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.215037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.916303+0000","last_change":"2024-03-21T14:03:47.916324+0000","last_active":"2024-03-21T14:03:47.916303+0000","last_peered":"2024-03-21T14:03:47.916303+0000","last_clean":"2024-03-21T14:03:47.916303+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:03:47.916303+0000","last_undegraded":"2024-03-21T14:03:47.916303+0000","last_fullsized":"2024-03-21T14:03:47.916303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.916245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.916245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.354491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.509555+0000","last_change":"2024-03-21T14:03:45.509584+0000","last_active":"2024-03-21T14:03:45.509555+0000","last_peered":"2024-03-21T14:03:45.509555+0000","last_clean":"2024-03-21T14:03:45.509555+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:03:45.509555+0000","last_undegraded":"2024-03-21T14:03:45.509555+0000","last_fullsized":"2024-03-21T14:03:45.509555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:03:45.509492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.509492+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.086854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395138+0000","last_change":"2024-03-21T14:02:39.517579+0000","last_active":"2024-03-21T14:03:42.395138+0000","last_peered":"2024-03-21T14:03:42.395138+0000","last_clean":"2024-03-21T14:03:42.395138+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:03:42.395138+0000","last_undegraded":"2024-03-21T14:03:42.395138+0000","last_fullsized":"2024-03-21T14:03:42.395138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.492381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.836823+0000","last_change":"2024-03-21T14:03:47.836847+0000","last_active":"2024-03-21T14:03:47.836823+0000","last_peered":"2024-03-21T14:03:47.836823+0000","last_clean":"2024-03-21T14:03:47.836823+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:03:47.836823+0000","last_undegraded":"2024-03-21T14:03:47.836823+0000","last_fullsized":"2024-03-21T14:03:47.836823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.836751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.836751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:07.563311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:03:41.362147+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:41.362005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.362005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:10.804533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.598777+0000","last_change":"2024-03-21T14:03:54.598813+0000","last_active":"2024-03-21T14:03:54.598777+0000","last_peered":"2024-03-21T14:03:54.598777+0000","last_clean":"2024-03-21T14:03:54.598777+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:03:54.598777+0000","last_undegraded":"2024-03-21T14:03:54.598777+0000","last_fullsized":"2024-03-21T14:03:54.598777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:03:54.598701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.598701+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:59.201207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995540+0000","last_change":"2024-03-21T14:02:39.471220+0000","last_active":"2024-03-21T14:03:42.995540+0000","last_peered":"2024-03-21T14:03:42.995540+0000","last_clean":"2024-03-21T14:03:42.995540+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:03:42.995540+0000","last_undegraded":"2024-03-21T14:03:42.995540+0000","last_fullsized":"2024-03-21T14:03:42.995540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.470185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.547235+0000","last_change":"2024-03-21T14:03:53.547267+0000","last_active":"2024-03-21T14:03:53.547235+0000","last_peered":"2024-03-21T14:03:53.547235+0000","last_clean":"2024-03-21T14:03:53.547235+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:03:53.547235+0000","last_undegraded":"2024-03-21T14:03:53.547235+0000","last_fullsized":"2024-03-21T14:03:53.547235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.547160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.547160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:19.476340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533251+0000","last_change":"2024-03-21T14:02:33.449232+0000","last_active":"2024-03-21T14:03:43.533251+0000","last_peered":"2024-03-21T14:03:43.533251+0000","last_clean":"2024-03-21T14:03:43.533251+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:03:43.533251+0000","last_undegraded":"2024-03-21T14:03:43.533251+0000","last_fullsized":"2024-03-21T14:03:43.533251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.141490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.759191+0000","last_change":"2024-03-21T14:03:43.759222+0000","last_active":"2024-03-21T14:03:43.759191+0000","last_peered":"2024-03-21T14:03:43.759191+0000","last_clean":"2024-03-21T14:03:43.759191+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:03:43.759191+0000","last_undegraded":"2024-03-21T14:03:43.759191+0000","last_fullsized":"2024-03-21T14:03:43.759191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:03:43.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.759108+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.080054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394090+0000","last_change":"2024-03-21T14:02:39.471886+0000","last_active":"2024-03-21T14:03:42.394090+0000","last_peered":"2024-03-21T14:03:42.394090+0000","last_clean":"2024-03-21T14:03:42.394090+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:03:42.394090+0000","last_undegraded":"2024-03-21T14:03:42.394090+0000","last_fullsized":"2024-03-21T14:03:42.394090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.715774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757272+0000","last_change":"2024-03-21T14:02:37.455324+0000","last_active":"2024-03-21T14:03:43.757272+0000","last_peered":"2024-03-21T14:03:43.757272+0000","last_clean":"2024-03-21T14:03:43.757272+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:03:43.757272+0000","last_undegraded":"2024-03-21T14:03:43.757272+0000","last_fullsized":"2024-03-21T14:03:43.757272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.837893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.994403+0000","last_change":"2024-03-21T14:03:46.994427+0000","last_active":"2024-03-21T14:03:46.994403+0000","last_peered":"2024-03-21T14:03:46.994403+0000","last_clean":"2024-03-21T14:03:46.994403+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:03:46.994403+0000","last_undegraded":"2024-03-21T14:03:46.994403+0000","last_fullsized":"2024-03-21T14:03:46.994403+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.994338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.994338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.043818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389901+0000","last_change":"2024-03-21T14:02:58.514129+0000","last_active":"2024-03-21T14:03:42.389901+0000","last_peered":"2024-03-21T14:03:42.389901+0000","last_clean":"2024-03-21T14:03:42.389901+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:03:42.389901+0000","last_undegraded":"2024-03-21T14:03:42.389901+0000","last_fullsized":"2024-03-21T14:03:42.389901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.398980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.031159+0000","last_change":"2024-03-21T14:03:58.031180+0000","last_active":"2024-03-21T14:03:58.031159+0000","last_peered":"2024-03-21T14:03:58.031159+0000","last_clean":"2024-03-21T14:03:58.031159+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:03:58.031159+0000","last_undegraded":"2024-03-21T14:03:58.031159+0000","last_fullsized":"2024-03-21T14:03:58.031159+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:58.031098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:58.031098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:13.713947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501685+0000","last_change":"2024-03-21T14:03:37.352147+0000","last_active":"2024-03-21T14:03:53.501685+0000","last_peered":"2024-03-21T14:03:53.501685+0000","last_clean":"2024-03-21T14:03:53.501685+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:03:53.501685+0000","last_undegraded":"2024-03-21T14:03:53.501685+0000","last_fullsized":"2024-03-21T14:03:53.501685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:37.352012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.352012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.843950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533005+0000","last_change":"2024-03-21T14:02:33.446863+0000","last_active":"2024-03-21T14:03:43.533005+0000","last_peered":"2024-03-21T14:03:43.533005+0000","last_clean":"2024-03-21T14:03:43.533005+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:03:43.533005+0000","last_undegraded":"2024-03-21T14:03:43.533005+0000","last_fullsized":"2024-03-21T14:03:43.533005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.983498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395982+0000","last_change":"2024-03-21T14:02:35.775119+0000","last_active":"2024-03-21T14:03:42.395982+0000","last_peered":"2024-03-21T14:03:42.395982+0000","last_clean":"2024-03-21T14:03:42.395982+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:03:42.395982+0000","last_undegraded":"2024-03-21T14:03:42.395982+0000","last_fullsized":"2024-03-21T14:03:42.395982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.079104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.390025+0000","last_change":"2024-03-21T14:02:39.467603+0000","last_active":"2024-03-21T14:03:42.390025+0000","last_peered":"2024-03-21T14:03:42.390025+0000","last_clean":"2024-03-21T14:03:42.390025+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:03:42.390025+0000","last_undegraded":"2024-03-21T14:03:42.390025+0000","last_fullsized":"2024-03-21T14:03:42.390025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.883243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501616+0000","last_change":"2024-03-21T14:03:47.500013+0000","last_active":"2024-03-21T14:03:53.501616+0000","last_peered":"2024-03-21T14:03:53.501616+0000","last_clean":"2024-03-21T14:03:53.501616+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:03:53.501616+0000","last_undegraded":"2024-03-21T14:03:53.501616+0000","last_fullsized":"2024-03-21T14:03:53.501616+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:47.499917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.499917+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.440649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995810+0000","last_change":"2024-03-21T14:02:33.439871+0000","last_active":"2024-03-21T14:03:42.995810+0000","last_peered":"2024-03-21T14:03:42.995810+0000","last_clean":"2024-03-21T14:03:42.995810+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:03:42.995810+0000","last_undegraded":"2024-03-21T14:03:42.995810+0000","last_fullsized":"2024-03-21T14:03:42.995810+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.912588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:02:35.775497+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:03.620820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:01.013228+0000","last_change":"2024-03-21T14:04:01.013250+0000","last_active":"2024-03-21T14:04:01.013228+0000","last_peered":"2024-03-21T14:04:01.013228+0000","last_clean":"2024-03-21T14:04:01.013228+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:04:01.013228+0000","last_undegraded":"2024-03-21T14:04:01.013228+0000","last_fullsized":"2024-03-21T14:04:01.013228+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:01.013167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:01.013167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:25.253471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396096+0000","last_change":"2024-03-21T14:02:37.456859+0000","last_active":"2024-03-21T14:03:42.396096+0000","last_peered":"2024-03-21T14:03:42.396096+0000","last_clean":"2024-03-21T14:03:42.396096+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:03:42.396096+0000","last_undegraded":"2024-03-21T14:03:42.396096+0000","last_fullsized":"2024-03-21T14:03:42.396096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.881127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532903+0000","last_change":"2024-03-21T14:03:38.848893+0000","last_active":"2024-03-21T14:03:43.532903+0000","last_peered":"2024-03-21T14:03:43.532903+0000","last_clean":"2024-03-21T14:03:43.532903+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:03:43.532903+0000","last_undegraded":"2024-03-21T14:03:43.532903+0000","last_fullsized":"2024-03-21T14:03:43.532903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.848793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.848793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.951970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996491+0000","last_change":"2024-03-21T14:02:35.774922+0000","last_active":"2024-03-21T14:03:42.996491+0000","last_peered":"2024-03-21T14:03:42.996491+0000","last_clean":"2024-03-21T14:03:42.996491+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:03:42.996491+0000","last_undegraded":"2024-03-21T14:03:42.996491+0000","last_fullsized":"2024-03-21T14:03:42.996491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.077976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532613+0000","last_change":"2024-03-21T14:03:41.906118+0000","last_active":"2024-03-21T14:03:43.532613+0000","last_peered":"2024-03-21T14:03:43.532613+0000","last_clean":"2024-03-21T14:03:43.532613+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:03:43.532613+0000","last_undegraded":"2024-03-21T14:03:43.532613+0000","last_fullsized":"2024-03-21T14:03:43.532613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.906016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.906016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:46.096482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999520+0000","last_change":"2024-03-21T14:02:37.454805+0000","last_active":"2024-03-21T14:03:42.999520+0000","last_peered":"2024-03-21T14:03:42.999520+0000","last_clean":"2024-03-21T14:03:42.999520+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:03:42.999520+0000","last_undegraded":"2024-03-21T14:03:42.999520+0000","last_fullsized":"2024-03-21T14:03:42.999520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.374515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734655+0000","last_change":"2024-03-21T14:03:38.199089+0000","last_active":"2024-03-21T14:03:42.734655+0000","last_peered":"2024-03-21T14:03:42.734655+0000","last_clean":"2024-03-21T14:03:42.734655+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:03:42.734655+0000","last_undegraded":"2024-03-21T14:03:42.734655+0000","last_fullsized":"2024-03-21T14:03:42.734655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.198986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.198986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.188712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395952+0000","last_change":"2024-03-21T14:02:35.774132+0000","last_active":"2024-03-21T14:03:42.395952+0000","last_peered":"2024-03-21T14:03:42.395952+0000","last_clean":"2024-03-21T14:03:42.395952+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:03:42.395952+0000","last_undegraded":"2024-03-21T14:03:42.395952+0000","last_fullsized":"2024-03-21T14:03:42.395952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.386896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999034+0000","last_change":"2024-03-21T14:02:39.517249+0000","last_active":"2024-03-21T14:03:42.999034+0000","last_peered":"2024-03-21T14:03:42.999034+0000","last_clean":"2024-03-21T14:03:42.999034+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:03:42.999034+0000","last_undegraded":"2024-03-21T14:03:42.999034+0000","last_fullsized":"2024-03-21T14:03:42.999034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.493578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533577+0000","last_change":"2024-03-21T14:02:33.448412+0000","last_active":"2024-03-21T14:03:43.533577+0000","last_peered":"2024-03-21T14:03:43.533577+0000","last_clean":"2024-03-21T14:03:43.533577+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:03:43.533577+0000","last_undegraded":"2024-03-21T14:03:43.533577+0000","last_fullsized":"2024-03-21T14:03:43.533577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.124819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998643+0000","last_change":"2024-03-21T14:02:35.775475+0000","last_active":"2024-03-21T14:03:42.998643+0000","last_peered":"2024-03-21T14:03:42.998643+0000","last_clean":"2024-03-21T14:03:42.998643+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:03:42.998643+0000","last_undegraded":"2024-03-21T14:03:42.998643+0000","last_fullsized":"2024-03-21T14:03:42.998643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.820694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501686+0000","last_change":"2024-03-21T14:02:37.454898+0000","last_active":"2024-03-21T14:03:53.501686+0000","last_peered":"2024-03-21T14:03:53.501686+0000","last_clean":"2024-03-21T14:03:53.501686+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:03:53.501686+0000","last_undegraded":"2024-03-21T14:03:53.501686+0000","last_fullsized":"2024-03-21T14:03:53.501686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.360256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395170+0000","last_change":"2024-03-21T14:02:39.472019+0000","last_active":"2024-03-21T14:03:42.395170+0000","last_peered":"2024-03-21T14:03:42.395170+0000","last_clean":"2024-03-21T14:03:42.395170+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:03:42.395170+0000","last_undegraded":"2024-03-21T14:03:42.395170+0000","last_fullsized":"2024-03-21T14:03:42.395170+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.761931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.909142+0000","last_change":"2024-03-21T14:03:45.909165+0000","last_active":"2024-03-21T14:03:45.909142+0000","last_peered":"2024-03-21T14:03:45.909142+0000","last_clean":"2024-03-21T14:03:45.909142+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:03:45.909142+0000","last_undegraded":"2024-03-21T14:03:45.909142+0000","last_fullsized":"2024-03-21T14:03:45.909142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:45.909064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.909064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.828690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389532+0000","last_change":"2024-03-21T14:02:35.450101+0000","last_active":"2024-03-21T14:03:42.389532+0000","last_peered":"2024-03-21T14:03:42.389532+0000","last_clean":"2024-03-21T14:03:42.389532+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:03:42.389532+0000","last_undegraded":"2024-03-21T14:03:42.389532+0000","last_fullsized":"2024-03-21T14:03:42.389532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.268875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.969419+0000","last_change":"2024-03-21T14:03:59.969457+0000","last_active":"2024-03-21T14:03:59.969419+0000","last_peered":"2024-03-21T14:03:59.969419+0000","last_clean":"2024-03-21T14:03:59.969419+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:03:59.969419+0000","last_undegraded":"2024-03-21T14:03:59.969419+0000","last_fullsized":"2024-03-21T14:03:59.969419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:59.969354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:59.969354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:10.864954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.825087+0000","last_change":"2024-03-21T14:03:58.825134+0000","last_active":"2024-03-21T14:03:58.825087+0000","last_peered":"2024-03-21T14:03:58.825087+0000","last_clean":"2024-03-21T14:03:58.825087+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:03:58.825087+0000","last_undegraded":"2024-03-21T14:03:58.825087+0000","last_fullsized":"2024-03-21T14:03:58.825087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:58.825002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:58.825002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:18.466884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395196+0000","last_change":"2024-03-21T14:03:39.021466+0000","last_active":"2024-03-21T14:03:42.395196+0000","last_peered":"2024-03-21T14:03:42.395196+0000","last_clean":"2024-03-21T14:03:42.395196+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:03:42.395196+0000","last_undegraded":"2024-03-21T14:03:42.395196+0000","last_fullsized":"2024-03-21T14:03:42.395196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.021385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.021385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.898145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.833768+0000","last_change":"2024-03-21T14:03:56.833815+0000","last_active":"2024-03-21T14:03:56.833768+0000","last_peered":"2024-03-21T14:03:56.833768+0000","last_clean":"2024-03-21T14:03:56.833768+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:03:56.833768+0000","last_undegraded":"2024-03-21T14:03:56.833768+0000","last_fullsized":"2024-03-21T14:03:56.833768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.833681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.833681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:13.360093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734954+0000","last_change":"2024-03-21T14:02:35.773869+0000","last_active":"2024-03-21T14:03:42.734954+0000","last_peered":"2024-03-21T14:03:42.734954+0000","last_clean":"2024-03-21T14:03:42.734954+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:03:42.734954+0000","last_undegraded":"2024-03-21T14:03:42.734954+0000","last_fullsized":"2024-03-21T14:03:42.734954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.067670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995644+0000","last_change":"2024-03-21T14:03:38.925041+0000","last_active":"2024-03-21T14:03:42.995644+0000","last_peered":"2024-03-21T14:03:42.995644+0000","last_clean":"2024-03-21T14:03:42.995644+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:03:42.995644+0000","last_undegraded":"2024-03-21T14:03:42.995644+0000","last_fullsized":"2024-03-21T14:03:42.995644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.924957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.924957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.656539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394809+0000","last_change":"2024-03-21T14:03:41.007728+0000","last_active":"2024-03-21T14:03:42.394809+0000","last_peered":"2024-03-21T14:03:42.394809+0000","last_clean":"2024-03-21T14:03:42.394809+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:03:42.394809+0000","last_undegraded":"2024-03-21T14:03:42.394809+0000","last_fullsized":"2024-03-21T14:03:42.394809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.007598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.075924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999121+0000","last_change":"2024-03-21T14:02:33.442629+0000","last_active":"2024-03-21T14:03:42.999121+0000","last_peered":"2024-03-21T14:03:42.999121+0000","last_clean":"2024-03-21T14:03:42.999121+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:03:42.999121+0000","last_undegraded":"2024-03-21T14:03:42.999121+0000","last_fullsized":"2024-03-21T14:03:42.999121+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.962101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":79,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996413+0000","last_change":"2024-03-21T14:02:58.514359+0000","last_active":"2024-03-21T14:03:42.996413+0000","last_peered":"2024-03-21T14:03:42.996413+0000","last_clean":"2024-03-21T14:03:42.996413+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:03:42.996413+0000","last_undegraded":"2024-03-21T14:03:42.996413+0000","last_fullsized":"2024-03-21T14:03:42.996413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.552419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396180+0000","last_change":"2024-03-21T14:02:39.472519+0000","last_active":"2024-03-21T14:03:42.396180+0000","last_peered":"2024-03-21T14:03:42.396180+0000","last_clean":"2024-03-21T14:03:42.396180+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:03:42.396180+0000","last_undegraded":"2024-03-21T14:03:42.396180+0000","last_fullsized":"2024-03-21T14:03:42.396180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.779767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998816+0000","last_change":"2024-03-21T14:03:35.349742+0000","last_active":"2024-03-21T14:03:42.998816+0000","last_peered":"2024-03-21T14:03:42.998816+0000","last_clean":"2024-03-21T14:03:42.998816+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:03:42.998816+0000","last_undegraded":"2024-03-21T14:03:42.998816+0000","last_fullsized":"2024-03-21T14:03:42.998816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:03:35.349636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.349636+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.186069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389279+0000","last_change":"2024-03-21T14:02:33.441265+0000","last_active":"2024-03-21T14:03:42.389279+0000","last_peered":"2024-03-21T14:03:42.389279+0000","last_clean":"2024-03-21T14:03:42.389279+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:03:42.389279+0000","last_undegraded":"2024-03-21T14:03:42.389279+0000","last_fullsized":"2024-03-21T14:03:42.389279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.975558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":75,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501697+0000","last_change":"2024-03-21T14:03:48.369792+0000","last_active":"2024-03-21T14:03:53.501697+0000","last_peered":"2024-03-21T14:03:53.501697+0000","last_clean":"2024-03-21T14:03:53.501697+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:03:53.501697+0000","last_undegraded":"2024-03-21T14:03:53.501697+0000","last_fullsized":"2024-03-21T14:03:53.501697+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:03:48.369672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:48.369672+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.178152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996233+0000","last_change":"2024-03-21T14:02:37.456968+0000","last_active":"2024-03-21T14:03:42.996233+0000","last_peered":"2024-03-21T14:03:42.996233+0000","last_clean":"2024-03-21T14:03:42.996233+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:03:42.996233+0000","last_undegraded":"2024-03-21T14:03:42.996233+0000","last_fullsized":"2024-03-21T14:03:42.996233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.729850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.872553+0000","last_change":"2024-03-21T14:03:46.872577+0000","last_active":"2024-03-21T14:03:46.872553+0000","last_peered":"2024-03-21T14:03:46.872553+0000","last_clean":"2024-03-21T14:03:46.872553+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:03:46.872553+0000","last_undegraded":"2024-03-21T14:03:46.872553+0000","last_fullsized":"2024-03-21T14:03:46.872553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.872477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.872477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.050084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756711+0000","last_change":"2024-03-21T14:02:33.442413+0000","last_active":"2024-03-21T14:03:43.756711+0000","last_peered":"2024-03-21T14:03:43.756711+0000","last_clean":"2024-03-21T14:03:43.756711+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:03:43.756711+0000","last_undegraded":"2024-03-21T14:03:43.756711+0000","last_fullsized":"2024-03-21T14:03:43.756711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.621090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:50.047827+0000","last_change":"2024-03-21T14:03:50.047857+0000","last_active":"2024-03-21T14:03:50.047827+0000","last_peered":"2024-03-21T14:03:50.047827+0000","last_clean":"2024-03-21T14:03:50.047827+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:03:50.047827+0000","last_undegraded":"2024-03-21T14:03:50.047827+0000","last_fullsized":"2024-03-21T14:03:50.047827+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:50.047757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:50.047757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:57.758773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757698+0000","last_change":"2024-03-21T14:03:39.830694+0000","last_active":"2024-03-21T14:03:43.757698+0000","last_peered":"2024-03-21T14:03:43.757698+0000","last_clean":"2024-03-21T14:03:43.757698+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:03:43.757698+0000","last_undegraded":"2024-03-21T14:03:43.757698+0000","last_fullsized":"2024-03-21T14:03:43.757698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.830542+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.877216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:55.062292+0000","last_change":"2024-03-21T14:03:55.062330+0000","last_active":"2024-03-21T14:03:55.062292+0000","last_peered":"2024-03-21T14:03:55.062292+0000","last_clean":"2024-03-21T14:03:55.062292+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:03:55.062292+0000","last_undegraded":"2024-03-21T14:03:55.062292+0000","last_fullsized":"2024-03-21T14:03:55.062292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:55.062188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:55.062188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.285350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999465+0000","last_change":"2024-03-21T14:02:39.470869+0000","last_active":"2024-03-21T14:03:42.999465+0000","last_peered":"2024-03-21T14:03:42.999465+0000","last_clean":"2024-03-21T14:03:42.999465+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:03:42.999465+0000","last_undegraded":"2024-03-21T14:03:42.999465+0000","last_fullsized":"2024-03-21T14:03:42.999465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.994358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":312,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.817888+0000","last_change":"2024-03-21T14:03:54.817955+0000","last_active":"2024-03-21T14:03:54.817888+0000","last_peered":"2024-03-21T14:03:54.817888+0000","last_clean":"2024-03-21T14:03:54.817888+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:03:54.817888+0000","last_undegraded":"2024-03-21T14:03:54.817888+0000","last_fullsized":"2024-03-21T14:03:54.817888+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-03-21T14:03:54.817784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.817784+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.112690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395005+0000","last_change":"2024-03-21T14:02:33.444001+0000","last_active":"2024-03-21T14:03:42.395005+0000","last_peered":"2024-03-21T14:03:42.395005+0000","last_clean":"2024-03-21T14:03:42.395005+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:03:42.395005+0000","last_undegraded":"2024-03-21T14:03:42.395005+0000","last_fullsized":"2024-03-21T14:03:42.395005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:47.524470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501718+0000","last_change":"2024-03-21T14:02:35.774062+0000","last_active":"2024-03-21T14:03:53.501718+0000","last_peered":"2024-03-21T14:03:53.501718+0000","last_clean":"2024-03-21T14:03:53.501718+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:03:53.501718+0000","last_undegraded":"2024-03-21T14:03:53.501718+0000","last_fullsized":"2024-03-21T14:03:53.501718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533239+0000","last_change":"2024-03-21T14:02:37.459722+0000","last_active":"2024-03-21T14:03:43.533239+0000","last_peered":"2024-03-21T14:03:43.533239+0000","last_clean":"2024-03-21T14:03:43.533239+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:03:43.533239+0000","last_undegraded":"2024-03-21T14:03:43.533239+0000","last_fullsized":"2024-03-21T14:03:43.533239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.571697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756191+0000","last_change":"2024-03-21T14:02:39.471788+0000","last_active":"2024-03-21T14:03:43.756191+0000","last_peered":"2024-03-21T14:03:43.756191+0000","last_clean":"2024-03-21T14:03:43.756191+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:03:43.756191+0000","last_undegraded":"2024-03-21T14:03:43.756191+0000","last_fullsized":"2024-03-21T14:03:43.756191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.372670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995815+0000","last_change":"2024-03-21T14:02:33.447469+0000","last_active":"2024-03-21T14:03:42.995815+0000","last_peered":"2024-03-21T14:03:42.995815+0000","last_clean":"2024-03-21T14:03:42.995815+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:03:42.995815+0000","last_undegraded":"2024-03-21T14:03:42.995815+0000","last_fullsized":"2024-03-21T14:03:42.995815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.063604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":152,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.575497+0000","last_change":"2024-03-21T14:03:49.845488+0000","last_active":"2024-03-21T14:03:59.575497+0000","last_peered":"2024-03-21T14:03:59.575497+0000","last_clean":"2024-03-21T14:03:59.575497+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:03:59.575497+0000","last_undegraded":"2024-03-21T14:03:59.575497+0000","last_fullsized":"2024-03-21T14:03:59.575497+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:03:49.845376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:49.845376+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:12.461975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996292+0000","last_change":"2024-03-21T14:02:35.449792+0000","last_active":"2024-03-21T14:03:42.996292+0000","last_peered":"2024-03-21T14:03:42.996292+0000","last_clean":"2024-03-21T14:03:42.996292+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:03:42.996292+0000","last_undegraded":"2024-03-21T14:03:42.996292+0000","last_fullsized":"2024-03-21T14:03:42.996292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.553541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999655+0000","last_change":"2024-03-21T14:02:37.455752+0000","last_active":"2024-03-21T14:03:42.999655+0000","last_peered":"2024-03-21T14:03:42.999655+0000","last_clean":"2024-03-21T14:03:42.999655+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:03:42.999655+0000","last_undegraded":"2024-03-21T14:03:42.999655+0000","last_fullsized":"2024-03-21T14:03:42.999655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.531727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756153+0000","last_change":"2024-03-21T14:02:39.516952+0000","last_active":"2024-03-21T14:03:43.756153+0000","last_peered":"2024-03-21T14:03:43.756153+0000","last_clean":"2024-03-21T14:03:43.756153+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:03:43.756153+0000","last_undegraded":"2024-03-21T14:03:43.756153+0000","last_fullsized":"2024-03-21T14:03:43.756153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.885390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757538+0000","last_change":"2024-03-21T14:02:33.444253+0000","last_active":"2024-03-21T14:03:43.757538+0000","last_peered":"2024-03-21T14:03:43.757538+0000","last_clean":"2024-03-21T14:03:43.757538+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:03:43.757538+0000","last_undegraded":"2024-03-21T14:03:43.757538+0000","last_fullsized":"2024-03-21T14:03:43.757538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.260994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":91,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996230+0000","last_change":"2024-03-21T14:03:41.967449+0000","last_active":"2024-03-21T14:03:42.996230+0000","last_peered":"2024-03-21T14:03:42.996230+0000","last_clean":"2024-03-21T14:03:42.996230+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:03:42.996230+0000","last_undegraded":"2024-03-21T14:03:42.996230+0000","last_fullsized":"2024-03-21T14:03:42.996230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:03:41.967343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.967343+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.280117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734983+0000","last_change":"2024-03-21T14:02:37.454582+0000","last_active":"2024-03-21T14:03:42.734983+0000","last_peered":"2024-03-21T14:03:42.734983+0000","last_clean":"2024-03-21T14:03:42.734983+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:03:42.734983+0000","last_undegraded":"2024-03-21T14:03:42.734983+0000","last_fullsized":"2024-03-21T14:03:42.734983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.341912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995555+0000","last_change":"2024-03-21T14:02:39.473493+0000","last_active":"2024-03-21T14:03:42.995555+0000","last_peered":"2024-03-21T14:03:42.995555+0000","last_clean":"2024-03-21T14:03:42.995555+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:03:42.995555+0000","last_undegraded":"2024-03-21T14:03:42.995555+0000","last_fullsized":"2024-03-21T14:03:42.995555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.101298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999221+0000","last_change":"2024-03-21T14:02:33.443262+0000","last_active":"2024-03-21T14:03:42.999221+0000","last_peered":"2024-03-21T14:03:42.999221+0000","last_clean":"2024-03-21T14:03:42.999221+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:03:42.999221+0000","last_undegraded":"2024-03-21T14:03:42.999221+0000","last_fullsized":"2024-03-21T14:03:42.999221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.529943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757322+0000","last_change":"2024-03-21T14:03:38.812824+0000","last_active":"2024-03-21T14:03:43.757322+0000","last_peered":"2024-03-21T14:03:43.757322+0000","last_clean":"2024-03-21T14:03:43.757322+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:03:43.757322+0000","last_undegraded":"2024-03-21T14:03:43.757322+0000","last_fullsized":"2024-03-21T14:03:43.757322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:38.812732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.812732+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.836722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.007959+0000","last_change":"2024-03-21T14:03:56.007959+0000","last_active":"2024-03-21T14:03:56.007959+0000","last_peered":"2024-03-21T14:03:56.007959+0000","last_clean":"2024-03-21T14:03:56.007959+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:03:56.007959+0000","last_undegraded":"2024-03-21T14:03:56.007959+0000","last_fullsized":"2024-03-21T14:03:56.007959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:56.007835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.007835+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:18.104333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532331+0000","last_change":"2024-03-21T14:02:39.517270+0000","last_active":"2024-03-21T14:03:43.532331+0000","last_peered":"2024-03-21T14:03:43.532331+0000","last_clean":"2024-03-21T14:03:43.532331+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:03:43.532331+0000","last_undegraded":"2024-03-21T14:03:43.532331+0000","last_fullsized":"2024-03-21T14:03:43.532331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.528174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.935772+0000","last_change":"2024-03-21T14:03:43.935798+0000","last_active":"2024-03-21T14:03:43.935772+0000","last_peered":"2024-03-21T14:03:43.935772+0000","last_clean":"2024-03-21T14:03:43.935772+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:03:43.935772+0000","last_undegraded":"2024-03-21T14:03:43.935772+0000","last_fullsized":"2024-03-21T14:03:43.935772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:43.935711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.935711+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.752006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389164+0000","last_change":"2024-03-21T14:02:35.444623+0000","last_active":"2024-03-21T14:03:42.389164+0000","last_peered":"2024-03-21T14:03:42.389164+0000","last_clean":"2024-03-21T14:03:42.389164+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:03:42.389164+0000","last_undegraded":"2024-03-21T14:03:42.389164+0000","last_fullsized":"2024-03-21T14:03:42.389164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.184966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501399+0000","last_change":"2024-03-21T14:02:37.457785+0000","last_active":"2024-03-21T14:03:53.501399+0000","last_peered":"2024-03-21T14:03:53.501399+0000","last_clean":"2024-03-21T14:03:53.501399+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:03:53.501399+0000","last_undegraded":"2024-03-21T14:03:53.501399+0000","last_fullsized":"2024-03-21T14:03:53.501399+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389071+0000","last_change":"2024-03-21T14:02:39.469558+0000","last_active":"2024-03-21T14:03:42.389071+0000","last_peered":"2024-03-21T14:03:42.389071+0000","last_clean":"2024-03-21T14:03:42.389071+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:03:42.389071+0000","last_undegraded":"2024-03-21T14:03:42.389071+0000","last_fullsized":"2024-03-21T14:03:42.389071+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:46.811931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395781+0000","last_change":"2024-03-21T14:03:36.591924+0000","last_active":"2024-03-21T14:03:42.395781+0000","last_peered":"2024-03-21T14:03:42.395781+0000","last_clean":"2024-03-21T14:03:42.395781+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:03:42.395781+0000","last_undegraded":"2024-03-21T14:03:42.395781+0000","last_fullsized":"2024-03-21T14:03:42.395781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:03:36.591844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.332405+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:50.545354+0000","last_change":"2024-03-21T14:03:50.545394+0000","last_active":"2024-03-21T14:03:50.545354+0000","last_peered":"2024-03-21T14:03:50.545354+0000","last_clean":"2024-03-21T14:03:50.545354+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:03:50.545354+0000","last_undegraded":"2024-03-21T14:03:50.545354+0000","last_fullsized":"2024-03-21T14:03:50.545354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:03:50.545283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:50.545283+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.508308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388989+0000","last_change":"2024-03-21T14:02:37.456640+0000","last_active":"2024-03-21T14:03:42.388989+0000","last_peered":"2024-03-21T14:03:42.388989+0000","last_clean":"2024-03-21T14:03:42.388989+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:03:42.388989+0000","last_undegraded":"2024-03-21T14:03:42.388989+0000","last_fullsized":"2024-03-21T14:03:42.388989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.154776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.381560+0000","last_change":"2024-03-21T14:03:54.381586+0000","last_active":"2024-03-21T14:03:54.381560+0000","last_peered":"2024-03-21T14:03:54.381560+0000","last_clean":"2024-03-21T14:03:54.381560+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:03:54.381560+0000","last_undegraded":"2024-03-21T14:03:54.381560+0000","last_fullsized":"2024-03-21T14:03:54.381560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.381485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:12.874634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.501716+0000","last_change":"2024-03-21T14:03:43.001482+0000","last_active":"2024-03-21T14:03:53.501716+0000","last_peered":"2024-03-21T14:03:53.501716+0000","last_clean":"2024-03-21T14:03:53.501716+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:03:53.501716+0000","last_undegraded":"2024-03-21T14:03:53.501716+0000","last_fullsized":"2024-03-21T14:03:53.501716+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:43.001363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.001363+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.127322+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.674672+0000","last_change":"2024-03-21T14:03:56.674692+0000","last_active":"2024-03-21T14:03:56.674672+0000","last_peered":"2024-03-21T14:03:56.674672+0000","last_clean":"2024-03-21T14:03:56.674672+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:03:56.674672+0000","last_undegraded":"2024-03-21T14:03:56.674672+0000","last_fullsized":"2024-03-21T14:03:56.674672+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.674610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.674610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:25.472031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.758265+0000","last_change":"2024-03-21T14:03:53.758307+0000","last_active":"2024-03-21T14:03:53.758265+0000","last_peered":"2024-03-21T14:03:53.758265+0000","last_clean":"2024-03-21T14:03:53.758265+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:03:53.758265+0000","last_undegraded":"2024-03-21T14:03:53.758265+0000","last_fullsized":"2024-03-21T14:03:53.758265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:03:53.758186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.758186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:22.866515+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395610+0000","last_change":"2024-03-21T14:02:39.516061+0000","last_active":"2024-03-21T14:03:42.395610+0000","last_peered":"2024-03-21T14:03:42.395610+0000","last_clean":"2024-03-21T14:03:42.395610+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:03:42.395610+0000","last_undegraded":"2024-03-21T14:03:42.395610+0000","last_fullsized":"2024-03-21T14:03:42.395610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.869674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999626+0000","last_change":"2024-03-21T14:02:37.454216+0000","last_active":"2024-03-21T14:03:42.999626+0000","last_peered":"2024-03-21T14:03:42.999626+0000","last_clean":"2024-03-21T14:03:42.999626+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:03:42.999626+0000","last_undegraded":"2024-03-21T14:03:42.999626+0000","last_fullsized":"2024-03-21T14:03:42.999626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.938463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389026+0000","last_change":"2024-03-21T14:03:34.139285+0000","last_active":"2024-03-21T14:03:42.389026+0000","last_peered":"2024-03-21T14:03:42.389026+0000","last_clean":"2024-03-21T14:03:42.389026+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:03:42.389026+0000","last_undegraded":"2024-03-21T14:03:42.389026+0000","last_fullsized":"2024-03-21T14:03:42.389026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:34.139127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:34.139127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.352287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389671+0000","last_change":"2024-03-21T14:02:35.450014+0000","last_active":"2024-03-21T14:03:42.389671+0000","last_peered":"2024-03-21T14:03:42.389671+0000","last_clean":"2024-03-21T14:03:42.389671+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:03:42.389671+0000","last_undegraded":"2024-03-21T14:03:42.389671+0000","last_fullsized":"2024-03-21T14:03:42.389671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.855239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396108+0000","last_change":"2024-03-21T14:02:39.468574+0000","last_active":"2024-03-21T14:03:42.396108+0000","last_peered":"2024-03-21T14:03:42.396108+0000","last_clean":"2024-03-21T14:03:42.396108+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:03:42.396108+0000","last_undegraded":"2024-03-21T14:03:42.396108+0000","last_fullsized":"2024-03-21T14:03:42.396108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.471894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757692+0000","last_change":"2024-03-21T14:02:37.456947+0000","last_active":"2024-03-21T14:03:43.757692+0000","last_peered":"2024-03-21T14:03:43.757692+0000","last_clean":"2024-03-21T14:03:43.757692+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:03:43.757692+0000","last_undegraded":"2024-03-21T14:03:43.757692+0000","last_fullsized":"2024-03-21T14:03:43.757692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.581511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735065+0000","last_change":"2024-03-21T14:02:33.442462+0000","last_active":"2024-03-21T14:03:42.735065+0000","last_peered":"2024-03-21T14:03:42.735065+0000","last_clean":"2024-03-21T14:03:42.735065+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:03:42.735065+0000","last_undegraded":"2024-03-21T14:03:42.735065+0000","last_fullsized":"2024-03-21T14:03:42.735065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.441774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":83,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:51.976591+0000","last_change":"2024-03-21T14:03:51.976637+0000","last_active":"2024-03-21T14:03:51.976591+0000","last_peered":"2024-03-21T14:03:51.976591+0000","last_clean":"2024-03-21T14:03:51.976591+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:03:51.976591+0000","last_undegraded":"2024-03-21T14:03:51.976591+0000","last_fullsized":"2024-03-21T14:03:51.976591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:03:51.976515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:51.976515+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:16.722963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389971+0000","last_change":"2024-03-21T14:02:39.459328+0000","last_active":"2024-03-21T14:03:42.389971+0000","last_peered":"2024-03-21T14:03:42.389971+0000","last_clean":"2024-03-21T14:03:42.389971+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:03:42.389971+0000","last_undegraded":"2024-03-21T14:03:42.389971+0000","last_fullsized":"2024-03-21T14:03:42.389971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.833817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":77,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:57.040828+0000","last_change":"2024-03-21T14:03:57.040828+0000","last_active":"2024-03-21T14:03:57.040828+0000","last_peered":"2024-03-21T14:03:57.040828+0000","last_clean":"2024-03-21T14:03:57.040828+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:03:57.040828+0000","last_undegraded":"2024-03-21T14:03:57.040828+0000","last_fullsized":"2024-03-21T14:03:57.040828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:57.040741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:57.040741+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:26.294593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395341+0000","last_change":"2024-03-21T14:02:33.442529+0000","last_active":"2024-03-21T14:03:42.395341+0000","last_peered":"2024-03-21T14:03:42.395341+0000","last_clean":"2024-03-21T14:03:42.395341+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:03:42.395341+0000","last_undegraded":"2024-03-21T14:03:42.395341+0000","last_fullsized":"2024-03-21T14:03:42.395341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.804413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389402+0000","last_change":"2024-03-21T14:02:35.450511+0000","last_active":"2024-03-21T14:03:42.389402+0000","last_peered":"2024-03-21T14:03:42.389402+0000","last_clean":"2024-03-21T14:03:42.389402+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:03:42.389402+0000","last_undegraded":"2024-03-21T14:03:42.389402+0000","last_fullsized":"2024-03-21T14:03:42.389402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.494861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103190,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":927293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430235,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28328,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716824,"statfs":{"total":95995035648,"available":95966027776,"internally_reserved":0,"allocated":1343488,"data_stored":925330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":337516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":926124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":757760,"data_stored":335310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":794624,"data_stored":336302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018918,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":786432,"data_stored":336016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":336557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:04:05.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:04:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:05 smithi118 ceph-mon[33862]: 6.2 deep-scrub starts 2024-03-21T14:04:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:05 smithi118 ceph-mon[33862]: 6.2 deep-scrub ok 2024-03-21T14:04:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:05 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:04:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:05 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:04:06.075 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:06.665 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:04:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:04:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:04:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 3.0 deep-scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 3.0 deep-scrub ok 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:04:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 3.0 deep-scrub starts 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 3.0 deep-scrub ok 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:04:06.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:04:06.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:04:06.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:04:06.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:04:06.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:04:06.757 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:06 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:04:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 3.0 deep-scrub starts 2024-03-21T14:04:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 3.0 deep-scrub ok 2024-03-21T14:04:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:04:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:04:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:04:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:04:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:06 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:04:07.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:04:07.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:04:07.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:07.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:04:07.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:04:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:04:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:04:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:04:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:07 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:04:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:07 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:04:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:07 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:04:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:07 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:04:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:07 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:04:08.133 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:04:08.135 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:04:08.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[29576]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[29576]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[40510]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:08.850 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:08 smithi003 ceph-mon[40510]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T14:04:08.851 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-03-21T14:04:07.672046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1020,"num_read_kb":866,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":615,"ondisk_log_size":615,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":253052,"kb_used_data":7804,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749708164,"statfs":{"total":767960285184,"available":767701159936,"internally_reserved":0,"allocated":7991296,"data_stored":4460448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.421298"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389060+0000","last_change":"2024-03-21T14:02:39.463103+0000","last_active":"2024-03-21T14:03:42.389060+0000","last_peered":"2024-03-21T14:03:42.389060+0000","last_clean":"2024-03-21T14:03:42.389060+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:03:42.389060+0000","last_undegraded":"2024-03-21T14:03:42.389060+0000","last_fullsized":"2024-03-21T14:03:42.389060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.364725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.788738+0000","last_change":"2024-03-21T14:03:59.788760+0000","last_active":"2024-03-21T14:03:59.788738+0000","last_peered":"2024-03-21T14:03:59.788738+0000","last_clean":"2024-03-21T14:03:59.788738+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:03:59.788738+0000","last_undegraded":"2024-03-21T14:03:59.788738+0000","last_fullsized":"2024-03-21T14:03:59.788738+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:59.788675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:59.788675+0000","last_clean_scrub_stamp":"2024-03-21T14:03:59.788675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:03.612884+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.957706+0000","last_change":"2024-03-21T14:03:46.957726+0000","last_active":"2024-03-21T14:03:46.957706+0000","last_peered":"2024-03-21T14:03:46.957706+0000","last_clean":"2024-03-21T14:03:46.957706+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:03:46.957706+0000","last_undegraded":"2024-03-21T14:03:46.957706+0000","last_fullsized":"2024-03-21T14:03:46.957706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.957648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.957648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.716204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388929+0000","last_change":"2024-03-21T14:02:35.450207+0000","last_active":"2024-03-21T14:03:42.388929+0000","last_peered":"2024-03-21T14:03:42.388929+0000","last_clean":"2024-03-21T14:03:42.388929+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:03:42.388929+0000","last_undegraded":"2024-03-21T14:03:42.388929+0000","last_fullsized":"2024-03-21T14:03:42.388929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.595411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395241+0000","last_change":"2024-03-21T14:02:39.517587+0000","last_active":"2024-03-21T14:03:42.395241+0000","last_peered":"2024-03-21T14:03:42.395241+0000","last_clean":"2024-03-21T14:03:42.395241+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:03:42.395241+0000","last_undegraded":"2024-03-21T14:03:42.395241+0000","last_fullsized":"2024-03-21T14:03:42.395241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.320905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.967003+0000","last_change":"2024-03-21T14:03:53.967027+0000","last_active":"2024-03-21T14:03:53.967003+0000","last_peered":"2024-03-21T14:03:53.967003+0000","last_clean":"2024-03-21T14:03:53.967003+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:03:53.967003+0000","last_undegraded":"2024-03-21T14:03:53.967003+0000","last_fullsized":"2024-03-21T14:03:53.967003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.966920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.966920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.338422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396378+0000","last_change":"2024-03-21T14:03:37.567249+0000","last_active":"2024-03-21T14:03:42.396378+0000","last_peered":"2024-03-21T14:03:42.396378+0000","last_clean":"2024-03-21T14:03:42.396378+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:03:42.396378+0000","last_undegraded":"2024-03-21T14:03:42.396378+0000","last_fullsized":"2024-03-21T14:03:42.396378+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:37.567167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.567167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.628456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394378+0000","last_change":"2024-03-21T14:03:35.571620+0000","last_active":"2024-03-21T14:03:42.394378+0000","last_peered":"2024-03-21T14:03:42.394378+0000","last_clean":"2024-03-21T14:03:42.394378+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:03:42.394378+0000","last_undegraded":"2024-03-21T14:03:42.394378+0000","last_fullsized":"2024-03-21T14:03:42.394378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.571531+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:41.701221+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995806+0000","last_change":"2024-03-21T14:02:39.469561+0000","last_active":"2024-03-21T14:03:42.995806+0000","last_peered":"2024-03-21T14:03:42.995806+0000","last_clean":"2024-03-21T14:03:42.995806+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:03:42.995806+0000","last_undegraded":"2024-03-21T14:03:42.995806+0000","last_fullsized":"2024-03-21T14:03:42.995806+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.240000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999035+0000","last_change":"2024-03-21T14:02:37.455588+0000","last_active":"2024-03-21T14:03:42.999035+0000","last_peered":"2024-03-21T14:03:42.999035+0000","last_clean":"2024-03-21T14:03:42.999035+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:03:42.999035+0000","last_undegraded":"2024-03-21T14:03:42.999035+0000","last_fullsized":"2024-03-21T14:03:42.999035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.535187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:05.516432+0000","last_change":"2024-03-21T14:04:05.516451+0000","last_active":"2024-03-21T14:04:05.516432+0000","last_peered":"2024-03-21T14:04:05.516432+0000","last_clean":"2024-03-21T14:04:05.516432+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:04:05.516432+0000","last_undegraded":"2024-03-21T14:04:05.516432+0000","last_fullsized":"2024-03-21T14:04:05.516432+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:05.516375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:05.516375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:20.787885+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389241+0000","last_change":"2024-03-21T14:02:35.445770+0000","last_active":"2024-03-21T14:03:42.389241+0000","last_peered":"2024-03-21T14:03:42.389241+0000","last_clean":"2024-03-21T14:03:42.389241+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:03:42.389241+0000","last_undegraded":"2024-03-21T14:03:42.389241+0000","last_fullsized":"2024-03-21T14:03:42.389241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.203589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.410037+0000","last_change":"2024-03-21T14:03:56.410063+0000","last_active":"2024-03-21T14:03:56.410037+0000","last_peered":"2024-03-21T14:03:56.410037+0000","last_clean":"2024-03-21T14:03:56.410037+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:03:56.410037+0000","last_undegraded":"2024-03-21T14:03:56.410037+0000","last_fullsized":"2024-03-21T14:03:56.410037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.409946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.409946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:17.565720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532890+0000","last_change":"2024-03-21T14:02:37.457212+0000","last_active":"2024-03-21T14:03:43.532890+0000","last_peered":"2024-03-21T14:03:43.532890+0000","last_clean":"2024-03-21T14:03:43.532890+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:03:43.532890+0000","last_undegraded":"2024-03-21T14:03:43.532890+0000","last_fullsized":"2024-03-21T14:03:43.532890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.210558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.357086+0000","last_change":"2024-03-21T14:03:46.357111+0000","last_active":"2024-03-21T14:03:46.357086+0000","last_peered":"2024-03-21T14:03:46.357086+0000","last_clean":"2024-03-21T14:03:46.357086+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:03:46.357086+0000","last_undegraded":"2024-03-21T14:03:46.357086+0000","last_fullsized":"2024-03-21T14:03:46.357086+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.357008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.357008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.267704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":57,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389583+0000","last_change":"2024-03-21T14:02:35.453933+0000","last_active":"2024-03-21T14:03:42.389583+0000","last_peered":"2024-03-21T14:03:42.389583+0000","last_clean":"2024-03-21T14:03:42.389583+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:03:42.389583+0000","last_undegraded":"2024-03-21T14:03:42.389583+0000","last_fullsized":"2024-03-21T14:03:42.389583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.127706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389793+0000","last_change":"2024-03-21T14:02:39.516305+0000","last_active":"2024-03-21T14:03:42.389793+0000","last_peered":"2024-03-21T14:03:42.389793+0000","last_clean":"2024-03-21T14:03:42.389793+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:03:42.389793+0000","last_undegraded":"2024-03-21T14:03:42.389793+0000","last_fullsized":"2024-03-21T14:03:42.389793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.252625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757380+0000","last_change":"2024-03-21T14:02:37.459191+0000","last_active":"2024-03-21T14:03:43.757380+0000","last_peered":"2024-03-21T14:03:43.757380+0000","last_clean":"2024-03-21T14:03:43.757380+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:03:43.757380+0000","last_undegraded":"2024-03-21T14:03:43.757380+0000","last_fullsized":"2024-03-21T14:03:43.757380+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.913816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.004769+0000","last_change":"2024-03-21T14:03:53.004791+0000","last_active":"2024-03-21T14:03:53.004769+0000","last_peered":"2024-03-21T14:03:53.004769+0000","last_clean":"2024-03-21T14:03:53.004769+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:03:53.004769+0000","last_undegraded":"2024-03-21T14:03:53.004769+0000","last_fullsized":"2024-03-21T14:03:53.004769+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.004692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.004692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.564997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:48.070675+0000","last_change":"2024-03-21T14:03:48.070734+0000","last_active":"2024-03-21T14:03:48.070675+0000","last_peered":"2024-03-21T14:03:48.070675+0000","last_clean":"2024-03-21T14:03:48.070675+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:03:48.070675+0000","last_undegraded":"2024-03-21T14:03:48.070675+0000","last_fullsized":"2024-03-21T14:03:48.070675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:03:48.070599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:48.070599+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.334848+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395547+0000","last_change":"2024-03-21T14:02:39.517386+0000","last_active":"2024-03-21T14:03:42.395547+0000","last_peered":"2024-03-21T14:03:42.395547+0000","last_clean":"2024-03-21T14:03:42.395547+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:03:42.395547+0000","last_undegraded":"2024-03-21T14:03:42.395547+0000","last_fullsized":"2024-03-21T14:03:42.395547+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.220059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389360+0000","last_change":"2024-03-21T14:02:37.458298+0000","last_active":"2024-03-21T14:03:42.389360+0000","last_peered":"2024-03-21T14:03:42.389360+0000","last_clean":"2024-03-21T14:03:42.389360+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:03:42.389360+0000","last_undegraded":"2024-03-21T14:03:42.389360+0000","last_fullsized":"2024-03-21T14:03:42.389360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.474405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.949455+0000","last_change":"2024-03-21T14:03:47.949482+0000","last_active":"2024-03-21T14:03:47.949455+0000","last_peered":"2024-03-21T14:03:47.949455+0000","last_clean":"2024-03-21T14:03:47.949455+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:03:47.949455+0000","last_undegraded":"2024-03-21T14:03:47.949455+0000","last_fullsized":"2024-03-21T14:03:47.949455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.949388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.443300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395701+0000","last_change":"2024-03-21T14:02:35.451688+0000","last_active":"2024-03-21T14:03:42.395701+0000","last_peered":"2024-03-21T14:03:42.395701+0000","last_clean":"2024-03-21T14:03:42.395701+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:03:42.395701+0000","last_undegraded":"2024-03-21T14:03:42.395701+0000","last_fullsized":"2024-03-21T14:03:42.395701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:50.784537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:52.884156+0000","last_change":"2024-03-21T14:03:52.884209+0000","last_active":"2024-03-21T14:03:52.884156+0000","last_peered":"2024-03-21T14:03:52.884156+0000","last_clean":"2024-03-21T14:03:52.884156+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:03:52.884156+0000","last_undegraded":"2024-03-21T14:03:52.884156+0000","last_fullsized":"2024-03-21T14:03:52.884156+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:03:52.884054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:52.884054+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:00.091462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":84,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:02.112670+0000","last_change":"2024-03-21T14:04:02.112670+0000","last_active":"2024-03-21T14:04:02.112670+0000","last_peered":"2024-03-21T14:04:02.112670+0000","last_clean":"2024-03-21T14:04:02.112670+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:04:02.112670+0000","last_undegraded":"2024-03-21T14:04:02.112670+0000","last_fullsized":"2024-03-21T14:04:02.112670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:04:02.112580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:02.112580+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.796615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.710291+0000","last_change":"2024-03-21T14:03:54.710330+0000","last_active":"2024-03-21T14:03:54.710291+0000","last_peered":"2024-03-21T14:03:54.710291+0000","last_clean":"2024-03-21T14:03:54.710291+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:03:54.710291+0000","last_undegraded":"2024-03-21T14:03:54.710291+0000","last_fullsized":"2024-03-21T14:03:54.710291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.710220+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.710102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:44.904151+0000","last_change":"2024-03-21T14:03:44.904204+0000","last_active":"2024-03-21T14:03:44.904151+0000","last_peered":"2024-03-21T14:03:44.904151+0000","last_clean":"2024-03-21T14:03:44.904151+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:03:44.904151+0000","last_undegraded":"2024-03-21T14:03:44.904151+0000","last_fullsized":"2024-03-21T14:03:44.904151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:44.904071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:44.904071+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.807993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:02.803387+0000","last_change":"2024-03-21T14:04:02.803426+0000","last_active":"2024-03-21T14:04:02.803387+0000","last_peered":"2024-03-21T14:04:02.803387+0000","last_clean":"2024-03-21T14:04:02.803387+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:04:02.803387+0000","last_undegraded":"2024-03-21T14:04:02.803387+0000","last_fullsized":"2024-03-21T14:04:02.803387+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:02.803297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:02.803297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:26.361453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996574+0000","last_change":"2024-03-21T14:02:37.463347+0000","last_active":"2024-03-21T14:03:42.996574+0000","last_peered":"2024-03-21T14:03:42.996574+0000","last_clean":"2024-03-21T14:03:42.996574+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:03:42.996574+0000","last_undegraded":"2024-03-21T14:03:42.996574+0000","last_fullsized":"2024-03-21T14:03:42.996574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.394323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394109+0000","last_change":"2024-03-21T14:03:39.522391+0000","last_active":"2024-03-21T14:03:42.394109+0000","last_peered":"2024-03-21T14:03:42.394109+0000","last_clean":"2024-03-21T14:03:42.394109+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:03:42.394109+0000","last_undegraded":"2024-03-21T14:03:42.394109+0000","last_fullsized":"2024-03-21T14:03:42.394109+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.522309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.085948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.279816+0000","last_change":"2024-03-21T14:03:53.279859+0000","last_active":"2024-03-21T14:03:53.279816+0000","last_peered":"2024-03-21T14:03:53.279816+0000","last_clean":"2024-03-21T14:03:53.279816+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:03:53.279816+0000","last_undegraded":"2024-03-21T14:03:53.279816+0000","last_fullsized":"2024-03-21T14:03:53.279816+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:03:53.279738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.279738+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:57.232403+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.862928+0000","last_change":"2024-03-21T14:03:59.862965+0000","last_active":"2024-03-21T14:03:59.862928+0000","last_peered":"2024-03-21T14:03:59.862928+0000","last_clean":"2024-03-21T14:03:59.862928+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:03:59.862928+0000","last_undegraded":"2024-03-21T14:03:59.862928+0000","last_fullsized":"2024-03-21T14:03:59.862928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:59.862840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:59.862840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.546256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:04.083775+0000","last_change":"2024-03-21T14:04:04.083812+0000","last_active":"2024-03-21T14:04:04.083775+0000","last_peered":"2024-03-21T14:04:04.083775+0000","last_clean":"2024-03-21T14:04:04.083775+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:04:04.083775+0000","last_undegraded":"2024-03-21T14:04:04.083775+0000","last_fullsized":"2024-03-21T14:04:04.083775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:04.083690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:04.083690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:33.768096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.916303+0000","last_change":"2024-03-21T14:03:47.916324+0000","last_active":"2024-03-21T14:03:47.916303+0000","last_peered":"2024-03-21T14:03:47.916303+0000","last_clean":"2024-03-21T14:03:47.916303+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:03:47.916303+0000","last_undegraded":"2024-03-21T14:03:47.916303+0000","last_fullsized":"2024-03-21T14:03:47.916303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.916245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.916245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.354491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":61,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.509555+0000","last_change":"2024-03-21T14:03:45.509584+0000","last_active":"2024-03-21T14:03:45.509555+0000","last_peered":"2024-03-21T14:03:45.509555+0000","last_clean":"2024-03-21T14:03:45.509555+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:03:45.509555+0000","last_undegraded":"2024-03-21T14:03:45.509555+0000","last_fullsized":"2024-03-21T14:03:45.509555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:03:45.509492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.509492+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.086854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:05.667068+0000","last_change":"2024-03-21T14:04:05.667110+0000","last_active":"2024-03-21T14:04:05.667068+0000","last_peered":"2024-03-21T14:04:05.667068+0000","last_clean":"2024-03-21T14:04:05.667068+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:04:05.667068+0000","last_undegraded":"2024-03-21T14:04:05.667068+0000","last_fullsized":"2024-03-21T14:04:05.667068+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:05.666988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:05.666988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.208155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:47.836823+0000","last_change":"2024-03-21T14:03:47.836847+0000","last_active":"2024-03-21T14:03:47.836823+0000","last_peered":"2024-03-21T14:03:47.836823+0000","last_clean":"2024-03-21T14:03:47.836823+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:03:47.836823+0000","last_undegraded":"2024-03-21T14:03:47.836823+0000","last_fullsized":"2024-03-21T14:03:47.836823+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:47.836751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.836751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:07.563311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:03:41.362147+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:41.362005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.362005+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:10.804533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.598777+0000","last_change":"2024-03-21T14:03:54.598813+0000","last_active":"2024-03-21T14:03:54.598777+0000","last_peered":"2024-03-21T14:03:54.598777+0000","last_clean":"2024-03-21T14:03:54.598777+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:03:54.598777+0000","last_undegraded":"2024-03-21T14:03:54.598777+0000","last_fullsized":"2024-03-21T14:03:54.598777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:03:54.598701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.598701+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:59.201207+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995540+0000","last_change":"2024-03-21T14:02:39.471220+0000","last_active":"2024-03-21T14:03:42.995540+0000","last_peered":"2024-03-21T14:03:42.995540+0000","last_clean":"2024-03-21T14:03:42.995540+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:03:42.995540+0000","last_undegraded":"2024-03-21T14:03:42.995540+0000","last_fullsized":"2024-03-21T14:03:42.995540+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.470185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.547235+0000","last_change":"2024-03-21T14:03:53.547267+0000","last_active":"2024-03-21T14:03:53.547235+0000","last_peered":"2024-03-21T14:03:53.547235+0000","last_clean":"2024-03-21T14:03:53.547235+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:03:53.547235+0000","last_undegraded":"2024-03-21T14:03:53.547235+0000","last_fullsized":"2024-03-21T14:03:53.547235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.547160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.547160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:19.476340+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:03.822231+0000","last_change":"2024-03-21T14:04:03.822268+0000","last_active":"2024-03-21T14:04:03.822231+0000","last_peered":"2024-03-21T14:04:03.822231+0000","last_clean":"2024-03-21T14:04:03.822231+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:04:03.822231+0000","last_undegraded":"2024-03-21T14:04:03.822231+0000","last_fullsized":"2024-03-21T14:04:03.822231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:03.822149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:03.822149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:31.198486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":71,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.759191+0000","last_change":"2024-03-21T14:03:43.759222+0000","last_active":"2024-03-21T14:03:43.759191+0000","last_peered":"2024-03-21T14:03:43.759191+0000","last_clean":"2024-03-21T14:03:43.759191+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:03:43.759191+0000","last_undegraded":"2024-03-21T14:03:43.759191+0000","last_fullsized":"2024-03-21T14:03:43.759191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:03:43.759108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.759108+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.080054+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394090+0000","last_change":"2024-03-21T14:02:39.471886+0000","last_active":"2024-03-21T14:03:42.394090+0000","last_peered":"2024-03-21T14:03:42.394090+0000","last_clean":"2024-03-21T14:03:42.394090+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:03:42.394090+0000","last_undegraded":"2024-03-21T14:03:42.394090+0000","last_fullsized":"2024-03-21T14:03:42.394090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.715774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757272+0000","last_change":"2024-03-21T14:02:37.455324+0000","last_active":"2024-03-21T14:03:43.757272+0000","last_peered":"2024-03-21T14:03:43.757272+0000","last_clean":"2024-03-21T14:03:43.757272+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:03:43.757272+0000","last_undegraded":"2024-03-21T14:03:43.757272+0000","last_fullsized":"2024-03-21T14:03:43.757272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.837893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.994403+0000","last_change":"2024-03-21T14:03:46.994427+0000","last_active":"2024-03-21T14:03:46.994403+0000","last_peered":"2024-03-21T14:03:46.994403+0000","last_clean":"2024-03-21T14:03:46.994403+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:03:46.994403+0000","last_undegraded":"2024-03-21T14:03:46.994403+0000","last_fullsized":"2024-03-21T14:03:46.994403+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.994338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.994338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.043818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":92,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:04.989858+0000","last_change":"2024-03-21T14:04:04.989918+0000","last_active":"2024-03-21T14:04:04.989858+0000","last_peered":"2024-03-21T14:04:04.989858+0000","last_clean":"2024-03-21T14:04:04.989858+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:04:04.989858+0000","last_undegraded":"2024-03-21T14:04:04.989858+0000","last_fullsized":"2024-03-21T14:04:04.989858+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:04:04.989785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:04:04.989785+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:14.386024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.031159+0000","last_change":"2024-03-21T14:03:58.031180+0000","last_active":"2024-03-21T14:03:58.031159+0000","last_peered":"2024-03-21T14:03:58.031159+0000","last_clean":"2024-03-21T14:03:58.031159+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:03:58.031159+0000","last_undegraded":"2024-03-21T14:03:58.031159+0000","last_fullsized":"2024-03-21T14:03:58.031159+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:58.031098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:58.031098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:13.713947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.503057+0000","last_change":"2024-03-21T14:03:37.352147+0000","last_active":"2024-03-21T14:03:58.503057+0000","last_peered":"2024-03-21T14:03:58.503057+0000","last_clean":"2024-03-21T14:03:58.503057+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:03:58.503057+0000","last_undegraded":"2024-03-21T14:03:58.503057+0000","last_fullsized":"2024-03-21T14:03:58.503057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:37.352012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:37.352012+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.843950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533005+0000","last_change":"2024-03-21T14:02:33.446863+0000","last_active":"2024-03-21T14:03:43.533005+0000","last_peered":"2024-03-21T14:03:43.533005+0000","last_clean":"2024-03-21T14:03:43.533005+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:03:43.533005+0000","last_undegraded":"2024-03-21T14:03:43.533005+0000","last_fullsized":"2024-03-21T14:03:43.533005+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.983498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395982+0000","last_change":"2024-03-21T14:02:35.775119+0000","last_active":"2024-03-21T14:03:42.395982+0000","last_peered":"2024-03-21T14:03:42.395982+0000","last_clean":"2024-03-21T14:03:42.395982+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:03:42.395982+0000","last_undegraded":"2024-03-21T14:03:42.395982+0000","last_fullsized":"2024-03-21T14:03:42.395982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.079104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.390025+0000","last_change":"2024-03-21T14:02:39.467603+0000","last_active":"2024-03-21T14:03:42.390025+0000","last_peered":"2024-03-21T14:03:42.390025+0000","last_clean":"2024-03-21T14:03:42.390025+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:03:42.390025+0000","last_undegraded":"2024-03-21T14:03:42.390025+0000","last_fullsized":"2024-03-21T14:03:42.390025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.883243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.502755+0000","last_change":"2024-03-21T14:03:47.500013+0000","last_active":"2024-03-21T14:03:58.502755+0000","last_peered":"2024-03-21T14:03:58.502755+0000","last_clean":"2024-03-21T14:03:58.502755+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:03:58.502755+0000","last_undegraded":"2024-03-21T14:03:58.502755+0000","last_fullsized":"2024-03-21T14:03:58.502755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:47.499917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:47.499917+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:09.440649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:06.084251+0000","last_change":"2024-03-21T14:04:06.084273+0000","last_active":"2024-03-21T14:04:06.084251+0000","last_peered":"2024-03-21T14:04:06.084251+0000","last_clean":"2024-03-21T14:04:06.084251+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:04:06.084251+0000","last_undegraded":"2024-03-21T14:04:06.084251+0000","last_fullsized":"2024-03-21T14:04:06.084251+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:06.084189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:06.084189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:33.196569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999402+0000","last_change":"2024-03-21T14:02:35.775497+0000","last_active":"2024-03-21T14:03:42.999402+0000","last_peered":"2024-03-21T14:03:42.999402+0000","last_clean":"2024-03-21T14:03:42.999402+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:03:42.999402+0000","last_undegraded":"2024-03-21T14:03:42.999402+0000","last_fullsized":"2024-03-21T14:03:42.999402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:03.620820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:01.013228+0000","last_change":"2024-03-21T14:04:01.013250+0000","last_active":"2024-03-21T14:04:01.013228+0000","last_peered":"2024-03-21T14:04:01.013228+0000","last_clean":"2024-03-21T14:04:01.013228+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:04:01.013228+0000","last_undegraded":"2024-03-21T14:04:01.013228+0000","last_fullsized":"2024-03-21T14:04:01.013228+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:01.013167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:01.013167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:25.253471+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396096+0000","last_change":"2024-03-21T14:02:37.456859+0000","last_active":"2024-03-21T14:03:42.396096+0000","last_peered":"2024-03-21T14:03:42.396096+0000","last_clean":"2024-03-21T14:03:42.396096+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:03:42.396096+0000","last_undegraded":"2024-03-21T14:03:42.396096+0000","last_fullsized":"2024-03-21T14:03:42.396096+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.881127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532903+0000","last_change":"2024-03-21T14:03:38.848893+0000","last_active":"2024-03-21T14:03:43.532903+0000","last_peered":"2024-03-21T14:03:43.532903+0000","last_clean":"2024-03-21T14:03:43.532903+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:03:43.532903+0000","last_undegraded":"2024-03-21T14:03:43.532903+0000","last_fullsized":"2024-03-21T14:03:43.532903+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.848793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.848793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.951970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996491+0000","last_change":"2024-03-21T14:02:35.774922+0000","last_active":"2024-03-21T14:03:42.996491+0000","last_peered":"2024-03-21T14:03:42.996491+0000","last_clean":"2024-03-21T14:03:42.996491+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:03:42.996491+0000","last_undegraded":"2024-03-21T14:03:42.996491+0000","last_fullsized":"2024-03-21T14:03:42.996491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.077976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532613+0000","last_change":"2024-03-21T14:03:41.906118+0000","last_active":"2024-03-21T14:03:43.532613+0000","last_peered":"2024-03-21T14:03:43.532613+0000","last_clean":"2024-03-21T14:03:43.532613+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:03:43.532613+0000","last_undegraded":"2024-03-21T14:03:43.532613+0000","last_fullsized":"2024-03-21T14:03:43.532613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.906016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.906016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:46.096482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999520+0000","last_change":"2024-03-21T14:02:37.454805+0000","last_active":"2024-03-21T14:03:42.999520+0000","last_peered":"2024-03-21T14:03:42.999520+0000","last_clean":"2024-03-21T14:03:42.999520+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:03:42.999520+0000","last_undegraded":"2024-03-21T14:03:42.999520+0000","last_fullsized":"2024-03-21T14:03:42.999520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:54.374515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734655+0000","last_change":"2024-03-21T14:03:38.199089+0000","last_active":"2024-03-21T14:03:42.734655+0000","last_peered":"2024-03-21T14:03:42.734655+0000","last_clean":"2024-03-21T14:03:42.734655+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:03:42.734655+0000","last_undegraded":"2024-03-21T14:03:42.734655+0000","last_fullsized":"2024-03-21T14:03:42.734655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.198986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.198986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:47.188712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":56,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395952+0000","last_change":"2024-03-21T14:02:35.774132+0000","last_active":"2024-03-21T14:03:42.395952+0000","last_peered":"2024-03-21T14:03:42.395952+0000","last_clean":"2024-03-21T14:03:42.395952+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:03:42.395952+0000","last_undegraded":"2024-03-21T14:03:42.395952+0000","last_fullsized":"2024-03-21T14:03:42.395952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.386896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999034+0000","last_change":"2024-03-21T14:02:39.517249+0000","last_active":"2024-03-21T14:03:42.999034+0000","last_peered":"2024-03-21T14:03:42.999034+0000","last_clean":"2024-03-21T14:03:42.999034+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:03:42.999034+0000","last_undegraded":"2024-03-21T14:03:42.999034+0000","last_fullsized":"2024-03-21T14:03:42.999034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.493578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533577+0000","last_change":"2024-03-21T14:02:33.448412+0000","last_active":"2024-03-21T14:03:43.533577+0000","last_peered":"2024-03-21T14:03:43.533577+0000","last_clean":"2024-03-21T14:03:43.533577+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:03:43.533577+0000","last_undegraded":"2024-03-21T14:03:43.533577+0000","last_fullsized":"2024-03-21T14:03:43.533577+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.124819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":72,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998643+0000","last_change":"2024-03-21T14:02:35.775475+0000","last_active":"2024-03-21T14:03:42.998643+0000","last_peered":"2024-03-21T14:03:42.998643+0000","last_clean":"2024-03-21T14:03:42.998643+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:03:42.998643+0000","last_undegraded":"2024-03-21T14:03:42.998643+0000","last_fullsized":"2024-03-21T14:03:42.998643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.820694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.503017+0000","last_change":"2024-03-21T14:02:37.454898+0000","last_active":"2024-03-21T14:03:58.503017+0000","last_peered":"2024-03-21T14:03:58.503017+0000","last_clean":"2024-03-21T14:03:58.503017+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:03:58.503017+0000","last_undegraded":"2024-03-21T14:03:58.503017+0000","last_fullsized":"2024-03-21T14:03:58.503017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395170+0000","last_change":"2024-03-21T14:02:39.472019+0000","last_active":"2024-03-21T14:03:42.395170+0000","last_peered":"2024-03-21T14:03:42.395170+0000","last_clean":"2024-03-21T14:03:42.395170+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:03:42.395170+0000","last_undegraded":"2024-03-21T14:03:42.395170+0000","last_fullsized":"2024-03-21T14:03:42.395170+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:07.761931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:45.909142+0000","last_change":"2024-03-21T14:03:45.909165+0000","last_active":"2024-03-21T14:03:45.909142+0000","last_peered":"2024-03-21T14:03:45.909142+0000","last_clean":"2024-03-21T14:03:45.909142+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:03:45.909142+0000","last_undegraded":"2024-03-21T14:03:45.909142+0000","last_fullsized":"2024-03-21T14:03:45.909142+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:45.909064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:45.909064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.828690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389532+0000","last_change":"2024-03-21T14:02:35.450101+0000","last_active":"2024-03-21T14:03:42.389532+0000","last_peered":"2024-03-21T14:03:42.389532+0000","last_clean":"2024-03-21T14:03:42.389532+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:03:42.389532+0000","last_undegraded":"2024-03-21T14:03:42.389532+0000","last_fullsized":"2024-03-21T14:03:42.389532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.268875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:59.969419+0000","last_change":"2024-03-21T14:03:59.969457+0000","last_active":"2024-03-21T14:03:59.969419+0000","last_peered":"2024-03-21T14:03:59.969419+0000","last_clean":"2024-03-21T14:03:59.969419+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:03:59.969419+0000","last_undegraded":"2024-03-21T14:03:59.969419+0000","last_fullsized":"2024-03-21T14:03:59.969419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:59.969354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:59.969354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:10.864954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.825087+0000","last_change":"2024-03-21T14:03:58.825134+0000","last_active":"2024-03-21T14:03:58.825087+0000","last_peered":"2024-03-21T14:03:58.825087+0000","last_clean":"2024-03-21T14:03:58.825087+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:03:58.825087+0000","last_undegraded":"2024-03-21T14:03:58.825087+0000","last_fullsized":"2024-03-21T14:03:58.825087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:58.825002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:58.825002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:18.466884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395196+0000","last_change":"2024-03-21T14:03:39.021466+0000","last_active":"2024-03-21T14:03:42.395196+0000","last_peered":"2024-03-21T14:03:42.395196+0000","last_clean":"2024-03-21T14:03:42.395196+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:03:42.395196+0000","last_undegraded":"2024-03-21T14:03:42.395196+0000","last_fullsized":"2024-03-21T14:03:42.395196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:39.021385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.021385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.898145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.833768+0000","last_change":"2024-03-21T14:03:56.833815+0000","last_active":"2024-03-21T14:03:56.833768+0000","last_peered":"2024-03-21T14:03:56.833768+0000","last_clean":"2024-03-21T14:03:56.833768+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:03:56.833768+0000","last_undegraded":"2024-03-21T14:03:56.833768+0000","last_fullsized":"2024-03-21T14:03:56.833768+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.833681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.833681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:13.360093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734954+0000","last_change":"2024-03-21T14:02:35.773869+0000","last_active":"2024-03-21T14:03:42.734954+0000","last_peered":"2024-03-21T14:03:42.734954+0000","last_clean":"2024-03-21T14:03:42.734954+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:03:42.734954+0000","last_undegraded":"2024-03-21T14:03:42.734954+0000","last_fullsized":"2024-03-21T14:03:42.734954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.067670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995644+0000","last_change":"2024-03-21T14:03:38.925041+0000","last_active":"2024-03-21T14:03:42.995644+0000","last_peered":"2024-03-21T14:03:42.995644+0000","last_clean":"2024-03-21T14:03:42.995644+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:03:42.995644+0000","last_undegraded":"2024-03-21T14:03:42.995644+0000","last_fullsized":"2024-03-21T14:03:42.995644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:38.924957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.924957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:06.656539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.394809+0000","last_change":"2024-03-21T14:03:41.007728+0000","last_active":"2024-03-21T14:03:42.394809+0000","last_peered":"2024-03-21T14:03:42.394809+0000","last_clean":"2024-03-21T14:03:42.394809+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:03:42.394809+0000","last_undegraded":"2024-03-21T14:03:42.394809+0000","last_fullsized":"2024-03-21T14:03:42.394809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.007598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:54.075924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:04.531346+0000","last_change":"2024-03-21T14:04:04.531383+0000","last_active":"2024-03-21T14:04:04.531346+0000","last_peered":"2024-03-21T14:04:04.531346+0000","last_clean":"2024-03-21T14:04:04.531346+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:04:04.531346+0000","last_undegraded":"2024-03-21T14:04:04.531346+0000","last_fullsized":"2024-03-21T14:04:04.531346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:04.531262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:04.531262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:12.017510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":79,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996413+0000","last_change":"2024-03-21T14:02:58.514359+0000","last_active":"2024-03-21T14:03:42.996413+0000","last_peered":"2024-03-21T14:03:42.996413+0000","last_clean":"2024-03-21T14:03:42.996413+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:03:42.996413+0000","last_undegraded":"2024-03-21T14:03:42.996413+0000","last_fullsized":"2024-03-21T14:03:42.996413+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.552419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.396180+0000","last_change":"2024-03-21T14:02:39.472519+0000","last_active":"2024-03-21T14:03:42.396180+0000","last_peered":"2024-03-21T14:03:42.396180+0000","last_clean":"2024-03-21T14:03:42.396180+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:03:42.396180+0000","last_undegraded":"2024-03-21T14:03:42.396180+0000","last_fullsized":"2024-03-21T14:03:42.396180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:02.779767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.998816+0000","last_change":"2024-03-21T14:03:35.349742+0000","last_active":"2024-03-21T14:03:42.998816+0000","last_peered":"2024-03-21T14:03:42.998816+0000","last_clean":"2024-03-21T14:03:42.998816+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:03:42.998816+0000","last_undegraded":"2024-03-21T14:03:42.998816+0000","last_fullsized":"2024-03-21T14:03:42.998816+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:03:35.349636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:35.349636+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:51.186069+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389279+0000","last_change":"2024-03-21T14:02:33.441265+0000","last_active":"2024-03-21T14:03:42.389279+0000","last_peered":"2024-03-21T14:03:42.389279+0000","last_clean":"2024-03-21T14:03:42.389279+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:03:42.389279+0000","last_undegraded":"2024-03-21T14:03:42.389279+0000","last_fullsized":"2024-03-21T14:03:42.389279+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.975558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":76,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.503055+0000","last_change":"2024-03-21T14:03:48.369792+0000","last_active":"2024-03-21T14:03:58.503055+0000","last_peered":"2024-03-21T14:03:58.503055+0000","last_clean":"2024-03-21T14:03:58.503055+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:03:58.503055+0000","last_undegraded":"2024-03-21T14:03:58.503055+0000","last_fullsized":"2024-03-21T14:03:58.503055+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:03:48.369672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:48.369672+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:04.178152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996233+0000","last_change":"2024-03-21T14:02:37.456968+0000","last_active":"2024-03-21T14:03:42.996233+0000","last_peered":"2024-03-21T14:03:42.996233+0000","last_clean":"2024-03-21T14:03:42.996233+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:03:42.996233+0000","last_undegraded":"2024-03-21T14:03:42.996233+0000","last_fullsized":"2024-03-21T14:03:42.996233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.729850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:46.872553+0000","last_change":"2024-03-21T14:03:46.872577+0000","last_active":"2024-03-21T14:03:46.872553+0000","last_peered":"2024-03-21T14:03:46.872553+0000","last_clean":"2024-03-21T14:03:46.872553+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:03:46.872553+0000","last_undegraded":"2024-03-21T14:03:46.872553+0000","last_fullsized":"2024-03-21T14:03:46.872553+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:46.872477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:46.872477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:11.050084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:04.811743+0000","last_change":"2024-03-21T14:04:04.811780+0000","last_active":"2024-03-21T14:04:04.811743+0000","last_peered":"2024-03-21T14:04:04.811743+0000","last_clean":"2024-03-21T14:04:04.811743+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:04:04.811743+0000","last_undegraded":"2024-03-21T14:04:04.811743+0000","last_fullsized":"2024-03-21T14:04:04.811743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:04:04.811636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:15.172795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:50.047827+0000","last_change":"2024-03-21T14:03:50.047857+0000","last_active":"2024-03-21T14:03:50.047827+0000","last_peered":"2024-03-21T14:03:50.047827+0000","last_clean":"2024-03-21T14:03:50.047827+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:03:50.047827+0000","last_undegraded":"2024-03-21T14:03:50.047827+0000","last_fullsized":"2024-03-21T14:03:50.047827+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:50.047757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:50.047757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:57.758773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757698+0000","last_change":"2024-03-21T14:03:39.830694+0000","last_active":"2024-03-21T14:03:43.757698+0000","last_peered":"2024-03-21T14:03:43.757698+0000","last_clean":"2024-03-21T14:03:43.757698+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:03:43.757698+0000","last_undegraded":"2024-03-21T14:03:43.757698+0000","last_fullsized":"2024-03-21T14:03:43.757698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:03:39.830542+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.877216+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:55.062292+0000","last_change":"2024-03-21T14:03:55.062330+0000","last_active":"2024-03-21T14:03:55.062292+0000","last_peered":"2024-03-21T14:03:55.062292+0000","last_clean":"2024-03-21T14:03:55.062292+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:03:55.062292+0000","last_undegraded":"2024-03-21T14:03:55.062292+0000","last_fullsized":"2024-03-21T14:03:55.062292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:55.062188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:55.062188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.285350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999465+0000","last_change":"2024-03-21T14:02:39.470869+0000","last_active":"2024-03-21T14:03:42.999465+0000","last_peered":"2024-03-21T14:03:42.999465+0000","last_clean":"2024-03-21T14:03:42.999465+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:03:42.999465+0000","last_undegraded":"2024-03-21T14:03:42.999465+0000","last_fullsized":"2024-03-21T14:03:42.999465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.994358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":312,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.817888+0000","last_change":"2024-03-21T14:03:54.817955+0000","last_active":"2024-03-21T14:03:54.817888+0000","last_peered":"2024-03-21T14:03:54.817888+0000","last_clean":"2024-03-21T14:03:54.817888+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:03:54.817888+0000","last_undegraded":"2024-03-21T14:03:54.817888+0000","last_fullsized":"2024-03-21T14:03:54.817888+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-03-21T14:03:54.817784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.817784+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.112690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:03.733548+0000","last_change":"2024-03-21T14:04:03.733569+0000","last_active":"2024-03-21T14:04:03.733548+0000","last_peered":"2024-03-21T14:04:03.733548+0000","last_clean":"2024-03-21T14:04:03.733548+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:04:03.733548+0000","last_undegraded":"2024-03-21T14:04:03.733548+0000","last_fullsized":"2024-03-21T14:04:03.733548+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:03.733488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:03.733488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:17.697000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":96,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:06.099989+0000","last_change":"2024-03-21T14:04:06.100022+0000","last_active":"2024-03-21T14:04:06.099989+0000","last_peered":"2024-03-21T14:04:06.099989+0000","last_clean":"2024-03-21T14:04:06.099989+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:04:06.099989+0000","last_undegraded":"2024-03-21T14:04:06.099989+0000","last_fullsized":"2024-03-21T14:04:06.099989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:04:06.099924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:04:06.099924+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:22.026109+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.533239+0000","last_change":"2024-03-21T14:02:37.459722+0000","last_active":"2024-03-21T14:03:43.533239+0000","last_peered":"2024-03-21T14:03:43.533239+0000","last_clean":"2024-03-21T14:03:43.533239+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:03:43.533239+0000","last_undegraded":"2024-03-21T14:03:43.533239+0000","last_fullsized":"2024-03-21T14:03:43.533239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.571697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756191+0000","last_change":"2024-03-21T14:02:39.471788+0000","last_active":"2024-03-21T14:03:43.756191+0000","last_peered":"2024-03-21T14:03:43.756191+0000","last_clean":"2024-03-21T14:03:43.756191+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:03:43.756191+0000","last_undegraded":"2024-03-21T14:03:43.756191+0000","last_fullsized":"2024-03-21T14:03:43.756191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:58.372670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995815+0000","last_change":"2024-03-21T14:02:33.447469+0000","last_active":"2024-03-21T14:03:42.995815+0000","last_peered":"2024-03-21T14:03:42.995815+0000","last_clean":"2024-03-21T14:03:42.995815+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:03:42.995815+0000","last_undegraded":"2024-03-21T14:03:42.995815+0000","last_fullsized":"2024-03-21T14:03:42.995815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:55.063604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":157,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:04.581307+0000","last_change":"2024-03-21T14:03:49.845488+0000","last_active":"2024-03-21T14:04:04.581307+0000","last_peered":"2024-03-21T14:04:04.581307+0000","last_clean":"2024-03-21T14:04:04.581307+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:04:04.581307+0000","last_undegraded":"2024-03-21T14:04:04.581307+0000","last_fullsized":"2024-03-21T14:04:04.581307+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:03:49.845376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:03:49.845376+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:12.461975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996292+0000","last_change":"2024-03-21T14:02:35.449792+0000","last_active":"2024-03-21T14:03:42.996292+0000","last_peered":"2024-03-21T14:03:42.996292+0000","last_clean":"2024-03-21T14:03:42.996292+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:03:42.996292+0000","last_undegraded":"2024-03-21T14:03:42.996292+0000","last_fullsized":"2024-03-21T14:03:42.996292+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.553541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999655+0000","last_change":"2024-03-21T14:02:37.455752+0000","last_active":"2024-03-21T14:03:42.999655+0000","last_peered":"2024-03-21T14:03:42.999655+0000","last_clean":"2024-03-21T14:03:42.999655+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:03:42.999655+0000","last_undegraded":"2024-03-21T14:03:42.999655+0000","last_fullsized":"2024-03-21T14:03:42.999655+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.531727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.756153+0000","last_change":"2024-03-21T14:02:39.516952+0000","last_active":"2024-03-21T14:03:43.756153+0000","last_peered":"2024-03-21T14:03:43.756153+0000","last_clean":"2024-03-21T14:03:43.756153+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:03:43.756153+0000","last_undegraded":"2024-03-21T14:03:43.756153+0000","last_fullsized":"2024-03-21T14:03:43.756153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:00.885390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:05.852636+0000","last_change":"2024-03-21T14:04:05.852662+0000","last_active":"2024-03-21T14:04:05.852636+0000","last_peered":"2024-03-21T14:04:05.852636+0000","last_clean":"2024-03-21T14:04:05.852636+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:04:05.852636+0000","last_undegraded":"2024-03-21T14:04:05.852636+0000","last_fullsized":"2024-03-21T14:04:05.852636+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:05.852550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:05.852550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:13.342227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":91,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.996230+0000","last_change":"2024-03-21T14:03:41.967449+0000","last_active":"2024-03-21T14:03:42.996230+0000","last_peered":"2024-03-21T14:03:42.996230+0000","last_clean":"2024-03-21T14:03:42.996230+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:03:42.996230+0000","last_undegraded":"2024-03-21T14:03:42.996230+0000","last_fullsized":"2024-03-21T14:03:42.996230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:03:41.967343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:41.967343+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:58.280117+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.734983+0000","last_change":"2024-03-21T14:02:37.454582+0000","last_active":"2024-03-21T14:03:42.734983+0000","last_peered":"2024-03-21T14:03:42.734983+0000","last_clean":"2024-03-21T14:03:42.734983+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:03:42.734983+0000","last_undegraded":"2024-03-21T14:03:42.734983+0000","last_fullsized":"2024-03-21T14:03:42.734983+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.341912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.995555+0000","last_change":"2024-03-21T14:02:39.473493+0000","last_active":"2024-03-21T14:03:42.995555+0000","last_peered":"2024-03-21T14:03:42.995555+0000","last_clean":"2024-03-21T14:03:42.995555+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:03:42.995555+0000","last_undegraded":"2024-03-21T14:03:42.995555+0000","last_fullsized":"2024-03-21T14:03:42.995555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.101298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999221+0000","last_change":"2024-03-21T14:02:33.443262+0000","last_active":"2024-03-21T14:03:42.999221+0000","last_peered":"2024-03-21T14:03:42.999221+0000","last_clean":"2024-03-21T14:03:42.999221+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:03:42.999221+0000","last_undegraded":"2024-03-21T14:03:42.999221+0000","last_fullsized":"2024-03-21T14:03:42.999221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.529943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757322+0000","last_change":"2024-03-21T14:03:38.812824+0000","last_active":"2024-03-21T14:03:43.757322+0000","last_peered":"2024-03-21T14:03:43.757322+0000","last_clean":"2024-03-21T14:03:43.757322+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:03:43.757322+0000","last_undegraded":"2024-03-21T14:03:43.757322+0000","last_fullsized":"2024-03-21T14:03:43.757322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:03:38.812732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:38.812732+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:02.836722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.502677+0000","last_change":"2024-03-21T14:03:56.007959+0000","last_active":"2024-03-21T14:03:58.502677+0000","last_peered":"2024-03-21T14:03:58.502677+0000","last_clean":"2024-03-21T14:03:58.502677+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:03:58.502677+0000","last_undegraded":"2024-03-21T14:03:58.502677+0000","last_fullsized":"2024-03-21T14:03:58.502677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:56.007835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.007835+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:18.104333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.532331+0000","last_change":"2024-03-21T14:02:39.517270+0000","last_active":"2024-03-21T14:03:43.532331+0000","last_peered":"2024-03-21T14:03:43.532331+0000","last_clean":"2024-03-21T14:03:43.532331+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:03:43.532331+0000","last_undegraded":"2024-03-21T14:03:43.532331+0000","last_fullsized":"2024-03-21T14:03:43.532331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:57.528174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":60,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.935772+0000","last_change":"2024-03-21T14:03:43.935798+0000","last_active":"2024-03-21T14:03:43.935772+0000","last_peered":"2024-03-21T14:03:43.935772+0000","last_clean":"2024-03-21T14:03:43.935772+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:03:43.935772+0000","last_undegraded":"2024-03-21T14:03:43.935772+0000","last_fullsized":"2024-03-21T14:03:43.935772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:03:43.935711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.935711+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:53.752006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389164+0000","last_change":"2024-03-21T14:02:35.444623+0000","last_active":"2024-03-21T14:03:42.389164+0000","last_peered":"2024-03-21T14:03:42.389164+0000","last_clean":"2024-03-21T14:03:42.389164+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:03:42.389164+0000","last_undegraded":"2024-03-21T14:03:42.389164+0000","last_fullsized":"2024-03-21T14:03:42.389164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:49.184966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.502706+0000","last_change":"2024-03-21T14:02:37.457785+0000","last_active":"2024-03-21T14:03:58.502706+0000","last_peered":"2024-03-21T14:03:58.502706+0000","last_clean":"2024-03-21T14:03:58.502706+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:03:58.502706+0000","last_undegraded":"2024-03-21T14:03:58.502706+0000","last_fullsized":"2024-03-21T14:03:58.502706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:04.006307+0000","last_change":"2024-03-21T14:04:04.006327+0000","last_active":"2024-03-21T14:04:04.006307+0000","last_peered":"2024-03-21T14:04:04.006307+0000","last_clean":"2024-03-21T14:04:04.006307+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:04:04.006307+0000","last_undegraded":"2024-03-21T14:04:04.006307+0000","last_fullsized":"2024-03-21T14:04:04.006307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:04:04.006246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:21.012335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395781+0000","last_change":"2024-03-21T14:03:36.591924+0000","last_active":"2024-03-21T14:03:42.395781+0000","last_peered":"2024-03-21T14:03:42.395781+0000","last_clean":"2024-03-21T14:03:42.395781+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:03:42.395781+0000","last_undegraded":"2024-03-21T14:03:42.395781+0000","last_fullsized":"2024-03-21T14:03:42.395781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:03:36.591844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:44.332405+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":87,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:50.545354+0000","last_change":"2024-03-21T14:03:50.545394+0000","last_active":"2024-03-21T14:03:50.545354+0000","last_peered":"2024-03-21T14:03:50.545354+0000","last_clean":"2024-03-21T14:03:50.545354+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:03:50.545354+0000","last_undegraded":"2024-03-21T14:03:50.545354+0000","last_fullsized":"2024-03-21T14:03:50.545354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:03:50.545283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:50.545283+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:08.508308+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.388989+0000","last_change":"2024-03-21T14:02:37.456640+0000","last_active":"2024-03-21T14:03:42.388989+0000","last_peered":"2024-03-21T14:03:42.388989+0000","last_clean":"2024-03-21T14:03:42.388989+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:03:42.388989+0000","last_undegraded":"2024-03-21T14:03:42.388989+0000","last_fullsized":"2024-03-21T14:03:42.388989+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:04.154776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:54.381560+0000","last_change":"2024-03-21T14:03:54.381586+0000","last_active":"2024-03-21T14:03:54.381560+0000","last_peered":"2024-03-21T14:03:54.381560+0000","last_clean":"2024-03-21T14:03:54.381560+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:03:54.381560+0000","last_undegraded":"2024-03-21T14:03:54.381560+0000","last_fullsized":"2024-03-21T14:03:54.381560+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:03:54.381485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:12.874634+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.503009+0000","last_change":"2024-03-21T14:03:43.001482+0000","last_active":"2024-03-21T14:03:58.503009+0000","last_peered":"2024-03-21T14:03:58.503009+0000","last_clean":"2024-03-21T14:03:58.503009+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:03:58.503009+0000","last_undegraded":"2024-03-21T14:03:58.503009+0000","last_fullsized":"2024-03-21T14:03:58.503009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:43.001363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:43.001363+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:05.127322+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:56.674672+0000","last_change":"2024-03-21T14:03:56.674692+0000","last_active":"2024-03-21T14:03:56.674672+0000","last_peered":"2024-03-21T14:03:56.674672+0000","last_clean":"2024-03-21T14:03:56.674672+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:03:56.674672+0000","last_undegraded":"2024-03-21T14:03:56.674672+0000","last_fullsized":"2024-03-21T14:03:56.674672+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.674610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.674610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:25.472031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":80,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:53.758265+0000","last_change":"2024-03-21T14:03:53.758307+0000","last_active":"2024-03-21T14:03:53.758265+0000","last_peered":"2024-03-21T14:03:53.758265+0000","last_clean":"2024-03-21T14:03:53.758265+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:03:53.758265+0000","last_undegraded":"2024-03-21T14:03:53.758265+0000","last_fullsized":"2024-03-21T14:03:53.758265+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:03:53.758186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.758186+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:22.866515+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395610+0000","last_change":"2024-03-21T14:02:39.516061+0000","last_active":"2024-03-21T14:03:42.395610+0000","last_peered":"2024-03-21T14:03:42.395610+0000","last_clean":"2024-03-21T14:03:42.395610+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:03:42.395610+0000","last_undegraded":"2024-03-21T14:03:42.395610+0000","last_fullsized":"2024-03-21T14:03:42.395610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:59.869674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.999626+0000","last_change":"2024-03-21T14:02:37.454216+0000","last_active":"2024-03-21T14:03:42.999626+0000","last_peered":"2024-03-21T14:03:42.999626+0000","last_clean":"2024-03-21T14:03:42.999626+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:03:42.999626+0000","last_undegraded":"2024-03-21T14:03:42.999626+0000","last_fullsized":"2024-03-21T14:03:42.999626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:01.938463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389026+0000","last_change":"2024-03-21T14:03:34.139285+0000","last_active":"2024-03-21T14:03:42.389026+0000","last_peered":"2024-03-21T14:03:42.389026+0000","last_clean":"2024-03-21T14:03:42.389026+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:03:42.389026+0000","last_undegraded":"2024-03-21T14:03:42.389026+0000","last_fullsized":"2024-03-21T14:03:42.389026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:34.139127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:34.139127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:56.352287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389671+0000","last_change":"2024-03-21T14:02:35.450014+0000","last_active":"2024-03-21T14:03:42.389671+0000","last_peered":"2024-03-21T14:03:42.389671+0000","last_clean":"2024-03-21T14:03:42.389671+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:03:42.389671+0000","last_undegraded":"2024-03-21T14:03:42.389671+0000","last_fullsized":"2024-03-21T14:03:42.389671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:48.855239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:04:05.099902+0000","last_change":"2024-03-21T14:04:05.099937+0000","last_active":"2024-03-21T14:04:05.099902+0000","last_peered":"2024-03-21T14:04:05.099902+0000","last_clean":"2024-03-21T14:04:05.099902+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:04:05.099902+0000","last_undegraded":"2024-03-21T14:04:05.099902+0000","last_fullsized":"2024-03-21T14:04:05.099902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:04:05.099841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:05.099841+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:30.741752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:43.757692+0000","last_change":"2024-03-21T14:02:37.456947+0000","last_active":"2024-03-21T14:03:43.757692+0000","last_peered":"2024-03-21T14:03:43.757692+0000","last_clean":"2024-03-21T14:03:43.757692+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:03:43.757692+0000","last_undegraded":"2024-03-21T14:03:43.757692+0000","last_fullsized":"2024-03-21T14:03:43.757692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:02:36.433777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:04:05.581511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.735065+0000","last_change":"2024-03-21T14:02:33.442462+0000","last_active":"2024-03-21T14:03:42.735065+0000","last_peered":"2024-03-21T14:03:42.735065+0000","last_clean":"2024-03-21T14:03:42.735065+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:03:42.735065+0000","last_undegraded":"2024-03-21T14:03:42.735065+0000","last_fullsized":"2024-03-21T14:03:42.735065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:53.441774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":83,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:51.976591+0000","last_change":"2024-03-21T14:03:51.976637+0000","last_active":"2024-03-21T14:03:51.976591+0000","last_peered":"2024-03-21T14:03:51.976591+0000","last_clean":"2024-03-21T14:03:51.976591+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:03:51.976591+0000","last_undegraded":"2024-03-21T14:03:51.976591+0000","last_fullsized":"2024-03-21T14:03:51.976591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:03:51.976515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:03:51.976515+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:16.722963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389971+0000","last_change":"2024-03-21T14:02:39.459328+0000","last_active":"2024-03-21T14:03:42.389971+0000","last_peered":"2024-03-21T14:03:42.389971+0000","last_clean":"2024-03-21T14:03:42.389971+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:03:42.389971+0000","last_undegraded":"2024-03-21T14:03:42.389971+0000","last_fullsized":"2024-03-21T14:03:42.389971+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:02:38.441204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:52.833817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"72'8","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:58.502710+0000","last_change":"2024-03-21T14:03:57.040828+0000","last_active":"2024-03-21T14:03:58.502710+0000","last_peered":"2024-03-21T14:03:58.502710+0000","last_clean":"2024-03-21T14:03:58.502710+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:03:58.502710+0000","last_undegraded":"2024-03-21T14:03:58.502710+0000","last_fullsized":"2024-03-21T14:03:58.502710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:03:57.040741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:57.040741+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:05:26.294593+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.395341+0000","last_change":"2024-03-21T14:02:33.442529+0000","last_active":"2024-03-21T14:03:42.395341+0000","last_peered":"2024-03-21T14:03:42.395341+0000","last_clean":"2024-03-21T14:03:42.395341+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:03:42.395341+0000","last_undegraded":"2024-03-21T14:03:42.395341+0000","last_fullsized":"2024-03-21T14:03:42.395341+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:02:32.427364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:51.804413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-03-21T14:03:42.389402+0000","last_change":"2024-03-21T14:02:35.450511+0000","last_active":"2024-03-21T14:03:42.389402+0000","last_peered":"2024-03-21T14:03:42.389402+0000","last_clean":"2024-03-21T14:03:42.389402+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:03:42.389402+0000","last_undegraded":"2024-03-21T14:03:42.389402+0000","last_fullsized":"2024-03-21T14:03:42.389402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:02:34.430360+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:03:56.494861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103191,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":927293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430236,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28328,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716824,"statfs":{"total":95995035648,"available":95966027776,"internally_reserved":0,"allocated":1343488,"data_stored":925330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31956,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713196,"statfs":{"total":95995035648,"available":95962312704,"internally_reserved":0,"allocated":798720,"data_stored":337516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1368064,"data_stored":926124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":757760,"data_stored":335310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182443,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":794624,"data_stored":336302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018919,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31944,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713208,"statfs":{"total":95995035648,"available":95962324992,"internally_reserved":0,"allocated":786432,"data_stored":336016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691874,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31932,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713220,"statfs":{"total":95995035648,"available":95962337280,"internally_reserved":0,"allocated":774144,"data_stored":336557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:04:08.857 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-21T14:04:08.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump 2024-03-21T14:04:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:08 smithi118 ceph-mon[33862]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:08 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:08 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:08 smithi118 ceph-mon[33862]: osdmap e86: 8 total, 8 up, 8 in 2024-03-21T14:04:09.105 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:04:09.894 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:04:09.526Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003370454s 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:09.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:09.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:04:09.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:04:09.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T14:04:09.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:04:09.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:09 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:04:10.115 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:10.115 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:10.115 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:04:10.115 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:04:10.115 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: osdmap e87: 8 total, 8 up, 8 in 2024-03-21T14:04:10.116 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:04:10.116 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:09 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:04:10.431 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:10 smithi118 systemd[1]: Starting Ceph grafana.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:04:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:10 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:04:10.932 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:10 smithi118 podman[64844]: 2024-03-21 14:04:10.41831986 +0000 UTC m=+0.018869925 image pull c8b91775d855b99270fc5d22f3c6737e8cca01ef4c25c8b0362295e0746fa39b quay.io/ceph/grafana:10.4.0 2024-03-21T14:04:10.932 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:10 smithi118 podman[64844]: 2024-03-21 14:04:10.523881418 +0000 UTC m=+0.124431458 container create 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 (image=quay.io/ceph/grafana:10.4.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T14:04:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:04:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:04:11.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:10 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 87 2024-03-21T14:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout:fsid c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-03-21T13:56:54.530125+0000 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2024-03-21T14:04:08.671301+0000 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:crush_version 18 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:full_ratio 0.95 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:backfillfull_ratio 0.9 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:nearfull_ratio 0.85 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:require_min_compat_client luminous 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:min_compat_client luminous 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:require_osd_release squid 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:stretch_mode_enabled false 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 68 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-21T14:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:max_osd 8 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 up in weight 1 up_from 8 up_thru 74 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6801/930096433 v2:172.21.15.3:6802/930096433 exists,up daf9bc3c-4e73-4379-afb6-7ee181b686fb 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 up in weight 1 up_from 13 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6805/1526485482 v2:172.21.15.3:6806/1526485482 exists,up 1652f24d-aa0e-471c-bf54-03814b1ee11d 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 up in weight 1 up_from 18 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6809/3751378277 v2:172.21.15.3:6810/3751378277 exists,up 5e67d907-f8d1-4ecc-85d3-df88da5c364c 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 up in weight 1 up_from 25 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6813/1918209225 v2:172.21.15.3:6814/1918209225 exists,up ad8dabdf-fce5-4312-9e80-dbbb6ba2947c 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 up in weight 1 up_from 30 up_thru 74 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6800/296472230 v2:172.21.15.118:6801/296472230 exists,up b5d486b1-2cb6-46c7-8664-994af07ff7b5 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 up in weight 1 up_from 40 up_thru 74 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6804/326385780 v2:172.21.15.118:6805/326385780 exists,up bff24605-38ce-47ce-9142-4ced3766476e 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 up in weight 1 up_from 48 up_thru 68 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6808/1322208109 v2:172.21.15.118:6809/1322208109 exists,up 73a781e7-8660-4885-b976-2aeb8515109a 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 up in weight 1 up_from 58 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6812/2128678671 v2:172.21.15.118:6813/2128678671 exists,up 3e95559b-79f6-40ba-9d98-d350bcdd60e0 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.4 [1,5] 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.b [1,2] 2024-03-21T14:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 4.13 [1,2] 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/45234419 expires 2024-03-22T14:03:15.227370+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/384540368 expires 2024-03-22T14:03:15.227370+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3883474825 expires 2024-03-22T14:03:15.227370+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1598547836 expires 2024-03-22T14:03:15.227370+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/3687155565 expires 2024-03-22T14:03:15.227370+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3884352348 expires 2024-03-22T13:57:56.576834+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2319227492 expires 2024-03-22T13:57:56.576834+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2154949954 expires 2024-03-22T13:57:29.327821+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/4162970769 expires 2024-03-22T13:57:29.327821+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2292819669 expires 2024-03-22T13:57:29.327821+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/3411869253 expires 2024-03-22T14:03:15.227370+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/520312891 expires 2024-03-22T13:57:56.576834+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/2267426691 expires 2024-03-22T13:57:29.327821+0000 2024-03-21T14:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/3211426112 expires 2024-03-22T13:57:56.576834+0000 2024-03-21T14:04:11.738 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 74 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6801/930096433 v2:172.21.15.3:6802/930096433 exists,up daf9bc3c-4e73-4379-afb6-7ee181b686fb', 'osd.1 up in weight 1 up_from 13 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6805/1526485482 v2:172.21.15.3:6806/1526485482 exists,up 1652f24d-aa0e-471c-bf54-03814b1ee11d', 'osd.2 up in weight 1 up_from 18 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6809/3751378277 v2:172.21.15.3:6810/3751378277 exists,up 5e67d907-f8d1-4ecc-85d3-df88da5c364c', 'osd.3 up in weight 1 up_from 25 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.3:6813/1918209225 v2:172.21.15.3:6814/1918209225 exists,up ad8dabdf-fce5-4312-9e80-dbbb6ba2947c', 'osd.4 up in weight 1 up_from 30 up_thru 74 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6800/296472230 v2:172.21.15.118:6801/296472230 exists,up b5d486b1-2cb6-46c7-8664-994af07ff7b5', 'osd.5 up in weight 1 up_from 40 up_thru 74 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6804/326385780 v2:172.21.15.118:6805/326385780 exists,up bff24605-38ce-47ce-9142-4ced3766476e', 'osd.6 up in weight 1 up_from 48 up_thru 68 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6808/1322208109 v2:172.21.15.118:6809/1322208109 exists,up 73a781e7-8660-4885-b976-2aeb8515109a', 'osd.7 up in weight 1 up_from 58 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.118:6812/2128678671 v2:172.21.15.118:6813/2128678671 exists,up 3e95559b-79f6-40ba-9d98-d350bcdd60e0'] 2024-03-21T14:04:11.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon mon.a config show 2024-03-21T14:04:11.982 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:11 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:04:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:11 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:04:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:11 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1955074764' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T14:04:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:11 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:04:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:11 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:04:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:11 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1955074764' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T14:04:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:11 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:04:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:11 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:04:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:11 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1955074764' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 podman[64844]: 2024-03-21 14:04:12.206553986 +0000 UTC m=+1.807104038 container init 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 (image=quay.io/ceph/grafana:10.4.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 podman[64844]: 2024-03-21 14:04:12.210135768 +0000 UTC m=+1.810685801 container start 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 (image=quay.io/ceph/grafana:10.4.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 bash[64844]: 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 systemd[1]: Started Ceph grafana.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337360847Z level=info msg="Starting Grafana" version=10.4.0 commit=03f502a94d17f7dc4e6c34acdf8428aedd986e4c branch=HEAD compiled=2024-03-21T14:04:12Z 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337559739Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337566537Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337569543Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337572134Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337574662Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337577151Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-21T14:04:12.457 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337579741Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.33758259Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337585178Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337587559Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337589949Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337592493Z level=info msg=Target target=[all] 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.33759823Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-21T14:04:12.458 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337600749Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337603038Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337605437Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337607816Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=settings t=2024-03-21T14:04:12.337610251Z level=info msg="App mode production" 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore t=2024-03-21T14:04:12.337846894Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore t=2024-03-21T14:04:12.337858767Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.338331946Z level=info msg="Starting DB migrations" 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.339174366Z level=info msg="Executing migration" id="create migration_log table" 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.33991548Z level=info msg="Migration successfully executed" id="create migration_log table" duration=740.455µs 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.340409102Z level=info msg="Executing migration" id="create user table" 2024-03-21T14:04:12.459 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.34093945Z level=info msg="Migration successfully executed" id="create user table" duration=530.715µs 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.341357037Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.342001726Z level=info msg="Migration successfully executed" id="add unique index user.login" duration=645.198µs 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.342363478Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.342851567Z level=info msg="Migration successfully executed" id="add unique index user.email" duration=488.003µs 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.343225397Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.343669654Z level=info msg="Migration successfully executed" id="drop index UQE_user_login - v1" duration=444.359µs 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.34412684Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.344550049Z level=info msg="Migration successfully executed" id="drop index UQE_user_email - v1" duration=423.199µs 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.344844497Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.346554158Z level=info msg="Migration successfully executed" id="Rename table user to user_v1 - v1" duration=1.707385ms 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.347053292Z level=info msg="Executing migration" id="create user table v2" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.347772046Z level=info msg="Migration successfully executed" id="create user table v2" duration=718.703µs 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.348180003Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-21T14:04:12.460 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.348759873Z level=info msg="Migration successfully executed" id="create index UQE_user_login - v2" duration=579.837µs 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.349170514Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.349724342Z level=info msg="Migration successfully executed" id="create index UQE_user_email - v2" duration=555.269µs 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.350169319Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.350507531Z level=info msg="Migration successfully executed" id="copy data_source v1 to v2" duration=338.045µs 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.350875907Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.351373899Z level=info msg="Migration successfully executed" id="Drop old table user_v1" duration=497.95µs 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.351750015Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.352760709Z level=info msg="Migration successfully executed" id="Add column help_flags1 to user table" duration=1.01015ms 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.353157605Z level=info msg="Executing migration" id="Update user table charset" 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.353180247Z level=info msg="Migration successfully executed" id="Update user table charset" duration=23.7µs 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.353656405Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-21T14:04:12.461 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.354606307Z level=info msg="Migration successfully executed" id="Add last_seen_at column to user" duration=949.752µs 2024-03-21T14:04:12.462 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.355005509Z level=info msg="Executing migration" id="Add missing user data" 2024-03-21T14:04:12.462 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.355183431Z level=info msg="Migration successfully executed" id="Add missing user data" duration=178.043µs 2024-03-21T14:04:12.462 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.355660495Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-21T14:04:12.462 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.356691403Z level=info msg="Migration successfully executed" id="Add is_disabled column to user" duration=1.029456ms 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.357152971Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.357794177Z level=info msg="Migration successfully executed" id="Add index user.login/user.email" duration=643.224µs 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.35826031Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.359353888Z level=info msg="Migration successfully executed" id="Add is_service_account column to user" duration=1.091178ms 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.359786468Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.367308294Z level=info msg="Migration successfully executed" id="Update is_service_account column to nullable" duration=7.518469ms 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.367951465Z level=info msg="Executing migration" id="Add uid column to user" 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.36911972Z level=info msg="Migration successfully executed" id="Add uid column to user" duration=1.167456ms 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.369580148Z level=info msg="Executing migration" id="Update uid column values for users" 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.369759016Z level=info msg="Migration successfully executed" id="Update uid column values for users" duration=178.808µs 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.37022734Z level=info msg="Executing migration" id="Add unique index user_uid" 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.370874321Z level=info msg="Migration successfully executed" id="Add unique index user_uid" duration=646.357µs 2024-03-21T14:04:12.463 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.371361709Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.371946788Z level=info msg="Migration successfully executed" id="create temp user table v1-7" duration=585.085µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.372341259Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.372814817Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v1-7" duration=473.386µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.373261727Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.373759951Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v1-7" duration=498.763µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.374108858Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.374569572Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v1-7" duration=460.991µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.374919385Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.375380126Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v1-7" duration=461.293µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.375705637Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.375726719Z level=info msg="Migration successfully executed" id="Update temp_user table charset" duration=22.047µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.37611352Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.376596377Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_email - v1" duration=483.103µs 2024-03-21T14:04:12.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.376987121Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.377438723Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_org_id - v1" duration=449.975µs 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.377765081Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.378246985Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_code - v1" duration=482.188µs 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.378554489Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.379012259Z level=info msg="Migration successfully executed" id="drop index IDX_temp_user_status - v1" duration=458.202µs 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.379318096Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.381262671Z level=info msg="Migration successfully executed" id="Rename table temp_user to temp_user_tmp_qwerty - v1" duration=1.94377ms 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.381614761Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.382160028Z level=info msg="Migration successfully executed" id="create temp_user v2" duration=545.515µs 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.382476058Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.382942889Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_email - v2" duration=468.596µs 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.383277746Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-21T14:04:12.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.383759105Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_org_id - v2" duration=511.366µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.384066228Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.384510606Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_code - v2" duration=444.112µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.384798284Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.385255729Z level=info msg="Migration successfully executed" id="create index IDX_temp_user_status - v2" duration=457.621µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.385626425Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.385891356Z level=info msg="Migration successfully executed" id="copy temp_user v1 to v2" duration=265.43µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.386206059Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.386586154Z level=info msg="Migration successfully executed" id="drop temp_user_tmp_qwerty" duration=380.387µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.386875597Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.38712476Z level=info msg="Migration successfully executed" id="Set created for temp users that will otherwise prematurely expire" duration=249.176µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.387473297Z level=info msg="Executing migration" id="create star table" 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.387892498Z level=info msg="Migration successfully executed" id="create star table" duration=419.55µs 2024-03-21T14:04:12.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.388243377Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.388728389Z level=info msg="Migration successfully executed" id="add unique index star.user_id_dashboard_id" duration=485.328µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.389075443Z level=info msg="Executing migration" id="create org table v1" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.389542749Z level=info msg="Migration successfully executed" id="create org table v1" duration=467.748µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.389839352Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.390308712Z level=info msg="Migration successfully executed" id="create index UQE_org_name - v1" duration=468.287µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.390626261Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.391074627Z level=info msg="Migration successfully executed" id="create org_user table v1" duration=447.331µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.391397592Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.391930582Z level=info msg="Migration successfully executed" id="create index IDX_org_user_org_id - v1" duration=532.526µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.392238911Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.392718446Z level=info msg="Migration successfully executed" id="create index UQE_org_user_org_id_user_id - v1" duration=479.293µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.393061495Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.393520743Z level=info msg="Migration successfully executed" id="create index IDX_org_user_user_id - v1" duration=459.177µs 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.393819817Z level=info msg="Executing migration" id="Update org table charset" 2024-03-21T14:04:12.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.393841551Z level=info msg="Migration successfully executed" id="Update org table charset" duration=23.898µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.394212498Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.394233139Z level=info msg="Migration successfully executed" id="Update org_user table charset" duration=20.663µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.394548272Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.394657714Z level=info msg="Migration successfully executed" id="Migrate all Read Only Viewers to Viewers" duration=109.792µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.394975112Z level=info msg="Executing migration" id="create dashboard table" 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.395472954Z level=info msg="Migration successfully executed" id="create dashboard table" duration=498.332µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.395771067Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.396312153Z level=info msg="Migration successfully executed" id="add index dashboard.account_id" duration=516.653µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.396619442Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.39714507Z level=info msg="Migration successfully executed" id="add unique index dashboard_account_id_slug" duration=525.971µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.39745546Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.397875316Z level=info msg="Migration successfully executed" id="create dashboard_tag table" duration=420.291µs 2024-03-21T14:04:12.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.39819655Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-21T14:04:12.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.398685602Z level=info msg="Migration successfully executed" id="add unique index dashboard_tag.dasboard_id_term" duration=489.425µs 2024-03-21T14:04:12.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.398989973Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-21T14:04:12.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.399468829Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" duration=480.324µs 2024-03-21T14:04:12.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.39975693Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-21T14:04:12.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.402992515Z level=info msg="Migration successfully executed" id="Rename table dashboard to dashboard_v1 - v1" duration=3.235686ms 2024-03-21T14:04:12.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.40332833Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-21T14:04:12.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.403826058Z level=info msg="Migration successfully executed" id="create dashboard v2" duration=497.442µs 2024-03-21T14:04:12.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.404144571Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-21T14:04:12.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.404612453Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_org_id - v2" duration=467.478µs 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.404933642Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.405425272Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_org_id_slug - v2" duration=491.467µs 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.405744013Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.405997151Z level=info msg="Migration successfully executed" id="copy dashboard v1 to v2" duration=252.488µs 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.406289783Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.406881954Z level=info msg="Migration successfully executed" id="drop table dashboard_v1" duration=590.549µs 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.407209592Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.4072511Z level=info msg="Migration successfully executed" id="alter dashboard.data to mediumtext v1" duration=41.782µs 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.407609584Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.408704784Z level=info msg="Migration successfully executed" id="Add column updated_by in dashboard - v2" duration=1.094648ms 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.409024847Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.410105672Z level=info msg="Migration successfully executed" id="Add column created_by in dashboard - v2" duration=1.080417ms 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.410406255Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.411468251Z level=info msg="Migration successfully executed" id="Add column gnetId in dashboard" duration=1.061688ms 2024-03-21T14:04:12.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.411782879Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.412291803Z level=info msg="Migration successfully executed" id="Add index for gnetId in dashboard" duration=508.644µs 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.412602711Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.413674142Z level=info msg="Migration successfully executed" id="Add column plugin_id in dashboard" duration=1.070985ms 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.413981869Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.414455991Z level=info msg="Migration successfully executed" id="Add index for plugin_id in dashboard" duration=473.819µs 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.414758393Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.415223033Z level=info msg="Migration successfully executed" id="Add index for dashboard_id in dashboard_tag" duration=464.2µs 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.41551979Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.415534354Z level=info msg="Migration successfully executed" id="Update dashboard table charset" duration=15.08µs 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.415899281Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.415942221Z level=info msg="Migration successfully executed" id="Update dashboard_tag table charset" duration=43.584µs 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.41629581Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.417445308Z level=info msg="Migration successfully executed" id="Add column folder_id in dashboard" duration=1.149027ms 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.417748899Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-21T14:04:12.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.418919442Z level=info msg="Migration successfully executed" id="Add column isFolder in dashboard" duration=1.169587ms 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.419232979Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.420388042Z level=info msg="Migration successfully executed" id="Add column has_acl in dashboard" duration=1.155349ms 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.420700224Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.421871332Z level=info msg="Migration successfully executed" id="Add column uid in dashboard" duration=1.170717ms 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.422179981Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.422315743Z level=info msg="Migration successfully executed" id="Update uid column values in dashboard" duration=135.612µs 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.422674244Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.42319407Z level=info msg="Migration successfully executed" id="Add unique index dashboard_org_id_uid" duration=519.384µs 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.423529375Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.424027723Z level=info msg="Migration successfully executed" id="Remove unique index org_id_slug" duration=498.402µs 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.424351515Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.424367334Z level=info msg="Migration successfully executed" id="Update dashboard title length" duration=15.935µs 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.42477149Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-21T14:04:12.473 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.425307407Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_title_folder_id" duration=535.686µs 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.425629461Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.426111859Z level=info msg="Migration successfully executed" id="create dashboard_provisioning" duration=482.027µs 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.426524928Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.429645668Z level=info msg="Migration successfully executed" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" duration=3.118878ms 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.430027866Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.430508195Z level=info msg="Migration successfully executed" id="create dashboard_provisioning v2" duration=480.163µs 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.430852498Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.431354295Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id - v2" duration=501.619µs 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.431679163Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.432181429Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" duration=501.943µs 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.432503812Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-21T14:04:12.474 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.432734689Z level=info msg="Migration successfully executed" id="copy dashboard_provisioning v1 to v2" duration=229.241µs 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.433081791Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.433476379Z level=info msg="Migration successfully executed" id="drop dashboard_provisioning_tmp_qwerty" duration=393.138µs 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.433802515Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.435025791Z level=info msg="Migration successfully executed" id="Add check_sum column" duration=1.223065ms 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.435351546Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.435827494Z level=info msg="Migration successfully executed" id="Add index for dashboard_title" duration=475.82µs 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.436151434Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.436281084Z level=info msg="Migration successfully executed" id="delete tags for deleted dashboards" duration=128.361µs 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.436631356Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.436751722Z level=info msg="Migration successfully executed" id="delete stars for deleted dashboards" duration=120.357µs 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.437124918Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.437612801Z level=info msg="Migration successfully executed" id="Add index for dashboard_is_folder" duration=487.746µs 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.437943231Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.439280028Z level=info msg="Migration successfully executed" id="Add isPublic for dashboard" duration=1.336427ms 2024-03-21T14:04:12.475 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.439634068Z level=info msg="Executing migration" id="create data_source table" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.440264767Z level=info msg="Migration successfully executed" id="create data_source table" duration=630.542µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.440634541Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.441138852Z level=info msg="Migration successfully executed" id="add index data_source.account_id" duration=504.021µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.44146366Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.441967724Z level=info msg="Migration successfully executed" id="add unique index data_source.account_id_name" duration=503.931µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.442298606Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.442771991Z level=info msg="Migration successfully executed" id="drop index IDX_data_source_account_id - v1" duration=473.449µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.443114208Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.443586209Z level=info msg="Migration successfully executed" id="drop index UQE_data_source_account_id_name - v1" duration=471.936µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.443896928Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.447328114Z level=info msg="Migration successfully executed" id="Rename table data_source to data_source_v1 - v1" duration=3.431798ms 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.447703679Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.448379628Z level=info msg="Migration successfully executed" id="create data_source table v2" duration=675.664µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.44871984Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.449258945Z level=info msg="Migration successfully executed" id="create index IDX_data_source_org_id - v2" duration=539.049µs 2024-03-21T14:04:12.476 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.449598012Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.45013082Z level=info msg="Migration successfully executed" id="create index UQE_data_source_org_id_name - v2" duration=532.392µs 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.450477309Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.450862652Z level=info msg="Migration successfully executed" id="Drop old table data_source_v1 #2" duration=385.15µs 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.451205753Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.452581406Z level=info msg="Migration successfully executed" id="Add column with_credentials" duration=1.375339ms 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.452932638Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.45432536Z level=info msg="Migration successfully executed" id="Add secure json data column" duration=1.392487ms 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.454677177Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.454692423Z level=info msg="Migration successfully executed" id="Update data_source table charset" duration=15.861µs 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.455114513Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.455257528Z level=info msg="Migration successfully executed" id="Update initial version to 1" duration=143.027µs 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.45561936Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.457022167Z level=info msg="Migration successfully executed" id="Add read_only data column" duration=1.402301ms 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.457391755Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-21T14:04:12.477 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.457539432Z level=info msg="Migration successfully executed" id="Migrate logging ds to loki ds" duration=147.677µs 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.45793625Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.45807102Z level=info msg="Migration successfully executed" id="Update json_data with nulls" duration=134.832µs 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.458440368Z level=info msg="Executing migration" id="Add uid column" 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.459872108Z level=info msg="Migration successfully executed" id="Add uid column" duration=1.42951ms 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.460263432Z level=info msg="Executing migration" id="Update uid value" 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.460432649Z level=info msg="Migration successfully executed" id="Update uid value" duration=169.504µs 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.46083101Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.461408109Z level=info msg="Migration successfully executed" id="Add unique index datasource_org_id_uid" duration=577.172µs 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.461738236Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.462253588Z level=info msg="Migration successfully executed" id="add unique index datasource_org_id_is_default" duration=515.233µs 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.462596262Z level=info msg="Executing migration" id="create api_key table" 2024-03-21T14:04:12.478 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.463142509Z level=info msg="Migration successfully executed" id="create api_key table" duration=545.79µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.463484903Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.464005681Z level=info msg="Migration successfully executed" id="add index api_key.account_id" duration=520.61µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.464339121Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.464831783Z level=info msg="Migration successfully executed" id="add index api_key.key" duration=492.549µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.465203736Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.465737983Z level=info msg="Migration successfully executed" id="add index api_key.account_id_name" duration=534.029µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.466180585Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.466677831Z level=info msg="Migration successfully executed" id="drop index IDX_api_key_account_id - v1" duration=498.335µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.467003166Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.467492699Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_key - v1" duration=489.426µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.467812108Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.46832948Z level=info msg="Migration successfully executed" id="drop index UQE_api_key_account_id_name - v1" duration=517.157µs 2024-03-21T14:04:12.479 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.468659408Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-21T14:04:12.706 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-mon[33862]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:12.706 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.706 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.707 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.707 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.707 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.472828693Z level=info msg="Migration successfully executed" id="Rename table api_key to api_key_v1 - v1" duration=4.168409ms 2024-03-21T14:04:12.707 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.473178976Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-21T14:04:12.707 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.473680428Z level=info msg="Migration successfully executed" id="create api_key table v2" duration=501.234µs 2024-03-21T14:04:12.707 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.474020638Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-21T14:04:12.707 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.474529775Z level=info msg="Migration successfully executed" id="create index IDX_api_key_org_id - v2" duration=508.993µs 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.474859136Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.475360729Z level=info msg="Migration successfully executed" id="create index UQE_api_key_key - v2" duration=501.333µs 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.475683185Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.476187433Z level=info msg="Migration successfully executed" id="create index UQE_api_key_org_id_name - v2" duration=503.978µs 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.47651905Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.476773192Z level=info msg="Migration successfully executed" id="copy api_key v1 to v2" duration=253.693µs 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.477104791Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.477491511Z level=info msg="Migration successfully executed" id="Drop old table api_key_v1" duration=386.445µs 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.477802857Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.477818606Z level=info msg="Migration successfully executed" id="Update api_key table charset" duration=15.792µs 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.47822774Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.479708478Z level=info msg="Migration successfully executed" id="Add expires to api_key table" duration=1.480243ms 2024-03-21T14:04:12.708 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.480037588Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.481524604Z level=info msg="Migration successfully executed" id="Add service account foreign key" duration=1.485961ms 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.481854683Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.481990075Z level=info msg="Migration successfully executed" id="set service account foreign key to nil if 0" duration=135.419µs 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.482358712Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.483875927Z level=info msg="Migration successfully executed" id="Add last_used_at to api_key table" duration=1.516887ms 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.484211022Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.485713485Z level=info msg="Migration successfully executed" id="Add is_revoked column to api_key table" duration=1.50215ms 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.486078526Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.48658439Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v4" duration=505.438µs 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.486902811Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-21T14:04:12.709 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.487298296Z level=info msg="Migration successfully executed" id="drop table dashboard_snapshot_v4 #1" duration=395.472µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.487678668Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.488254553Z level=info msg="Migration successfully executed" id="create dashboard_snapshot table v5 #2" duration=575.73µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.488617039Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.489154184Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_key - v5" duration=536.907µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.489484246Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.490045458Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_snapshot_delete_key - v5" duration=559.578µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.490400988Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.490929075Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_snapshot_user_id - v5" duration=528.894µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.491266931Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.491313624Z level=info msg="Migration successfully executed" id="alter dashboard_snapshot to mediumtext v2" duration=46.391µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.491727852Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.491743447Z level=info msg="Migration successfully executed" id="Update dashboard_snapshot table charset" duration=15.746µs 2024-03-21T14:04:12.710 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.492145303Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.493733988Z level=info msg="Migration successfully executed" id="Add column external_delete_url to dashboard_snapshots table" duration=1.588347ms 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.494093342Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.495692548Z level=info msg="Migration successfully executed" id="Add encrypted dashboard json column" duration=1.597133ms 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.496073116Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.496118935Z level=info msg="Migration successfully executed" id="Change dashboard_encrypted column to MEDIUMBLOB" duration=45.456µs 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.496533537Z level=info msg="Executing migration" id="create quota table v1" 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.497013393Z level=info msg="Migration successfully executed" id="create quota table v1" duration=479.68µs 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.497365117Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.497873304Z level=info msg="Migration successfully executed" id="create index UQE_quota_org_id_user_id_target - v1" duration=507.818µs 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.498202688Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-21T14:04:12.711 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.498218011Z level=info msg="Migration successfully executed" id="Update quota table charset" duration=15.901µs 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.4986084Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.499121753Z level=info msg="Migration successfully executed" id="create plugin_setting table" duration=513.1µs 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.499474107Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.500005802Z level=info msg="Migration successfully executed" id="create index UQE_plugin_setting_org_id_plugin_id - v1" duration=531.529µs 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.500352334Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.50203415Z level=info msg="Migration successfully executed" id="Add column plugin_version to plugin_settings" duration=1.679949ms 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.502381532Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.502397209Z level=info msg="Migration successfully executed" id="Update plugin_setting table charset" duration=16.446µs 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.502821751Z level=info msg="Executing migration" id="create session table" 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.503419529Z level=info msg="Migration successfully executed" id="create session table" duration=597.5µs 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.503873396Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-21T14:04:12.714 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.503954069Z level=info msg="Migration successfully executed" id="Drop old table playlist table" duration=80.149µs 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.504329463Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.504398115Z level=info msg="Migration successfully executed" id="Drop old table playlist_item table" duration=66.832µs 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.504739278Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.505237477Z level=info msg="Migration successfully executed" id="create playlist table v2" duration=497.8µs 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.505571872Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.506062992Z level=info msg="Migration successfully executed" id="create playlist item table v2" duration=490.718µs 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.506391642Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.506406893Z level=info msg="Migration successfully executed" id="Update playlist table charset" duration=15.598µs 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.50680749Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.506823105Z level=info msg="Migration successfully executed" id="Update playlist_item table charset" duration=16.177µs 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.507234907Z level=info msg="Executing migration" id="Add playlist column created_at" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.509060282Z level=info msg="Migration successfully executed" id="Add playlist column created_at" duration=1.824931ms 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.509387059Z level=info msg="Executing migration" id="Add playlist column updated_at" 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.51119845Z level=info msg="Migration successfully executed" id="Add playlist column updated_at" duration=1.81108ms 2024-03-21T14:04:12.715 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.511555578Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.511626307Z level=info msg="Migration successfully executed" id="drop preferences table v2" duration=70.331µs 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.512018837Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.512096267Z level=info msg="Migration successfully executed" id="drop preferences table v3" duration=77.869µs 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.512476444Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.513005414Z level=info msg="Migration successfully executed" id="create preferences table v3" duration=530.032µs 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.51332139Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.513339939Z level=info msg="Migration successfully executed" id="Update preferences table charset" duration=18.556µs 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.51369593Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.515525219Z level=info msg="Migration successfully executed" id="Add column team_id in preferences" duration=1.828952ms 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.515836041Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.515951025Z level=info msg="Migration successfully executed" id="Update team_id column values in preferences" duration=114.948µs 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.516307552Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.518247504Z level=info msg="Migration successfully executed" id="Add column week_start in preferences" duration=1.939638ms 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.518576718Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-21T14:04:12.716 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.520397802Z level=info msg="Migration successfully executed" id="Add column preferences.json_data" duration=1.820733ms 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.52071727Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.520766136Z level=info msg="Migration successfully executed" id="alter preferences.json_data to mediumtext v1" duration=49.076µs 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.521192186Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.521810425Z level=info msg="Migration successfully executed" id="Add preferences index org_id" duration=618.347µs 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.522170169Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.52272312Z level=info msg="Migration successfully executed" id="Add preferences index user_id" duration=552.851µs 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.52307164Z level=info msg="Executing migration" id="create alert table v1" 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.523788273Z level=info msg="Migration successfully executed" id="create alert table v1" duration=716.367µs 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.524146717Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.524739918Z level=info msg="Migration successfully executed" id="add index alert org_id & id " duration=592.957µs 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.525095149Z level=info msg="Executing migration" id="add index alert state" 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.525657155Z level=info msg="Migration successfully executed" id="add index alert state" duration=561.992µs 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.526007041Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-21T14:04:12.717 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.526561995Z level=info msg="Migration successfully executed" id="add index alert dashboard_id" duration=553.419µs 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.526904235Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.527363152Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v1" duration=458.636µs 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.527688794Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.528251621Z level=info msg="Migration successfully executed" id="Add unique index alert_rule_tag.alert_id_tag_id" duration=562.555µs 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.528582901Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.529123244Z level=info msg="Migration successfully executed" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" duration=540.206µs 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.529439286Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.535045184Z level=info msg="Migration successfully executed" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" duration=5.60466ms 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.535405745Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.535881028Z level=info msg="Migration successfully executed" id="Create alert_rule_tag table v2" duration=475.203µs 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.536215387Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.536753695Z level=info msg="Migration successfully executed" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" duration=538.043µs 2024-03-21T14:04:12.718 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.537093839Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.537304638Z level=info msg="Migration successfully executed" id="copy alert_rule_tag v1 to v2" duration=210.665µs 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.537607191Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.538005Z level=info msg="Migration successfully executed" id="drop table alert_rule_tag_v1" duration=397.726µs 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.538330185Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.538826144Z level=info msg="Migration successfully executed" id="create alert_notification table v1" duration=495.618µs 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.539153058Z level=info msg="Executing migration" id="Add column is_default" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.541369591Z level=info msg="Migration successfully executed" id="Add column is_default" duration=2.216019ms 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.541723542Z level=info msg="Executing migration" id="Add column frequency" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.543875914Z level=info msg="Migration successfully executed" id="Add column frequency" duration=2.151882ms 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.544235359Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.546505585Z level=info msg="Migration successfully executed" id="Add column send_reminder" duration=2.27003ms 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.546847592Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-21T14:04:12.719 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.548929528Z level=info msg="Migration successfully executed" id="Add column disable_resolve_message" duration=2.081698ms 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.549256548Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.549805201Z level=info msg="Migration successfully executed" id="add index alert_notification org_id & name" duration=549.522µs 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.550172484Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.550188492Z level=info msg="Migration successfully executed" id="Update alert table charset" duration=16.701µs 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.550573611Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.550588249Z level=info msg="Migration successfully executed" id="Update alert_notification table charset" duration=15.436µs 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.550952413Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.551446992Z level=info msg="Migration successfully executed" id="create notification_journal table v1" duration=494.426µs 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.551791824Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.552355648Z level=info msg="Migration successfully executed" id="add index notification_journal org_id & alert_id & notifier_id" duration=565.221µs 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.552696935Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.553233307Z level=info msg="Migration successfully executed" id="drop alert_notification_journal" duration=535.929µs 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.55356929Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-21T14:04:12.720 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.55411388Z level=info msg="Migration successfully executed" id="create alert_notification_state table v1" duration=542.985µs 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.554518829Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.555071637Z level=info msg="Migration successfully executed" id="add index alert_notification_state org_id & alert_id & notifier_id" duration=552.546µs 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.555393861Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.557546305Z level=info msg="Migration successfully executed" id="Add for to alert table" duration=2.151891ms 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.557867927Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.56008044Z level=info msg="Migration successfully executed" id="Add column uid in alert_notification" duration=2.212015ms 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.560403792Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.560540942Z level=info msg="Migration successfully executed" id="Update uid column values in alert_notification" duration=137.15µs 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.560896641Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.561436873Z level=info msg="Migration successfully executed" id="Add unique index alert_notification_org_id_uid" duration=539.908µs 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.561762485Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.562289884Z level=info msg="Migration successfully executed" id="Remove unique index org_id_name" duration=527.281µs 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.562604145Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.564744403Z level=info msg="Migration successfully executed" id="Add column secure_settings in alert_notification" duration=2.141486ms 2024-03-21T14:04:12.721 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.565087695Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.565133362Z level=info msg="Migration successfully executed" id="alter alert.settings to mediumtext" duration=45.406µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.565532464Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.566062849Z level=info msg="Migration successfully executed" id="Add non-unique index alert_notification_state_alert_id" duration=530.268µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.566381468Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.566977309Z level=info msg="Migration successfully executed" id="Add non-unique index alert_rule_tag_alert_id" duration=595.499µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.567315032Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.56738851Z level=info msg="Migration successfully executed" id="Drop old annotation table v4" duration=73.075µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.567752252Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.568358186Z level=info msg="Migration successfully executed" id="create annotation table v5" duration=605.864µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.568704785Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.56924724Z level=info msg="Migration successfully executed" id="add index annotation 0 v3" duration=542.175µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.569573164Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.570111625Z level=info msg="Migration successfully executed" id="add index annotation 1 v3" duration=538.149µs 2024-03-21T14:04:12.722 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.570439289Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.570988327Z level=info msg="Migration successfully executed" id="add index annotation 2 v3" duration=548.872µs 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.571316541Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.571928308Z level=info msg="Migration successfully executed" id="add index annotation 3 v3" duration=611.561µs 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.572269068Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.572865448Z level=info msg="Migration successfully executed" id="add index annotation 4 v3" duration=596.029µs 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.573220759Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.573238498Z level=info msg="Migration successfully executed" id="Update annotation table charset" duration=18.056µs 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.573627713Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.576096807Z level=info msg="Migration successfully executed" id="Add column region_id to annotation table" duration=2.468689ms 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.576445299Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.576990082Z level=info msg="Migration successfully executed" id="Drop category_id index" duration=544.63µs 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.577305874Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.579589865Z level=info msg="Migration successfully executed" id="Add column tags to annotation table" duration=2.281957ms 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.579958356Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-21T14:04:12.723 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.580416953Z level=info msg="Migration successfully executed" id="Create annotation_tag table v2" duration=458.902µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.58073401Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.5813091Z level=info msg="Migration successfully executed" id="Add unique index annotation_tag.annotation_id_tag_id" duration=574.872µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.58165428Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.582204417Z level=info msg="Migration successfully executed" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" duration=550.071µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.582527991Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.58896911Z level=info msg="Migration successfully executed" id="Rename table annotation_tag to annotation_tag_v2 - v2" duration=6.438733ms 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.589365741Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.589860439Z level=info msg="Migration successfully executed" id="Create annotation_tag table v3" duration=494.782µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.590217503Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.590775372Z level=info msg="Migration successfully executed" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" duration=557.783µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.591162799Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.591382271Z level=info msg="Migration successfully executed" id="copy annotation_tag v2 to v3" duration=219.768µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.591681912Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.592065815Z level=info msg="Migration successfully executed" id="drop table annotation_tag_v2" duration=383.613µs 2024-03-21T14:04:12.724 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.592353411Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.592464955Z level=info msg="Migration successfully executed" id="Update alert annotations and set TEXT to empty" duration=111.595µs 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.592788439Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.595335307Z level=info msg="Migration successfully executed" id="Add created time to annotation table" duration=2.542176ms 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.595665618Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.59817492Z level=info msg="Migration successfully executed" id="Add updated time to annotation table" duration=2.508822ms 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.598497312Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.599068444Z level=info msg="Migration successfully executed" id="Add index for created in annotation table" duration=571.143µs 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.599371167Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.599892098Z level=info msg="Migration successfully executed" id="Add index for updated in annotation table" duration=520.697µs 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.600210864Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.600360431Z level=info msg="Migration successfully executed" id="Convert existing annotations from seconds to milliseconds" duration=149.555µs 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.600723883Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.603315049Z level=info msg="Migration successfully executed" id="Add epoch_end column" duration=2.589927ms 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.6037756Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-21T14:04:12.725 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.604363474Z level=info msg="Migration successfully executed" id="Add index for epoch_end" duration=587.906µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.604684099Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.604801525Z level=info msg="Migration successfully executed" id="Make epoch_end the same as epoch" duration=117.462µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.605150597Z level=info msg="Executing migration" id="Move region to single row" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.605370679Z level=info msg="Migration successfully executed" id="Move region to single row" duration=219.93µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.605707205Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.606266463Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch from annotation table" duration=558.963µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.606567714Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.607101256Z level=info msg="Migration successfully executed" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" duration=533.285µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.607408062Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.60798214Z level=info msg="Migration successfully executed" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" duration=573.807µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.608292392Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.608825835Z level=info msg="Migration successfully executed" id="Add index for org_id_epoch_end_epoch on annotation table" duration=533.124µs 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.609137959Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-21T14:04:12.726 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.609653199Z level=info msg="Migration successfully executed" id="Remove index org_id_epoch_epoch_end from annotation table" duration=517.01µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.609961644Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.610483655Z level=info msg="Migration successfully executed" id="Add index for alert_id on annotation table" duration=521.887µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.610777435Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.610824826Z level=info msg="Migration successfully executed" id="Increase tags column to length 4096" duration=47.449µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.611228175Z level=info msg="Executing migration" id="create test_data table" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.611752114Z level=info msg="Migration successfully executed" id="create test_data table" duration=521.903µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.61207515Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.612584177Z level=info msg="Migration successfully executed" id="create dashboard_version table v1" duration=508.726µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.612975494Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.613497Z level=info msg="Migration successfully executed" id="add index dashboard_version.dashboard_id" duration=521.325µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.613803971Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.614375471Z level=info msg="Migration successfully executed" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" duration=571.345µs 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.61469858Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-21T14:04:12.727 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.614816059Z level=info msg="Migration successfully executed" id="Set dashboard version to 1 where 0" duration=117.411µs 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.615162932Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.615408141Z level=info msg="Migration successfully executed" id="save existing dashboard data in dashboard_version table v1" duration=245.083µs 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.615687663Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.615733625Z level=info msg="Migration successfully executed" id="alter dashboard_version.data to mediumtext v1" duration=46.126µs 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.616115532Z level=info msg="Executing migration" id="create team table" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.616583544Z level=info msg="Migration successfully executed" id="create team table" duration=467.963µs 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.616889038Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.617494954Z level=info msg="Migration successfully executed" id="add index team.org_id" duration=605.627µs 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.617817737Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.618383325Z level=info msg="Migration successfully executed" id="add unique index team_org_id_name" duration=562.465µs 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.618695575Z level=info msg="Executing migration" id="Add column uid in team" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.621295141Z level=info msg="Migration successfully executed" id="Add column uid in team" duration=2.600624ms 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.621643222Z level=info msg="Executing migration" id="Update uid column values in team" 2024-03-21T14:04:12.728 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.621767894Z level=info msg="Migration successfully executed" id="Update uid column values in team" duration=124.119µs 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.622133011Z level=info msg="Executing migration" id="Add unique index team_org_id_uid" 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.622700503Z level=info msg="Migration successfully executed" id="Add unique index team_org_id_uid" duration=567.32µs 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.623019648Z level=info msg="Executing migration" id="create team member table" 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.623514062Z level=info msg="Migration successfully executed" id="create team member table" duration=494.174µs 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.623818319Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.624367639Z level=info msg="Migration successfully executed" id="add index team_member.org_id" duration=548.832µs 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.624670788Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.625277055Z level=info msg="Migration successfully executed" id="add unique index team_member_org_id_team_id_user_id" duration=606.137µs 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.625582884Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.626115608Z level=info msg="Migration successfully executed" id="add index team_member.team_id" duration=532.604µs 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.626414001Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.629172601Z level=info msg="Migration successfully executed" id="Add column email to team table" duration=2.758269ms 2024-03-21T14:04:12.729 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.62946914Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.632182983Z level=info msg="Migration successfully executed" id="Add column external to team_member table" duration=2.713487ms 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.63247551Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.635119947Z level=info msg="Migration successfully executed" id="Add column permission to team_member table" duration=2.643921ms 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.635441594Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.636050317Z level=info msg="Migration successfully executed" id="create dashboard acl table" duration=608.46µs 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.636380264Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.636963527Z level=info msg="Migration successfully executed" id="add index dashboard_acl_dashboard_id" duration=582.924µs 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.637261644Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.637860872Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_user_id" duration=598.88µs 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.638216889Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-21T14:04:12.730 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.638770957Z level=info msg="Migration successfully executed" id="add unique index dashboard_acl_dashboard_id_team_id" duration=553.929µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.639084409Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.639613751Z level=info msg="Migration successfully executed" id="add index dashboard_acl_user_id" duration=529.252µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.639903638Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.640455032Z level=info msg="Migration successfully executed" id="add index dashboard_acl_team_id" duration=551.053µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.640755133Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.641322247Z level=info msg="Migration successfully executed" id="add index dashboard_acl_org_id_role" duration=566.706µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.641615741Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.642182775Z level=info msg="Migration successfully executed" id="add index dashboard_permission" duration=566.516µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.642505193Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.642836722Z level=info msg="Migration successfully executed" id="save default acl rules in dashboard_acl table" duration=331.358µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.643167323Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.643318215Z level=info msg="Migration successfully executed" id="delete acl rules for deleted dashboards and folders" duration=150.005µs 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.643656548Z level=info msg="Executing migration" id="create tag table" 2024-03-21T14:04:12.731 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.644149398Z level=info msg="Migration successfully executed" id="create tag table" duration=492.626µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.644460747Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.645022819Z level=info msg="Migration successfully executed" id="add index tag.key_value" duration=561.755µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.645333293Z level=info msg="Executing migration" id="create login attempt table" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.64579924Z level=info msg="Migration successfully executed" id="create login attempt table" duration=465.77µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.646125884Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.646706281Z level=info msg="Migration successfully executed" id="add index login_attempt.username" duration=577.289µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.647047604Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.647634842Z level=info msg="Migration successfully executed" id="drop index IDX_login_attempt_username - v1" duration=586.922µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.647948821Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.656315716Z level=info msg="Migration successfully executed" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" duration=8.36589ms 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.656672715Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.65716622Z level=info msg="Migration successfully executed" id="create login_attempt v2" duration=491.964µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.657508557Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.658069835Z level=info msg="Migration successfully executed" id="create index IDX_login_attempt_username - v2" duration=560.758µs 2024-03-21T14:04:12.732 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.65838782Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.658593373Z level=info msg="Migration successfully executed" id="copy login_attempt v1 to v2" duration=205.299µs 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.658877787Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.659286591Z level=info msg="Migration successfully executed" id="drop login_attempt_tmp_qwerty" duration=408.638µs 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.659585154Z level=info msg="Executing migration" id="create user auth table" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.660059951Z level=info msg="Migration successfully executed" id="create user auth table" duration=474.498µs 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.660363199Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.660953502Z level=info msg="Migration successfully executed" id="create index IDX_user_auth_auth_module_auth_id - v1" duration=589.586µs 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.661284338Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.661331547Z level=info msg="Migration successfully executed" id="alter user_auth.auth_id to length 190" duration=49.39µs 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.66171557Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.664910291Z level=info msg="Migration successfully executed" id="Add OAuth access token to user_auth" duration=3.189208ms 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.665207197Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.668067107Z level=info msg="Migration successfully executed" id="Add OAuth refresh token to user_auth" duration=2.859786ms 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.668380555Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-21T14:04:12.733 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.671261479Z level=info msg="Migration successfully executed" id="Add OAuth token type to user_auth" duration=2.880737ms 2024-03-21T14:04:12.734 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.671574471Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-21T14:04:12.734 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.674518714Z level=info msg="Migration successfully executed" id="Add OAuth expiry to user_auth" duration=2.943788ms 2024-03-21T14:04:12.734 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.674834958Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-21T14:04:12.734 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.675390306Z level=info msg="Migration successfully executed" id="Add index to user_id column in user_auth" duration=555.047µs 2024-03-21T14:04:12.734 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.675722291Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.678655956Z level=info msg="Migration successfully executed" id="Add OAuth ID token to user_auth" duration=2.932709ms 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.679007179Z level=info msg="Executing migration" id="create server_lock table" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.679534812Z level=info msg="Migration successfully executed" id="create server_lock table" duration=527.742µs 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.679873195Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.680436698Z level=info msg="Migration successfully executed" id="add index server_lock.operation_uid" duration=560.688µs 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.68073868Z level=info msg="Executing migration" id="create user auth token table" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.681270785Z level=info msg="Migration successfully executed" id="create user auth token table" duration=531.825µs 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.681566733Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.682146952Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.auth_token" duration=579.878µs 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.682496016Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.683068866Z level=info msg="Migration successfully executed" id="add unique index user_auth_token.prev_auth_token" duration=573.953µs 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.683380416Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-21T14:04:12.735 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.684005911Z level=info msg="Migration successfully executed" id="add index user_auth_token.user_id" duration=625.156µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.684325508Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.687535251Z level=info msg="Migration successfully executed" id="Add revoked_at to the user auth token" duration=3.20782ms 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.687900277Z level=info msg="Executing migration" id="add index user_auth_token.revoked_at" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.688517066Z level=info msg="Migration successfully executed" id="add index user_auth_token.revoked_at" duration=616.838µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.688841659Z level=info msg="Executing migration" id="create cache_data table" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.689415936Z level=info msg="Migration successfully executed" id="create cache_data table" duration=574.651µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.689729991Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.690359567Z level=info msg="Migration successfully executed" id="add unique index cache_data.cache_key" duration=629.123µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.690693749Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.691296679Z level=info msg="Migration successfully executed" id="create short_url table v1" duration=603.204µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.69163808Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.692245709Z level=info msg="Migration successfully executed" id="add index short_url.org_id-uid" duration=607.456µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.692543758Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.69258398Z level=info msg="Migration successfully executed" id="alter table short_url alter column created_by type to bigint" duration=40.489µs 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.692957289Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-21T14:04:12.736 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.693015098Z level=info msg="Migration successfully executed" id="delete alert_definition table" duration=57.779µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.693359242Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.693920722Z level=info msg="Migration successfully executed" id="recreate alert_definition table" duration=561.314µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.694237958Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.694896428Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and title columns" duration=658.126µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.695270715Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.695897712Z level=info msg="Migration successfully executed" id="add index in alert_definition on org_id and uid columns" duration=626.964µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.696228409Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.696267468Z level=info msg="Migration successfully executed" id="alter alert_definition table data column to mediumtext in mysql" duration=39.422µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.696618668Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.697215139Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and title columns" duration=596.206µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.697505404Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.698062607Z level=info msg="Migration successfully executed" id="drop index in alert_definition on org_id and uid columns" duration=557.043µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.698354617Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.699087712Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and title columns" duration=732.653µs 2024-03-21T14:04:12.737 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.699412181Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.700039203Z level=info msg="Migration successfully executed" id="add unique index in alert_definition on org_id and uid columns" duration=628.026µs 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.700358775Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.703649591Z level=info msg="Migration successfully executed" id="Add column paused in alert_definition" duration=3.29056ms 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.703992315Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.704608211Z level=info msg="Migration successfully executed" id="drop alert_definition table" duration=615.556µs 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.704951584Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.705011363Z level=info msg="Migration successfully executed" id="delete alert_definition_version table" duration=57.744µs 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.7053649Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.705958064Z level=info msg="Migration successfully executed" id="recreate alert_definition_version table" duration=592.77µs 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.706274368Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.706894143Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_id and version columns" duration=619.576µs 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.707214516Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-21T14:04:12.738 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.707962624Z level=info msg="Migration successfully executed" id="add index in alert_definition_version table on alert_definition_uid and version columns" duration=749.407µs 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.708298196Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.708338322Z level=info msg="Migration successfully executed" id="alter alert_definition_version table data column to mediumtext in mysql" duration=40.257µs 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.708724612Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.709453211Z level=info msg="Migration successfully executed" id="drop alert_definition_version table" duration=728.465µs 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.709809271Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.710512587Z level=info msg="Migration successfully executed" id="create alert_instance table" duration=703.536µs 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.710833963Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.71151283Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" duration=677.664µs 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.711830611Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.712480875Z level=info msg="Migration successfully executed" id="add index in alert_instance table on def_org_id, current_state columns" duration=649.999µs 2024-03-21T14:04:12.739 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.712819715Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[29576]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[40510]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:12 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.716686505Z level=info msg="Migration successfully executed" id="add column current_state_end to alert_instance" duration=3.865051ms 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.71704978Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.717663366Z level=info msg="Migration successfully executed" id="remove index def_org_id, def_uid, current_state on alert_instance" duration=614.137µs 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.717974098Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.718562436Z level=info msg="Migration successfully executed" id="remove index def_org_id, current_state on alert_instance" duration=588.151µs 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.7188558Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-21T14:04:12.962 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.736122293Z level=info msg="Migration successfully executed" id="rename def_org_id to rule_org_id in alert_instance" duration=17.263681ms 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.736478446Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.752965934Z level=info msg="Migration successfully executed" id="rename def_uid to rule_uid in alert_instance" duration=16.484628ms 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.75342314Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.754101037Z level=info msg="Migration successfully executed" id="add index rule_org_id, rule_uid, current_state on alert_instance" duration=677.958µs 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.754410503Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.755004087Z level=info msg="Migration successfully executed" id="add index rule_org_id, current_state on alert_instance" duration=593.197µs 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.755340734Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.758591592Z level=info msg="Migration successfully executed" id="add current_reason column related to current_state" duration=3.251858ms 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.758929323Z level=info msg="Executing migration" id="add result_fingerprint column to alert_instance" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.762262036Z level=info msg="Migration successfully executed" id="add result_fingerprint column to alert_instance" duration=3.331709ms 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.762639805Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.763291971Z level=info msg="Migration successfully executed" id="create alert_rule table" duration=652.29µs 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.763630487Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-21T14:04:12.963 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.76428452Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and title columns" duration=653.643µs 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.764672753Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.765515896Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id and uid columns" duration=843.993µs 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.765955327Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.766977231Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" duration=1.021348ms 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.767402794Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.767454312Z level=info msg="Migration successfully executed" id="alter alert_rule table data column to mediumtext in mysql" duration=51.787µs 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.767941728Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.771738172Z level=info msg="Migration successfully executed" id="add column for to alert_rule" duration=3.796676ms 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.772063004Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.775406445Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule" duration=3.343164ms 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.775725428Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-21T14:04:12.964 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.779078072Z level=info msg="Migration successfully executed" id="add column labels to alert_rule" duration=3.352299ms 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.779393586Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.780018186Z level=info msg="Migration successfully executed" id="remove unique index from alert_rule on org_id, title columns" duration=623.889µs 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.780428931Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.781221078Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, namespase_uid and title columns" duration=792.381µs 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.781516858Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.784968545Z level=info msg="Migration successfully executed" id="add dashboard_uid column to alert_rule" duration=3.451135ms 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.785279501Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.788616761Z level=info msg="Migration successfully executed" id="add panel_id column to alert_rule" duration=3.339254ms 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.788930482Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-21T14:04:12.966 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.789531217Z level=info msg="Migration successfully executed" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" duration=600.461µs 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.789830326Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.793196183Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule" duration=3.365399ms 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.793507077Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.796874195Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule table" duration=3.366774ms 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.797222197Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.797260863Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule table" duration=39.028µs 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.797620246Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.798325246Z level=info msg="Migration successfully executed" id="create alert_rule_version table" duration=704.583µs 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.798651265Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.799298458Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" duration=646.891µs 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.799597826Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.800299942Z level=info msg="Migration successfully executed" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" duration=701.766µs 2024-03-21T14:04:12.967 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.800630422Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.800668042Z level=info msg="Migration successfully executed" id="alter alert_rule_version table data column to mediumtext in mysql" duration=37.907µs 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.801027817Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.804635918Z level=info msg="Migration successfully executed" id="add column for to alert_rule_version" duration=3.607544ms 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.80496045Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.808475107Z level=info msg="Migration successfully executed" id="add column annotations to alert_rule_version" duration=3.51436ms 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.808772271Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.812275565Z level=info msg="Migration successfully executed" id="add column labels to alert_rule_version" duration=3.503222ms 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.812590991Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.816100466Z level=info msg="Migration successfully executed" id="add rule_group_idx column to alert_rule_version" duration=3.508588ms 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.816408996Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.81995761Z level=info msg="Migration successfully executed" id="add is_paused column to alert_rule_versions table" duration=3.548052ms 2024-03-21T14:04:12.968 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.820264116Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.820302232Z level=info msg="Migration successfully executed" id="fix is_paused column for alert_rule_version table" duration=38.372µs 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.820651109Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.821139416Z level=info msg="Migration successfully executed" id=create_alert_configuration_table duration=488.109µs 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.82144054Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.825051589Z level=info msg="Migration successfully executed" id="Add column default in alert_configuration" duration=3.610585ms 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.825350674Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.825388487Z level=info msg="Migration successfully executed" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" duration=38.419µs 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.82573214Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.82935521Z level=info msg="Migration successfully executed" id="add column org_id in alert_configuration" duration=3.622461ms 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.829661837Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.830270014Z level=info msg="Migration successfully executed" id="add index in alert_configuration table on org_id column" duration=608.077µs 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.830567015Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-21T14:04:12.969 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.834180605Z level=info msg="Migration successfully executed" id="add configuration_hash column to alert_configuration" duration=3.6133ms 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.834478647Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.834973379Z level=info msg="Migration successfully executed" id=create_ngalert_configuration_table duration=494.543µs 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.835280404Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.835872027Z level=info msg="Migration successfully executed" id="add index in ngalert_configuration on org_id column" duration=593.057µs 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.836179801Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.839879436Z level=info msg="Migration successfully executed" id="add column send_alerts_to in ngalert_configuration" duration=3.698936ms 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.840225637Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.840716095Z level=info msg="Migration successfully executed" id="create provenance_type table" duration=490.285µs 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.841041324Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.84164935Z level=info msg="Migration successfully executed" id="add index to uniquify (record_key, record_type, org_id) columns" duration=606.064µs 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.841959022Z level=info msg="Executing migration" id="create alert_image table" 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.842449119Z level=info msg="Migration successfully executed" id="create alert_image table" duration=489.931µs 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.842753727Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-21T14:04:12.970 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.843342681Z level=info msg="Migration successfully executed" id="add unique index on token to alert_image table" duration=588.799µs 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.843635547Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.843675634Z level=info msg="Migration successfully executed" id="support longer URLs in alert_image table" duration=39.99µs 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.844034956Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.844617586Z level=info msg="Migration successfully executed" id=create_alert_configuration_history_table duration=582.574µs 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.844936178Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.845516365Z level=info msg="Migration successfully executed" id="drop non-unique orgID index on alert_configuration" duration=580.046µs 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.845801229Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.846026991Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.846353865Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.846611424Z level=info msg="Migration successfully executed" id="extract alertmanager configuration history to separate table" duration=257.409µs 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.846892868Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.847487565Z level=info msg="Migration successfully executed" id="add unique index on orgID to alert_configuration" duration=594.496µs 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.847771999Z level=info msg="Executing migration" id="add last_applied column to alert_configuration_history" 2024-03-21T14:04:12.971 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.851549552Z level=info msg="Migration successfully executed" id="add last_applied column to alert_configuration_history" duration=3.777999ms 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.851933061Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.852614112Z level=info msg="Migration successfully executed" id="create library_element table v1" duration=681.031µs 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.852949737Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.853588435Z level=info msg="Migration successfully executed" id="add index library_element org_id-folder_id-name-kind" duration=638.395µs 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.853892069Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.854412336Z level=info msg="Migration successfully executed" id="create library_element_connection table v1" duration=518.309µs 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.854707425Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.85535111Z level=info msg="Migration successfully executed" id="add index library_element_connection element_id-kind-connection_id" duration=643.389µs 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.855665781Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.856281297Z level=info msg="Migration successfully executed" id="add unique index library_element org_id_uid" duration=615.194µs 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.856590847Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-21T14:04:12.972 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.85660585Z level=info msg="Migration successfully executed" id="increase max description length to 2048" duration=15.5µs 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.856953898Z level=info msg="Executing migration" id="alter library_element model to mediumtext" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.85699334Z level=info msg="Migration successfully executed" id="alter library_element model to mediumtext" duration=39.571µs 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.857331374Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.857496979Z level=info msg="Migration successfully executed" id="clone move dashboard alerts to unified alerting" duration=163.59µs 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.857801622Z level=info msg="Executing migration" id="create data_keys table" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.85841008Z level=info msg="Migration successfully executed" id="create data_keys table" duration=608.075µs 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.858729201Z level=info msg="Executing migration" id="create secrets table" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.85925442Z level=info msg="Migration successfully executed" id="create secrets table" duration=525.14µs 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.859566345Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.880838797Z level=info msg="Migration successfully executed" id="rename data_keys name column to id" duration=21.269867ms 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.881231099Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.885289693Z level=info msg="Migration successfully executed" id="add name column into data_keys" duration=4.058015ms 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.88560045Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.885693528Z level=info msg="Migration successfully executed" id="copy data_keys id column values into name" duration=93.272µs 2024-03-21T14:04:12.973 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.886035936Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.907097597Z level=info msg="Migration successfully executed" id="rename data_keys name column to label" duration=21.061142ms 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.90742863Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.928815443Z level=info msg="Migration successfully executed" id="rename data_keys id column back to name" duration=21.385373ms 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.92915763Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.929694521Z level=info msg="Migration successfully executed" id="create kv_store table v1" duration=536.994µs 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.930004347Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.930648924Z level=info msg="Migration successfully executed" id="add index kv_store.org_id-namespace-key" duration=644.246µs 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.930970702Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.931108976Z level=info msg="Migration successfully executed" id="update dashboard_uid and panel_id from existing annotations" duration=138.345µs 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.931452327Z level=info msg="Executing migration" id="create permission table" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.93202454Z level=info msg="Migration successfully executed" id="create permission table" duration=572.122µs 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.932342228Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-21T14:04:12.974 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.932946496Z level=info msg="Migration successfully executed" id="add unique index permission.role_id" duration=604.169µs 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.93324861Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.933871147Z level=info msg="Migration successfully executed" id="add unique index role_id_action_scope" duration=622.434µs 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.934188285Z level=info msg="Executing migration" id="create role table" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.934725661Z level=info msg="Migration successfully executed" id="create role table" duration=537.18µs 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.935034061Z level=info msg="Executing migration" id="add column display_name" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.93942071Z level=info msg="Migration successfully executed" id="add column display_name" duration=4.383952ms 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.939727432Z level=info msg="Executing migration" id="add column group_name" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.943933064Z level=info msg="Migration successfully executed" id="add column group_name" duration=4.203288ms 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.944241427Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.944857614Z level=info msg="Migration successfully executed" id="add index role.org_id" duration=614.611µs 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.945168879Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.945828665Z level=info msg="Migration successfully executed" id="add unique index role_org_id_name" duration=659.67µs 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.946148048Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-21T14:04:12.975 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.946781235Z level=info msg="Migration successfully executed" id="add index role_org_id_uid" duration=632.916µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.947102865Z level=info msg="Executing migration" id="create team role table" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.947625644Z level=info msg="Migration successfully executed" id="create team role table" duration=522.62µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.947949789Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.948595585Z level=info msg="Migration successfully executed" id="add index team_role.org_id" duration=647.043µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.948914266Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.949586326Z level=info msg="Migration successfully executed" id="add unique index team_role_org_id_team_id_role_id" duration=680.305µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.949903087Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.950517459Z level=info msg="Migration successfully executed" id="add index team_role.team_id" duration=614.209µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.950816945Z level=info msg="Executing migration" id="create user role table" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.951366306Z level=info msg="Migration successfully executed" id="create user role table" duration=549.215µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.951702468Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.952344213Z level=info msg="Migration successfully executed" id="add index user_role.org_id" duration=641.764µs 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.95266283Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-21T14:04:12.976 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.953302022Z level=info msg="Migration successfully executed" id="add unique index user_role_org_id_user_id_role_id" duration=639.025µs 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.953666116Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.954287685Z level=info msg="Migration successfully executed" id="add index user_role.user_id" duration=619.901µs 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.954593533Z level=info msg="Executing migration" id="create builtin role table" 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.955121121Z level=info msg="Migration successfully executed" id="create builtin role table" duration=527.325µs 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.955424805Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.956061571Z level=info msg="Migration successfully executed" id="add index builtin_role.role_id" duration=636.56µs 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.956363148Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.957006417Z level=info msg="Migration successfully executed" id="add index builtin_role.name" duration=642.85µs 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.957306821Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.961885535Z level=info msg="Migration successfully executed" id="Add column org_id to builtin_role table" duration=4.57855ms 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.962272303Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.96296411Z level=info msg="Migration successfully executed" id="add index builtin_role.org_id" duration=692.294µs 2024-03-21T14:04:12.977 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.963321193Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.964039582Z level=info msg="Migration successfully executed" id="add unique index builtin_role_org_id_role_id_role" duration=718.396µs 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.964354288Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.965064314Z level=info msg="Migration successfully executed" id="Remove unique index role_org_id_uid" duration=710.496µs 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.965385479Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.966086988Z level=info msg="Migration successfully executed" id="add unique index role.uid" duration=701.403µs 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.966407487Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-21T14:04:12.978 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.966923683Z level=info msg="Migration successfully executed" id="create seed assignment table" duration=515.796µs 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.967268163Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.967955523Z level=info msg="Migration successfully executed" id="add unique index builtin_role_role_name" duration=687.05µs 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.968267075Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.973052981Z level=info msg="Migration successfully executed" id="add column hidden to role table" duration=4.784644ms 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.973381784Z level=info msg="Executing migration" id="permission kind migration" 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.977911465Z level=info msg="Migration successfully executed" id="permission kind migration" duration=4.523191ms 2024-03-21T14:04:13.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.978241589Z level=info msg="Executing migration" id="permission attribute migration" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.982724468Z level=info msg="Migration successfully executed" id="permission attribute migration" duration=4.482257ms 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.983047292Z level=info msg="Executing migration" id="permission identifier migration" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.987756104Z level=info msg="Migration successfully executed" id="permission identifier migration" duration=4.70779ms 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.988121643Z level=info msg="Executing migration" id="add permission identifier index" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.988877281Z level=info msg="Migration successfully executed" id="add permission identifier index" duration=755.338µs 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.989244076Z level=info msg="Executing migration" id="add permission action scope role_id index" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.98997296Z level=info msg="Migration successfully executed" id="add permission action scope role_id index" duration=728.432µs 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.990297344Z level=info msg="Executing migration" id="remove permission role_id action scope index" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.990952088Z level=info msg="Migration successfully executed" id="remove permission role_id action scope index" duration=653.02µs 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.991257421Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.991837218Z level=info msg="Migration successfully executed" id="create query_history table v1" duration=580.124µs 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.992148044Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.99280902Z level=info msg="Migration successfully executed" id="add index query_history.org_id-created_by-datasource_uid" duration=661.234µs 2024-03-21T14:04:13.213 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.993124534Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.993162599Z level=info msg="Migration successfully executed" id="alter table query_history alter column created_by type to bigint" duration=38.435µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.993522043Z level=info msg="Executing migration" id="rbac disabled migrator" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.993545855Z level=info msg="Migration successfully executed" id="rbac disabled migrator" duration=23.618µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.993862785Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.994141083Z level=info msg="Migration successfully executed" id="teams permissions migration" duration=278.764µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.994482616Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.994821739Z level=info msg="Migration successfully executed" id="dashboard permissions" duration=337.938µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.995176227Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.995574524Z level=info msg="Migration successfully executed" id="dashboard permissions uid scopes" duration=398.128µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.995946876Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.996082712Z level=info msg="Migration successfully executed" id="drop managed folder create actions" duration=136.492µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.996417653Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.99670963Z level=info msg="Migration successfully executed" id="alerting notification permissions" duration=291.798µs 2024-03-21T14:04:13.214 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.99707761Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.997576719Z level=info msg="Migration successfully executed" id="create query_history_star table v1" duration=499.393µs 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.997881199Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.998548487Z level=info msg="Migration successfully executed" id="add index query_history.user_id-query_uid" duration=666.716µs 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:12 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:12.998866158Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.00407614Z level=info msg="Migration successfully executed" id="add column org_id in query_history_star" duration=5.208168ms 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.004466202Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.004507112Z level=info msg="Migration successfully executed" id="alter table query_history_star_mig column user_id type to bigint" duration=41.238µs 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.004900591Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-21T14:04:13.216 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.005606115Z level=info msg="Migration successfully executed" id="create correlation table v1" duration=705.673µs 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.005966512Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.006640617Z level=info msg="Migration successfully executed" id="add index correlations.uid" duration=673.854µs 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.006969366Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.007621544Z level=info msg="Migration successfully executed" id="add index correlations.source_uid" duration=652.119µs 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.007932384Z level=info msg="Executing migration" id="add correlation config column" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.0127952Z level=info msg="Migration successfully executed" id="add correlation config column" duration=4.862197ms 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.013116754Z level=info msg="Executing migration" id="drop index IDX_correlation_uid - v1" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.013774528Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_uid - v1" duration=658.967µs 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.014188087Z level=info msg="Executing migration" id="drop index IDX_correlation_source_uid - v1" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.014837011Z level=info msg="Migration successfully executed" id="drop index IDX_correlation_source_uid - v1" duration=649.469µs 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.01516409Z level=info msg="Executing migration" id="Rename table correlation to correlation_tmp_qwerty - v1" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.028919508Z level=info msg="Migration successfully executed" id="Rename table correlation to correlation_tmp_qwerty - v1" duration=13.73832ms 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.029304122Z level=info msg="Executing migration" id="create correlation v2" 2024-03-21T14:04:13.217 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.030032381Z level=info msg="Migration successfully executed" id="create correlation v2" duration=728.043µs 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.030365898Z level=info msg="Executing migration" id="create index IDX_correlation_uid - v2" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.031049023Z level=info msg="Migration successfully executed" id="create index IDX_correlation_uid - v2" duration=682.618µs 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.031362565Z level=info msg="Executing migration" id="create index IDX_correlation_source_uid - v2" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.032088373Z level=info msg="Migration successfully executed" id="create index IDX_correlation_source_uid - v2" duration=726.988µs 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.032418465Z level=info msg="Executing migration" id="create index IDX_correlation_org_id - v2" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.033106483Z level=info msg="Migration successfully executed" id="create index IDX_correlation_org_id - v2" duration=688.317µs 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.033410835Z level=info msg="Executing migration" id="copy correlation v1 to v2" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.033567986Z level=info msg="Migration successfully executed" id="copy correlation v1 to v2" duration=155.645µs 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.033895209Z level=info msg="Executing migration" id="drop correlation_tmp_qwerty" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.03442353Z level=info msg="Migration successfully executed" id="drop correlation_tmp_qwerty" duration=528.151µs 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.034739951Z level=info msg="Executing migration" id="add provisioning column" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.039420824Z level=info msg="Migration successfully executed" id="add provisioning column" duration=4.680552ms 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.039744004Z level=info msg="Executing migration" id="create entity_events table" 2024-03-21T14:04:13.218 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.040293438Z level=info msg="Migration successfully executed" id="create entity_events table" duration=549.349µs 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.040614765Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.041278034Z level=info msg="Migration successfully executed" id="create dashboard public config v1" duration=663.113µs 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.041603661Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.04186035Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.042206337Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.042456367Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.042786033Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.04330488Z level=info msg="Migration successfully executed" id="Drop old dashboard public config table" duration=520.511µs 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.043617055Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.0442508Z level=info msg="Migration successfully executed" id="recreate dashboard public config v1" duration=635.286µs 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.044638655Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.04567843Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v1" duration=1.039928ms 2024-03-21T14:04:13.219 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.046124834Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.047169378Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" duration=1.044286ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.047627678Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.048636664Z level=info msg="Migration successfully executed" id="drop index UQE_dashboard_public_config_uid - v2" duration=1.007416ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.049066658Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.050087379Z level=info msg="Migration successfully executed" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=1.020511ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.050520327Z level=info msg="Executing migration" id="Drop public config table" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.051325952Z level=info msg="Migration successfully executed" id="Drop public config table" duration=805.428µs 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.05179031Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.052964928Z level=info msg="Migration successfully executed" id="Recreate dashboard public config v2" duration=1.174102ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.053387158Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.054379453Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_uid - v2" duration=1.000072ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.054829578Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.055894687Z level=info msg="Migration successfully executed" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" duration=1.064955ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.056335615Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.057385815Z level=info msg="Migration successfully executed" id="create index UQE_dashboard_public_config_access_token - v2" duration=1.049913ms 2024-03-21T14:04:13.220 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.057833804Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.084224296Z level=info msg="Migration successfully executed" id="Rename table dashboard_public_config to dashboard_public - v2" duration=26.390102ms 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.084647437Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.089655761Z level=info msg="Migration successfully executed" id="add annotations_enabled column" duration=5.008829ms 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.089987416Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.094809197Z level=info msg="Migration successfully executed" id="add time_selection_enabled column" duration=4.821691ms 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.095131304Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.095271738Z level=info msg="Migration successfully executed" id="delete orphaned public dashboards" duration=142.743µs 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.095600175Z level=info msg="Executing migration" id="add share column" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.100365702Z level=info msg="Migration successfully executed" id="add share column" duration=4.765257ms 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.100676508Z level=info msg="Executing migration" id="backfill empty share column fields with default of public" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.100800926Z level=info msg="Migration successfully executed" id="backfill empty share column fields with default of public" duration=124.49µs 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.101152467Z level=info msg="Executing migration" id="create file table" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.101725492Z level=info msg="Migration successfully executed" id="create file table" duration=572.906µs 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.102033207Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-21T14:04:13.221 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.102674234Z level=info msg="Migration successfully executed" id="file table idx: path natural pk" duration=642.328µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.102982266Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.10362409Z level=info msg="Migration successfully executed" id="file table idx: parent_folder_path_hash fast folder retrieval" duration=641.709µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.103928974Z level=info msg="Executing migration" id="create file_meta table" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.104409858Z level=info msg="Migration successfully executed" id="create file_meta table" duration=480.786µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.104703858Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.105358827Z level=info msg="Migration successfully executed" id="file table idx: path key" duration=654.727µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.105666069Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.105703141Z level=info msg="Migration successfully executed" id="set path collation in file table" duration=37.358µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.10605288Z level=info msg="Executing migration" id="migrate contents column to mediumblob for MySQL" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.10608972Z level=info msg="Migration successfully executed" id="migrate contents column to mediumblob for MySQL" duration=37.209µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.106422967Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.106721087Z level=info msg="Migration successfully executed" id="managed permissions migration" duration=298.047µs 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.107062151Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-21T14:04:13.222 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.107184698Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions migration" duration=122.288µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.107532147Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.108308375Z level=info msg="Migration successfully executed" id="RBAC action name migrator" duration=777.284µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.108643943Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.113549763Z level=info msg="Migration successfully executed" id="Add UID column to playlist" duration=4.905092ms 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.113847254Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.113962802Z level=info msg="Migration successfully executed" id="Update uid column values in playlist" duration=115.681µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.114298722Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.115048808Z level=info msg="Migration successfully executed" id="Add index for uid in playlist" duration=750.108µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.115362385Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.115599934Z level=info msg="Migration successfully executed" id="update group index for alert rules" duration=237.608µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.115969513Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.1161062Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated migration" duration=136.597µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.11644769Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.116736265Z level=info msg="Migration successfully executed" id="admin only folder/dashboard permission" duration=287.833µs 2024-03-21T14:04:13.223 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.117101118Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.122078948Z level=info msg="Migration successfully executed" id="add action column to seed_assignment" duration=4.979545ms 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.122396515Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.127334311Z level=info msg="Migration successfully executed" id="add scope column to seed_assignment" duration=4.936328ms 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.127651171Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.128303276Z level=info msg="Migration successfully executed" id="remove unique index builtin_role_role_name before nullable update" duration=652.182µs 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.128588663Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.184429851Z level=info msg="Migration successfully executed" id="update seed_assignment role_name column to nullable" duration=55.833965ms 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.184766701Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.185499682Z level=info msg="Migration successfully executed" id="add unique index builtin_role_name back" duration=731.444µs 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.185802722Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.186514283Z level=info msg="Migration successfully executed" id="add unique index builtin_role_action_scope" duration=711.776µs 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.186817499Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.203255827Z level=info msg="Migration successfully executed" id="add primary key to seed_assigment" duration=16.437503ms 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.203599356Z level=info msg="Executing migration" id="add origin column to seed_assignment" 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.208587845Z level=info msg="Migration successfully executed" id="add origin column to seed_assignment" duration=4.988486ms 2024-03-21T14:04:13.224 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.208920472Z level=info msg="Executing migration" id="add origin to plugin seed_assignment" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.209111731Z level=info msg="Migration successfully executed" id="add origin to plugin seed_assignment" duration=191.544µs 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.209451593Z level=info msg="Executing migration" id="prevent seeding OnCall access" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.20955692Z level=info msg="Migration successfully executed" id="prevent seeding OnCall access" duration=103.927µs 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.209862435Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.210007289Z level=info msg="Migration successfully executed" id="managed folder permissions alert actions repeated fixed migration" duration=143.511µs 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.210344114Z level=info msg="Executing migration" id="managed folder permissions library panel actions migration" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.210465882Z level=info msg="Migration successfully executed" id="managed folder permissions library panel actions migration" duration=121.816µs 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.210840645Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.210984709Z level=info msg="Migration successfully executed" id="migrate external alertmanagers to datsourcse" duration=144.514µs 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.211341161Z level=info msg="Executing migration" id="create folder table" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.211898521Z level=info msg="Migration successfully executed" id="create folder table" duration=557.684µs 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.212204888Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.213320922Z level=info msg="Migration successfully executed" id="Add index for parent_uid" duration=1.115346ms 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.213744951Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-21T14:04:13.225 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.214795996Z level=info msg="Migration successfully executed" id="Add unique index for folder.uid and folder.org_id" duration=1.050558ms 2024-03-21T14:04:13.226 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.215209527Z level=info msg="Executing migration" id="Update folder title length" 2024-03-21T14:04:13.226 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.215234244Z level=info msg="Migration successfully executed" id="Update folder title length" duration=25.43µs 2024-03-21T14:04:13.226 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.215703952Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-21T14:04:13.464 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.216758409Z level=info msg="Migration successfully executed" id="Add unique index for folder.title and folder.parent_uid" duration=1.053961ms 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.217179311Z level=info msg="Executing migration" id="Remove unique index for folder.title and folder.parent_uid" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.218185935Z level=info msg="Migration successfully executed" id="Remove unique index for folder.title and folder.parent_uid" duration=1.006404ms 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.218584858Z level=info msg="Executing migration" id="Add unique index for title, parent_uid, and org_id" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.219663687Z level=info msg="Migration successfully executed" id="Add unique index for title, parent_uid, and org_id" duration=1.078437ms 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.22008414Z level=info msg="Executing migration" id="Sync dashboard and folder table" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.220489955Z level=info msg="Migration successfully executed" id="Sync dashboard and folder table" duration=405.647µs 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.220859054Z level=info msg="Executing migration" id="Remove ghost folders from the folder table" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.221103983Z level=info msg="Migration successfully executed" id="Remove ghost folders from the folder table" duration=244.59µs 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.221553684Z level=info msg="Executing migration" id="Remove unique index UQE_folder_uid_org_id" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.222548138Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_uid_org_id" duration=994.322µs 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.222950467Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_uid" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.223989487Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_uid" duration=1.03846ms 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.224386564Z level=info msg="Executing migration" id="Remove unique index UQE_folder_title_parent_uid_org_id" 2024-03-21T14:04:13.465 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.225373937Z level=info msg="Migration successfully executed" id="Remove unique index UQE_folder_title_parent_uid_org_id" duration=987.122µs 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.225772217Z level=info msg="Executing migration" id="Add unique index UQE_folder_org_id_parent_uid_title" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.226813279Z level=info msg="Migration successfully executed" id="Add unique index UQE_folder_org_id_parent_uid_title" duration=1.040781ms 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.227217306Z level=info msg="Executing migration" id="Remove index IDX_folder_parent_uid_org_id" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.22821359Z level=info msg="Migration successfully executed" id="Remove index IDX_folder_parent_uid_org_id" duration=996.147µs 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.228636751Z level=info msg="Executing migration" id="create anon_device table" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.229399829Z level=info msg="Migration successfully executed" id="create anon_device table" duration=763.181µs 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.229695969Z level=info msg="Executing migration" id="add unique index anon_device.device_id" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.230437735Z level=info msg="Migration successfully executed" id="add unique index anon_device.device_id" duration=741.52µs 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.230747028Z level=info msg="Executing migration" id="add index anon_device.updated_at" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.231420017Z level=info msg="Migration successfully executed" id="add index anon_device.updated_at" duration=672.719µs 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.231721996Z level=info msg="Executing migration" id="create signing_key table" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.232315654Z level=info msg="Migration successfully executed" id="create signing_key table" duration=593.377µs 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.232648127Z level=info msg="Executing migration" id="add unique index signing_key.key_id" 2024-03-21T14:04:13.466 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.233315265Z level=info msg="Migration successfully executed" id="add unique index signing_key.key_id" duration=665.696µs 2024-03-21T14:04:13.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.233624208Z level=info msg="Executing migration" id="set legacy alert migration status in kvstore" 2024-03-21T14:04:13.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.234306393Z level=info msg="Migration successfully executed" id="set legacy alert migration status in kvstore" duration=681.956µs 2024-03-21T14:04:13.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.234766613Z level=info msg="Executing migration" id="migrate record of created folders during legacy migration to kvstore" 2024-03-21T14:04:13.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.234976868Z level=info msg="Migration successfully executed" id="migrate record of created folders during legacy migration to kvstore" duration=210.935µs 2024-03-21T14:04:13.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.235352313Z level=info msg="Executing migration" id="Add folder_uid for dashboard" 2024-03-21T14:04:13.467 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.240510442Z level=info msg="Migration successfully executed" id="Add folder_uid for dashboard" duration=5.157821ms 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.240861666Z level=info msg="Executing migration" id="Populate dashboard folder_uid column" 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.241338825Z level=info msg="Migration successfully executed" id="Populate dashboard folder_uid column" duration=477.805µs 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.241706922Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title" 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.242431389Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title" duration=722.456µs 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.242757974Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_id_title" 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.243460684Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_id_title" duration=702.559µs 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.243785181Z level=info msg="Executing migration" id="Delete unique index for dashboard_org_id_folder_uid_title" 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.244469843Z level=info msg="Migration successfully executed" id="Delete unique index for dashboard_org_id_folder_uid_title" duration=684.685µs 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.244792329Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.24553151Z level=info msg="Migration successfully executed" id="Add unique index for dashboard_org_id_folder_uid_title_is_folder" duration=738.698µs 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.245849583Z level=info msg="Executing migration" id="Restore index for dashboard_org_id_folder_id_title" 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.246578422Z level=info msg="Migration successfully executed" id="Restore index for dashboard_org_id_folder_id_title" duration=728.641µs 2024-03-21T14:04:13.468 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.246916595Z level=info msg="Executing migration" id="create sso_setting table" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.24758413Z level=info msg="Migration successfully executed" id="create sso_setting table" duration=668.024µs 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.247958748Z level=info msg="Executing migration" id="copy kvstore migration status to each org" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.24845752Z level=info msg="Migration successfully executed" id="copy kvstore migration status to each org" duration=499.624µs 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.248774821Z level=info msg="Executing migration" id="add back entry for orgid=0 migrated status" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.248969237Z level=info msg="Migration successfully executed" id="add back entry for orgid=0 migrated status" duration=194.972µs 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.249346728Z level=info msg="Executing migration" id="alter kv_store.value to longtext" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.249392368Z level=info msg="Migration successfully executed" id="alter kv_store.value to longtext" duration=45.145µs 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.24976501Z level=info msg="Executing migration" id="add notification_settings column to alert_rule table" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.255012421Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule table" duration=5.25299ms 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.255339195Z level=info msg="Executing migration" id="add notification_settings column to alert_rule_version table" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.260759759Z level=info msg="Migration successfully executed" id="add notification_settings column to alert_rule_version table" duration=5.420207ms 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.261091191Z level=info msg="Executing migration" id="removing scope from alert.instances:read action migration" 2024-03-21T14:04:13.469 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.261316784Z level=info msg="Migration successfully executed" id="removing scope from alert.instances:read action migration" duration=225.561µs 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=migrator t=2024-03-21T14:04:13.261675146Z level=info msg="migrations completed" performed=547 skipped=0 duration=922.533413ms 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore t=2024-03-21T14:04:13.262513112Z level=info msg="Created default organization" 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=secrets t=2024-03-21T14:04:13.263028829Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=plugin.store t=2024-03-21T14:04:13.283807577Z level=info msg="Loading plugins..." 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=local.finder t=2024-03-21T14:04:13.339101869Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=plugin.store t=2024-03-21T14:04:13.339119764Z level=info msg="Plugins loaded" count=55 duration=55.313923ms 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=query_data t=2024-03-21T14:04:13.341177741Z level=info msg="Query Service initialization" 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=live.push_http t=2024-03-21T14:04:13.343484128Z level=info msg="Live Push Gateway initialization" 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.migration t=2024-03-21T14:04:13.344493067Z level=info msg=Starting 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.migration t=2024-03-21T14:04:13.344764994Z level=info msg="Applying transition" currentType=Legacy desiredType=UnifiedAlerting cleanOnDowngrade=false cleanOnUpgrade=false 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.migration orgID=1 t=2024-03-21T14:04:13.34503244Z level=info msg="Migrating alerts for organisation" 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.migration orgID=1 t=2024-03-21T14:04:13.345431318Z level=info msg="Alerts found to migrate" alerts=0 2024-03-21T14:04:13.470 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.migration t=2024-03-21T14:04:13.346526964Z level=info msg="Completed alerting migration" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.state.manager t=2024-03-21T14:04:13.356668359Z level=info msg="Running in alternative execution of Error/NoData mode" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=infra.usagestats.collector t=2024-03-21T14:04:13.357943135Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=provisioning.datasources t=2024-03-21T14:04:13.359504408Z level=info msg="inserting datasource from configuration" name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=provisioning.datasources t=2024-03-21T14:04:13.371670093Z level=info msg="inserting datasource from configuration" name=Loki uid=P8E80F9AEF21F6940 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=provisioning.alerting t=2024-03-21T14:04:13.38434237Z level=info msg="starting to provision alerting" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=provisioning.alerting t=2024-03-21T14:04:13.384353009Z level=info msg="finished to provision alerting" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafanaStorageLogger t=2024-03-21T14:04:13.384492604Z level=info msg="Storage starting" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.state.manager t=2024-03-21T14:04:13.385027307Z level=info msg="Warming state cache for startup" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.state.manager t=2024-03-21T14:04:13.385223349Z level=info msg="State cache has been initialized" states=0 duration=195.193µs 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.multiorg.alertmanager t=2024-03-21T14:04:13.385280506Z level=info msg="Starting MultiOrg Alertmanager" 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ngalert.scheduler t=2024-03-21T14:04:13.385358537Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ticker t=2024-03-21T14:04:13.385399236Z level=info msg=starting first_tick=2024-03-21T14:04:20Z 2024-03-21T14:04:13.471 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=http.server t=2024-03-21T14:04:13.387291054Z level=info msg="HTTP Server TLS settings" MinTLSVersion=TLS1.2 configuredciphers=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=http.server t=2024-03-21T14:04:13.387573899Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=provisioning.dashboard t=2024-03-21T14:04:13.413711816Z level=info msg="starting to provision dashboards" 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:04:13.423201361Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:04:13.433645269Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:04:13.444092971Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:04:13.454437255Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-21T14:04:13.472 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:04:13.464876723Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T14:04:13.678 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:04:13.678 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mon.a", 2024-03-21T14:04:13.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster": "ceph", 2024-03-21T14:04:13.678 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-21T14:04:13.678 INFO:teuthology.orchestra.run.smithi003.stdout: "admin_socket_mode": "", 2024-03-21T14:04:13.678 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_client_required": "cephx, none", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_cluster_required": "cephx", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_debug": "true", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_required": "cephx", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "auth_supported": "", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio": "true", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-21T14:04:13.679 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_poll_ms": "250", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_aio_reap_max": "16", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_async_discard_threads": "0", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_block_size": "4096", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio": "false", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_debug_inflight_ios": "false", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_enable_discard": "false", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry": "3", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash": "0", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring": "false", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_hipri": "false", 2024-03-21T14:04:13.680 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bdev_type": "", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_alloc_size": "1048576", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_allocator": "hybrid", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_buffered_io": "true", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_for_zeros": "false", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_compact_log_sync": "false", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-21T14:04:13.681 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_log_runway": "4194304", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_max_prefetch": "1048576", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_flush_size": "524288", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_min_log_runway": "1048576", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery": "false", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_sync_write": "false", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocation_from_file": "true", 2024-03-21T14:04:13.682 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_allocator": "hybrid", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_create": "true", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_create": "false", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_path": "", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_db_size": "0", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_path": "", 2024-03-21T14:04:13.683 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_preallocate_file": "false", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_size": "107374182400", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_create": "false", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_path": "", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_block_wal_size": "100663296", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs": "true", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune": "true", 2024-03-21T14:04:13.684 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size": "0", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_type": "2q", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_clone_cow": "true", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-21T14:04:13.685 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_mode": "none", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_csum_type": "crc32c", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_freelist": "false", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-21T14:04:13.686 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_small_allocations": "0", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_read": "true", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_default_buffered_write": "false", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-21T14:04:13.687 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fail_eio": "false", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-21T14:04:13.688 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_ignore_data_csum": "false", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-21T14:04:13.689 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_log_op_age": "5.000000", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_alloc_size": "0", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size": "0", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_max_deferred_txc": "32", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size": "0", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-21T14:04:13.690 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_nid_prealloc": "1024", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_retry_disk_reads": "3", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cf": "true", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-21T14:04:13.691 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_spdk_mem": "512", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_tracing": "false", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-21T14:04:13.692 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_zero_block_detection": "false", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-21T14:04:13.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_signatures": "false", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_cluster_require_version": "2", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_signatures": "false", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_require_version": "2", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_signatures": "false", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_service_require_version": "2", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "cephx_sign_messages": "true", 2024-03-21T14:04:13.694 INFO:teuthology.orchestra.run.smithi003.stdout: "chdir": "", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_acl_type": "", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_asio_thread_count": "2", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_mid": "0.750000", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_cache_size": "16384", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_caps_release_delay": "5", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_check_pool_perm": "true", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_force_sync_read": "false", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_getattr_caps": "false", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_debug_inject_tick_delay": "0", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_die_on_failed_remount": "false", 2024-03-21T14:04:13.695 INFO:teuthology.orchestra.run.smithi003.stdout: "client_dirsize_rbytes": "true", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_force_lazyio": "false", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_fs": "", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_inject_release_failure": "false", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_inline_size": "4096", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mds_namespace": "", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_metadata": "", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_gid": "-1", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_timeout": "300", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mount_uid": "-1", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_mountpoint": "/", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_notify_timeout": "10", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc": "true", 2024-03-21T14:04:13.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty": "104857600", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_max_objects": "1000", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_size": "209715200", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_oc_target_dirty": "8388608", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_permissions": "true", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota": "true", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_quota_df": "true", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_bytes": "0", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_max_periods": "4", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_readahead_min": "131072", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_reconnect_stale": "false", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_shutdown_timeout": "30", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_snapdir": ".snap", 2024-03-21T14:04:13.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_tick_interval": "1", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_trace": "", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_try_dentry_invalidate": "false", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_faked_inos": "false", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "client_use_random_mds": "false", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog": "false", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_graylog_port": "12201", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_monitors": "default=true", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog": "false", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "clog_to_syslog_level": "info", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "-", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network": "", 2024-03-21T14:04:13.698 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_network_interface": "", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_isal": "false", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_level": "5", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zlib_winsize": "-15", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "compressor_zstd_level": "1", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_op_num_threads": "6", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crimson_seastar_cpu_cores": "", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location": "", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook": "", 2024-03-21T14:04:13.699 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_location_hook_timeout": "10", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "daemonize": "false", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "daos_pool": "tank", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_alienstore": "0/5", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_allow_any_pool_priority": "false", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok": "1/5", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asok_assert_abort": "false", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_asserts_on_shutdown": "true", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_auth": "1/5", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bdev": "1/3", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluefs": "1/5", 2024-03-21T14:04:13.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_bluestore": "1/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_buffer": "0/1", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ceph_exporter": "1/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephfs_mirror": "0/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cephsqlite": "0/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_client": "0/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_compressor": "1/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_context": "0/1", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crimson_interrupt": "0/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crush": "1/1", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_crypto": "1/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_cyanstore": "0/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_deliberately_leak_memory": "false", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_disable_randomized_ping": "false", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_dpdk": "1/5", 2024-03-21T14:04:13.701 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_eventtrace": "1/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filer": "0/1", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_filestore": "1/3", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_finisher": "1/1", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_fuse": "1/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeat_testing_span": "0", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_heartbeatmap": "1/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_javaclient": "1/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journal": "1/3", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_journaler": "0/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_kstore": "1/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_lockdep": "0/1", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mclock": "1/5", 2024-03-21T14:04:13.702 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_balancer": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_locker": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_log_expire": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_migrator": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mds_quiesce": "3/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_memstore": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgr": "2/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mgrc": "1/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_mon": "20/20", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_monc": "0/10", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_ms": "1/1", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_none": "0/5", 2024-03-21T14:04:13.703 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objclass": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objectcacher": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_objecter": "0/1", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_optracker": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_osd": "1/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_paxos": "20/20", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_perfcounter": "1/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_prioritycache": "1/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rados": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_mirror": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_pwl": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rbd_replay": "0/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_refs": "0/0", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_reserver": "1/1", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw": "1/5", 2024-03-21T14:04:13.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_access": "1/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_datacache": "1/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_dbstore": "1/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_flight": "1/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rgw_sync": "1/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_rocksdb": "4/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_backref": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cache": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_cleaner": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_device": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_epm": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_journal": "0/5", 2024-03-21T14:04:13.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_lba": "0/5", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_odata": "0/5", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_omap": "0/5", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_onode": "0/5", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_t": "0/5", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_seastore_tm": "0/5", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_striper": "0/1", 2024-03-21T14:04:13.706 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_test": "0/5", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_throttle": "1/1", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_timer": "0/1", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_tp": "0/5", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "debug_trace": "1/5", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "device_failure_prediction_mode": "none", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_graylog": "false", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_journald": "false", 2024-03-21T14:04:13.707 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_stderr": "true", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "err_to_syslog": "false", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event_tracing": "false", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_addr": "0.0.0.0", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_http_port": "9926", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_prio_limit": "5", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_sort_metrics": "true", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "exporter_stats_period": "5", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "fake_statfs_for_testing": "0", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "fatal_signal_handlers": "true", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_purge_ops": "10", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "filer_max_truncate_ops": "128", 2024-03-21T14:04:13.708 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_apply_finisher_threads": "1", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_blackhole": "false", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_clone_range": "true", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_btrfs_snap": "true", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_caller_concurrency": "10", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_collect_device_partition_information": "true", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_commit_timeout": "600.000000", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_inject_read_err": "false", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_omap_check": "false", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_debug_verify_split": "false", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_dump_file": "", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fadvise": "true", 2024-03-21T14:04:13.709 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fail_eio": "true", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_shards": "16", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fd_cache_size": "128", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap": "false", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fiemap_threshold": "4096", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_inject_stall": "0", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_parallel": "false", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_trailing": "false", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_journal_writeahead": "false", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_kill_at": "0", 2024-03-21T14:04:13.710 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs": "0", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size": "0", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_merge_threshold": "-10", 2024-03-21T14:04:13.711 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_odsync_write": "false", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend": "rocksdb", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_backend_path": "", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_thread_timeout": "60", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_op_threads": "2", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_punch_hole": "false", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-21T14:04:13.712 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_queue_max_ops": "50", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_seek_data_hole": "false", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc": "false", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_splice": "false", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_multiple": "2", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_split_rand_factor": "20", 2024-03-21T14:04:13.713 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_update_to": "1000", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_enable": "true", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-21T14:04:13.714 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_xfs_extsize": "false", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "filestore_zfs_snap": "false", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fio_dir": "/tmp/fio", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_allow_other": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_atomic_o_trunc": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_big_writes": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_debug": "false", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_default_permissions": "false", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_disable_pagecache": "false", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_max_write": "0", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_multithreaded": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_require_active_mds": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_set_user_groups": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_move": "true", 2024-03-21T14:04:13.715 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_read": "true", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_splice_write": "true", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "fuse_use_invalidate_cb": "true", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "gss_target_name": "ceph", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_file": "", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_inject_failure": "0", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_interval": "5", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "host": "smithi003", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-21T14:04:13.716 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "inject_early_sigterm": "false", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_agent_port": "6799", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "jaeger_tracing_enable": "false", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_aio": "true", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_align_min_size": "65536", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_align": "true", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_block_size": "4096", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_dio": "true", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_discard": "false", 2024-03-21T14:04:13.717 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_force_aio": "false", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_ignore_corruption": "false", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_bytes": "10485760", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_max_write_entries": "100", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_replay_from": "0", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_write_header_frequency": "0", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journal_zero_on_create": "false", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prefetch_periods": "10", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_prezero_periods": "5", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "journaler_write_head_interval": "15", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "key": "", 2024-03-21T14:04:13.718 INFO:teuthology.orchestra.run.smithi003.stdout: "keyfile": "", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_backend": "rocksdb", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_default_stripe_size": "65536", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount": "false", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_bytes": "67108864", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_max_ops": "512", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_nid_prealloc": "1024", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_onode_map_size": "1024", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_submit_transaction": "false", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "kstore_sync_transaction": "false", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "librados_thread_count": "2", 2024-03-21T14:04:13.719 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep": "false", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "lockdep_force_backtrace": "false", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_coarse_timestamps": "true", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_flush_on_exit": "false", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_host": "127.0.0.1", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_graylog_port": "12201", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_new": "1000", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_max_recent": "10000", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stderr_prefix": "", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_stop_at_utilization": "0.970000", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": "true", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_graylog": "false", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_journald": "false", 2024-03-21T14:04:13.720 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_stderr": "false", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_syslog": "false", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "max_rotating_auth_attempts": "10", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_action_on_write_error": "1", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_alternate_name_max": "8192", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_asio_thread_count": "2", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_export_pin": "true", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_dirs": "true", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_interval": "5", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-21T14:04:13.721 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_interval": "10", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max": "-1", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_max_until": "-1", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_merge_size": "50", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_midchunk": "0.300000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_min_start": "0.200000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_minchunk": "0.001000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_mode": "0", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_max": "1.200000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_need_min": "0.800000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_overload_epochs": "2", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-21T14:04:13.722 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_bits": "3", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_size": "10000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_target_decay": "10.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_grace": "15.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_interval": "4.000000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_mid": "0.700000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_release_free_interval": "10", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_reservation": "0.050000", 2024-03-21T14:04:13.723 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_interval": "1", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache_trim_threshold": "262144", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_prealloc_inos": "1000", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_connect_bootstrapping": "false", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_damage_table_max_entries": "10000", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_auth_pins": "false", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_frag": "false", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_scatterstat": "false", 2024-03-21T14:04:13.724 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_debug_subtrees": "false", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_decay_halflife": "5.000000", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_default_dir_hash": "2", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_defer_session_stale": "true", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_deny_all_reconnect": "false", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_keys_per_op": "16384", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_commit_size": "10", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_max_entries": "0", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dir_prefetch": "true", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_on_map": "false", 2024-03-21T14:04:13.725 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_early_reply": "true", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enable_op_tracker": "true", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_enforce_unique_name": "true", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random": "true", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-21T14:04:13.726 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_health_summarize_threshold": "10", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_health_dummy": "false", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_migrator_session_race": "false", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_join_fs": "", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_journal_format": "1", 2024-03-21T14:04:13.727 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_create_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_export_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_import_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_expire_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_journal_replay_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_link_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_mdstable_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_openc_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_rename_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_shutdown_at": "0", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_event_large_threshold": "524288", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_events_per_segment": "1024", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-21T14:04:13.728 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_events": "-1", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_max_segments": "128", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_pause": "false", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_segment_size": "0", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_threshold": "128", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log_warn_factor": "2.000000", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_caps_per_client": "1048576", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_flushes": "100000", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_completed_requests": "100000", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_export_size": "20971520", 2024-03-21T14:04:13.729 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_file_recover": "32", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_files": "64", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops": "8192", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_max_snaps_per_dir": "100", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_metrics_update_interval": "2", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_per_client": "100", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_min_caps_working_set": "10000", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_numa_node": "-1", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_complaint_time": "30.000000", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_duration": "600", 2024-03-21T14:04:13.730 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_size": "20", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_size": "20", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_op_log_threshold": "5", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_grace": "15", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_ping_interval": "5", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_caps": "30000", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_recall_warning_threshold": "262144", 2024-03-21T14:04:13.731 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_interval": "1.000000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_gid": "0", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_root_ino_uid": "0", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-21T14:04:13.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_shutdown_check": "0", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_skip_ino": "0", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_max_uid": "4294967294", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_min_uid": "0", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_snap_rstat": "false", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_standby_replay_damaged": "false", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_symlink_recovery": "true", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_exports": "0", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_thrash_fragments": "0", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_tick_interval": "5.000000", 2024-03-21T14:04:13.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_valgrind_exit": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_backtrace": "1", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_verify_scatter": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_wipe_sessions": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool_debug": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_device_bytes": "1073741824", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_set": "false", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "memstore_page_size": "65536", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_bytes": "134217728", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_messages": "512", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-21T14:04:13.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_disabled_modules": "", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_max_pg_num_change": "128", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_bytes": "134217728", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mds_messages": "128", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_bytes": "134217728", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_mon_messages": "128", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_bytes": "536870912", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_osd_messages": "8192", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_pool": "true", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_standby_modules": "true", 2024-03-21T14:04:13.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_period": "5", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_stats_threshold": "5", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_tick_period": "2", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_delete": "true", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_allow_pool_size_one": "false", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_auth_validate_all_caps": "true", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_bytes": "104857600", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_directed_command_retry": "2", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-21T14:04:13.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_hunt_parallel": "3", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_log_interval": "1.000000", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_interval": "10.000000", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_client_target_rank": "-1", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_file_level": "debug", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_file": "true", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-21T14:04:13.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_journald": "true", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_bootstrap": "false", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_start": "false", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_compact_on_trim": "true", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_cpu_threads": "4", 2024-03-21T14:04:13.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_crush_min_required_version": "hammer", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_daemon_bytes": "419430400", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_crit": "5", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_avail_warn": "5", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_data_size_warn": "16106127360", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_json": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_dump_transactions": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_extra_checks": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_reef": "false", 2024-03-21T14:04:13.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_no_require_squid": "false", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_down_mkfs_grace": "60", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_default_strategy": "1", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_election_timeout": "5.000000", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_divisor": "2", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_enable_op_tracker": "true", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_fake_pool_delete": "false", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_force_quorum_join": "false", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_globalid_prealloc": "10000", 2024-03-21T14:04:13.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_detail_to_clog": "true", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_log_update_period": "5", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_max_detail": "50", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog": "true", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_interval": "600", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host": "[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_host_override": "", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_initial_members": "", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease": "5.000000", 2024-03-21T14:04:13.741 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_full_interval": "50", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max": "10000", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_log_max_summary": "50", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_log_epochs": "500", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_osd": "10000", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pg_per_osd": "10000", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_pool_pg_num": "65536", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-21T14:04:13.742 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_force_trim_to": "0", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mds_skip_sanity": "false", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_autotune": "true", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_memory_target": "2147483648", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_beacon_grace": "30", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_digest_period": "5", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_inactive_grace": "60", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_min_osdmap_epochs": "50", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_complaint_time": "30", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_duration": "600", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_size": "20", 2024-03-21T14:04:13.743 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_size": "20", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_op_log_threshold": "5", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_in": "false", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size": "500", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-21T14:04:13.744 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_interval": "600", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_force_trim_to": "0", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_down_reporters": "2", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-21T14:04:13.745 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_report_timeout": "900", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_stuck_threshold": "60", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-21T14:04:13.746 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_probe_timeout": "2.000000", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_change": "0.050000", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_max_osds": "4", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_interval": "86400", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_max_keys": "100", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_scrub_timeout": "300", 2024-03-21T14:04:13.747 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_session_timeout": "300", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_smart_report_timeout": "5", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stat_smooth_intervals": "6", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_min_size": "2", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_pool_size": "4", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_debug": "false", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-21T14:04:13.748 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-21T14:04:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_provider_kill_at": "0", 2024-03-21T14:04:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_requester_kill_at": "0", 2024-03-21T14:04:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_sync_timeout": "60.000000", 2024-03-21T14:04:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_target_pg_per_osd": "100", 2024-03-21T14:04:13.749 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_tick_interval": "5", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_interval": "300.000000", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_use_min_delay_socket": "false", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_older_version_delay": "604800", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_misplaced": "false", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-21T14:04:13.750 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_older_version": "true", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "monmap": "", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-21T14:04:13.751 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "motr_tracing_enabled": "false", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_op_threads": "3", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_cm": "false", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_device_name": "", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_dscp": "96", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_local_gid": "", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_port_num": "1", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-21T14:04:13.752 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_sl": "3", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_support_srq": "true", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_rdma_type": "ib", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_async_reap_threshold": "5", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_before_connect": "false", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv4": "true", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_ipv6": "false", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr1": "false", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_msgr2": "true", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_max": "7568", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_port_min": "6800", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_count": "3", 2024-03-21T14:04:13.753 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_bind_retry_delay": "5", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_client": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mds": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mgr": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_mon": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_blackhole_osd": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_client_mode": "crc secure", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_mode": "crc secure", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_cluster_type": "", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_compress_secure": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_idle_timeout": "900", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_connection_ready_timeout": "10", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_data": "true", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_crc_header": "true", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bad_msg": "false", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_bug": "true", 2024-03-21T14:04:13.754 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_old_message": "true", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_skipped_message": "false", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_coremask": "0xF", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_enable_tso": "true", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hugepages": "", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_lro": "true", 2024-03-21T14:04:13.755 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_memory_channel": "4", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_pmd": "", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_port_id": "0", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_dump_on_send": "false", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_initial_backoff": "0.200000", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_max": "1.000000", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_delay_type": "", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_network_congestion": "0", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_inject_socket_failures": "0", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_learn_addr_from_peer": "true", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_accept_failures": "4", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_max_backoff": "15.000000", 2024-03-21T14:04:13.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_client_mode": "secure crc", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_mon_service_mode": "secure crc", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_min_size": "1024", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compress_mode": "none", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_pq_min_cost": "65536", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_public_type": "", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_service_mode": "crc secure", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_listen_backlog": "512", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_nodelay": "true", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_tcp_rcvbuf": "0", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "ms_type": "async+posix", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "no_config_file": "false", 2024-03-21T14:04:13.757 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_completion_locks_per_session": "32", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inflight_ops": "1024", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_tick_interval": "5.000000", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter_timeout": "10.000000", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_blackhole": "false", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "openssl_engine_opts": "", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_delay_time": "5.000000", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_hist_halflife": "1000", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_low_ops": "2", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_max_ops": "4", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-21T14:04:13.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_agent_slop": "0.020000", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_asio_thread_count": "2", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_async_recovery_min_cost": "100", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_max": "512", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backfill_scan_min": "64", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_degraded": "false", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_peering": "false", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_backoff_on_unfound": "true", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_beacon_report_interval": "300", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_duration": "30", 2024-03-21T14:04:13.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_max_block_size": "67108864", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blkin_trace_all": "false", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_for_log_corruption": "false", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_class_update_on_start": "true", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_cap": "256", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_message_size_cap": "524288000", 2024-03-21T14:04:13.760 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_op_priority": "63", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_client_watch_timeout": "30", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_command_max_records": "256", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_compact_on_start": "false", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_on_start": "true", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_crush_update_weight_set": "true", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_feed_pullee": "-1", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-21T14:04:13.761 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_misdirected_ops": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_acting_change": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_no_purge_strays": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_op_order": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_shutdown": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_trim_objects": "false", 2024-03-21T14:04:13.762 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_snaps": "false", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_keys": "1024", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_stride": "524288", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_default_notify_timeout": "30", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep": "0.000000", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-21T14:04:13.763 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_discard_disconnected_ops": "true", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_enable_op_tracker": "true", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_info": "true", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown": "true", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_function_tracing": "false", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_grace": "20", 2024-03-21T14:04:13.764 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_interval": "6", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_peers": "10", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_min_size": "2000", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_stale": "600", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_max_size": "100000", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_min_size": "1000", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-21T14:04:13.765 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_journal_size": "5120", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_kill_backfill_at": "0", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_cache_size": "50", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_dedup": "true", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max": "40", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_map_share_max_epochs": "40", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_name_len": "100", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_attr_size": "0", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_backfills": "1", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_count": "5", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_markdown_period": "600", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_name_len": "2048", 2024-03-21T14:04:13.766 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_namespace_len": "256", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_object_size": "134217728", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_blocked_by": "16", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_log_entries": "10000", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_pgls": "1024", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_cost": "8388608", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_push_objects": "10", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_scrubs": "3", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_trimming_pgs": "2", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_op_reply_len": "64", 2024-03-21T14:04:13.767 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_max_write_size": "90", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_profile": "balanced", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-21T14:04:13.768 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_base": "805306368", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_min": "134217728", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target": "4294967296", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_autotune": "false", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_pg_log_entries": "250", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_min_recovery_priority": "0", 2024-03-21T14:04:13.769 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_interval": "5", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_cache_shards": "32", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_num_op_tracker_shard": "32", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_auto_affinity": "true", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_node": "-1", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_numa_prefer_iface": "true", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objecter_finishers": "1", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore": "bluestore", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_fuse": "false", 2024-03-21T14:04:13.770 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_objectstore_tracing": "false", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_complaint_time": "30.000000", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_duration": "600", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_size": "20", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_size": "20", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_log_threshold": "5", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards": "0", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_hdd": "5", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_shards_ssd": "8", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_pq_min_cost": "65536", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-21T14:04:13.771 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_queue_cut_off": "high", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_op_thread_timeout": "15", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_open_classes_on_start": "true", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_os_flags": "0", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_peering_op_priority": "255", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_cost": "1048576", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_delete_priority": "5", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_max": "10000", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_log_trim_min": "100", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-21T14:04:13.772 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crimson": "false", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_flags": "0", 2024-03-21T14:04:13.773 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_min_size": "0", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pg_num": "32", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_pgp_num": "0", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_read_ratio": "70", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_size": "3", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_default_type": "replicated", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_push_per_object_cost": "1000", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap": "true", 2024-03-21T14:04:13.774 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_cost": "20971520", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active": "0", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_max_single_start": "1", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_priority": "3", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_priority": "5", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep": "0.000000", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-21T14:04:13.775 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_repair_during_recovery": "false", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_requested_scrub_priority": "120", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair": "false", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_hour": "0", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_begin_week_day": "0", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_max": "25", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_chunk_min": "5", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_cost": "52428800", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_during_recovery": "false", 2024-03-21T14:04:13.776 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_hour": "0", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_end_week_day": "0", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_event_cost": "4096", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_invalid_stats": "true", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_max_preemptions": "5", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_priority": "5", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_sleep": "0.000000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-21T14:04:13.777 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_skip_data_digest": "false", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_smart_report_timeout": "5", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_cost": "1048576", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_priority": "5", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-21T14:04:13.778 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_target_transaction_size": "30", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-21T14:04:13.779 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_tracing": "false", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_use_stale_snap": "false", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "osdc_blkin_trace_all": "false", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_kill_at": "0", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_max_join_drift": "10", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min": "500", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_min_wait": "0.050000", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_propose_interval": "1.000000", 2024-03-21T14:04:13.780 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max": "500", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_service_trim_min": "10", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_stash_full_interval": "25", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_max": "500", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "paxos_trim_min": "250", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "perf": "true", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "pid_file": "", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "-", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrv": "", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "public_bind_addr": "-", 2024-03-21T14:04:13.781 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network": "", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "public_network_interface": "", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_enabled": "false", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "qat_compressor_session_max_number": "256", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_mon_op_timeout": "0", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_osd_op_timeout": "0", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rados_tracing": "false", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_atime_update_interval": "60", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_parent_reads": "false", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_balance_snap_reads": "false", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blkin_trace_all": "false", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-21T14:04:13.782 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache": "true", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_policy": "writearound", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_size": "33554432", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_clone_copy_on_read": "false", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_compression_hint": "none", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_concurrent_management_ops": "10", 2024-03-21T14:04:13.783 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_clone_format": "auto", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_data_pool": "", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_features": "61", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_format": "2", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_map_options": "", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_order": "22", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_pool": "rbd", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_count": "0", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_default_stripe_unit": "0", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-21T14:04:13.784 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_enable_alloc_hint": "true", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler": "simple", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_order": "24", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_pool": "", 2024-03-21T14:04:13.785 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_journal_splay_width": "4", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_parent_reads": "false", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_localize_snap_reads": "false", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-21T14:04:13.786 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-21T14:04:13.787 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_mtime_update_interval": "60", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_non_blocking_aio": "true", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_thread_timeout": "60", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_op_threads": "1", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_parent_cache_enabled": "false", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_plugins": "", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst": "0", 2024-03-21T14:04:13.788 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_bps_limit": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_exclude_ops": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_iops_limit": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-21T14:04:13.789 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_read_from_replica_policy": "default", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-21T14:04:13.790 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_skip_partial_discard": "true", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_tracing": "false", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_names": "true", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_validate_pool": "true", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_base_url": "", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "restapi_log_level": "", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_acl_grants_max_num": "100", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_admin_entry": "admin", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_backend_store": "rados", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_barbican_url": "", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_beast_enable_async": "true", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache": "false", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-21T14:04:13.791 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_enabled": "true", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_expiry_interval": "900", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cache_lru_size": "10000", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_config_store": "rados", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_content_length_compat": "false", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress": "true", 2024-03-21T14:04:13.792 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cors_rules_max_num": "100", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_cross_domain_policy": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_addr": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_password": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_kmip_username": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_require_ssl": "true", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-21T14:04:13.793 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_addr": "", 2024-03-21T14:04:13.794 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_auth": "token", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_namespace": "", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_prefix": "", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_token_file": "", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_buffersize": "524288", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_low_speed_time": "300", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-21T14:04:13.795 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_d4n_port": "6379", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_changes_size": "1000", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_num_shards": "128", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_log_window": "30", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-21T14:04:13.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-21T14:04:13.797 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_name": "", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dns_s3website_name": "", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_dynamic_resharding": "true", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_gc_threads": "true", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_lc_threads": "true", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_ops_log": "false", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_quota_threads": "true", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_static_website": "false", 2024-03-21T14:04:13.798 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enable_usage_log": "false", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_enforce_swift_acls": "true", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_exit_timeout_secs": "120", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_expose_bucket": "false", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_extended_http_attrs": "", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_filter": "none", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_frontends": "beast port=7480", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred": "50", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_objs": "32", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-21T14:04:13.799 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_gc_processor_period": "3600", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_init_timeout": "300", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_connection_idle": "300", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_domain": "", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password": "", 2024-03-21T14:04:13.800 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_password_path": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_project": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_tenant": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_token_path": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_admin_user": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_api_version": "2", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_domain": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_password": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_project": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_barbican_user": "", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-21T14:04:13.801 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_url": "", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_debug_interval": "-1", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_lock_max_time": "90", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_objs": "32", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_rules": "1000", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_worker": "3", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lc_thread_delay": "0", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_dnattr": "uid", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_searchfilter": "", 2024-03-21T14:04:13.802 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_http_headers": "", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_log_object_name_utc": "false", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_name_len": "0", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attr_size": "0", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_chunk_size": "4194304", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-21T14:04:13.803 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_listing_results": "1000", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_notify_retries": "10", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_param_size": "1048576", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_put_size": "5368709120", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_max_slo_entries": "1000", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_log_max_shards": "64", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_mp_lock_max_time": "600", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-21T14:04:13.804 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_max_gc": "300", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-21T14:04:13.805 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_async_rados_threads": "32", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_num_control_oids": "8", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_numa_node": "-1", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_chunk_size": "100", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_gc_interval": "600", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_thread_timeout": "600", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_op_tracing": "false", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_token": "", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_url": "", 2024-03-21T14:04:13.806 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_opa_verify_ssl": "true", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_rados": "false", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_ops_log_socket_path": "", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval": "2.000000", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lanes": "3", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-21T14:04:13.807 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_cache_partitions": "3", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_continue": "true", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rados_tracing": "false", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm": "", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_id": "", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region": "", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-21T14:04:13.808 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_request_uri": "", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_batch_size": "64", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_max_aio": "128", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_num_logs": "16", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_reshard_thread_interval": "600", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_resolve_cname": "false", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_run_sync_thread": "true", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-21T14:04:13.809 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_scheduler_type": "throttler", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_script_uri": "", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_service_provider_name": "", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_id": "", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_client_secret": "", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_entry": "sts", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_key": "sts", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_min_session_duration": "900", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sts_token_introspection_url": "", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_account_in_url": "false", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_entry": "auth", 2024-03-21T14:04:13.810 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_auth_url": "", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_custom_header": "", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_need_stats": "true", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_tenant_name": "", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_token_expiration": "86400", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url": "", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_url_prefix": "swift", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_lease_period": "120", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-21T14:04:13.811 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_thread_pool_size": "512", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_comment": "", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_createby": "", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_encoding": "", 2024-03-21T14:04:13.812 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_flag": "false", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_origin": "", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_torrent_tracker": "", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_trust_forwarded_https": "false", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_shards": "32", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_usage_max_user_shards": "1", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_use_opa_authz": "false", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache": "false", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-21T14:04:13.813 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_max_buckets": "1000", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_policies_max_num": "100", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_user_unique_email": "true", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_verify_ssl": "true", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone": "", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_id": "", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup": "", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_id": "", 2024-03-21T14:04:13.814 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_auto_session_clear": "true", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_block_size": "4096", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_size": "536870912", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-21T14:04:13.815 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_index_type": "binary_search", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_partition_filters": "false", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_perf": "false", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "run_dir": "/var/run/ceph", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_block_create": "true", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cache_lru_size": "67108864", 2024-03-21T14:04:13.816 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_max_object_size": "16777216", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_device_size": "53687091200", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_capacity": "16", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_main_device_type": "SSD", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-21T14:04:13.817 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "seastore_segment_size": "67108864", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "set_keepcaps": "false", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "setgroup": "ceph", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser": "ceph", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "setuser_match_path": "", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "thp": "false", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_default_timeout": "60", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout: "throttler_perf_counter": "true" 2024-03-21T14:04:13.818 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:04:13.830 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:04:13.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:04:13.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-mon[33862]: Deploying daemon node-exporter.a on smithi003 2024-03-21T14:04:13.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:04:13.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:04:13.831 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=plugins.update.checker t=2024-03-21T14:04:13.469729149Z level=info msg="Update check succeeded" duration=85.265722ms 2024-03-21T14:04:13.831 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafana.update.checker t=2024-03-21T14:04:13.469733207Z level=info msg="Update check succeeded" duration=85.268168ms 2024-03-21T14:04:13.831 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=provisioning.dashboard t=2024-03-21T14:04:13.588295191Z level=info msg="finished to provision dashboards" 2024-03-21T14:04:13.831 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafana-apiserver t=2024-03-21T14:04:13.831305508Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager" 2024-03-21T14:04:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:04:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[29576]: Deploying daemon node-exporter.a on smithi003 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[40510]: Deploying daemon node-exporter.a on smithi003 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:04:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:13 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:04:14.181 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:04:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafana-apiserver t=2024-03-21T14:04:13.831856954Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager" 2024-03-21T14:04:14.430 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:14 smithi003 systemd[1]: Starting Ceph node-exporter.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:04:14.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:15.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:14 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:15.109 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:14 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:15.358 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:15 smithi003 bash[76707]: Trying to pull quay.io/prometheus/node-exporter:v1.7.0... 2024-03-21T14:04:15.711 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:15 smithi003 bash[76707]: Getting image source signatures 2024-03-21T14:04:15.711 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:15 smithi003 bash[76707]: Copying blob sha256:324153f2810a9927fcce320af9e4e291e0b6e805cbdd1f338386c756b9defa24 2024-03-21T14:04:15.711 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:15 smithi003 bash[76707]: Copying blob sha256:2abcce694348cd2c949c0e98a7400ebdfd8341021bcf6b541bc72033ce982510 2024-03-21T14:04:15.711 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:15 smithi003 bash[76707]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: 5.1b deep-scrub starts 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: 5.1b deep-scrub ok 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:04:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: 5.1b deep-scrub starts 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: 5.1b deep-scrub ok 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: 5.1b deep-scrub starts 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: 5.1b deep-scrub ok 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: osdmap e88: 8 total, 8 up, 8 in 2024-03-21T14:04:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:16.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:04:16.712 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:04:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:04:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:04:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:04:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-03-21T14:04:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:04:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:04:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:04:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:04:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:16 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:04:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:04:17.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:04:17.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:16 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:04:17.657 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-03-21T14:04:17.657 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-03-21T14:04:17.657 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-03-21T14:04:17.657 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T14:04:17.657 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T14:04:17.657 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:04:17.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3827276325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:04:17.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:17 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3827276325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:04:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:04:18.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:04:18.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:04:18.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:17 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3827276325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T14:04:18.379 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:18 smithi003 bash[76707]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-03-21T14:04:18.379 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:18 smithi003 bash[76707]: Writing manifest to image destination 2024-03-21T14:04:19.004 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:18 smithi003 podman[76707]: 2024-03-21 14:04:18.737820597 +0000 UTC m=+3.755291220 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-03-21T14:04:19.004 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:18 smithi003 podman[76707]: 2024-03-21 14:04:18.961058715 +0000 UTC m=+3.978529318 container create d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: 3.13 deep-scrub starts 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: 3.13 deep-scrub ok 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: 3.13 deep-scrub starts 2024-03-21T14:04:19.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: 3.13 deep-scrub ok 2024-03-21T14:04:19.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:04:19.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:18 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:04:19.132 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-21T14:04:19.142 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T14:04:19.146 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T14:04:19.146 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-21T14:04:19.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-21T14:04:19.147 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-21T14:04:19.147 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:04:19.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T14:04:19.148 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-21T14:04:19.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:19.148 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-21T14:04:19.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:04:19.149 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-21T14:04:19.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:04:19.150 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-21T14:04:19.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:04:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: 3.13 deep-scrub starts 2024-03-21T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: 3.13 deep-scrub ok 2024-03-21T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:04:19.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:18 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:04:19.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:19.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:19.352 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 podman[76707]: 2024-03-21 14:04:19.255884725 +0000 UTC m=+4.273355337 container init d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T14:04:19.352 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 podman[76707]: 2024-03-21 14:04:19.260513489 +0000 UTC m=+4.277984101 container start d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T14:04:19.352 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.266Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-03-21T14:04:19.352 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.266Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-03-21T14:04:19.352 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.267Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.267Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T14:04:19.353 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T14:04:19.354 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T14:04:19.355 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.268Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-03-21T14:04:19.356 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a[76901]: ts=2024-03-21T14:04:19.269Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T14:04:19.357 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 bash[76707]: d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 2024-03-21T14:04:19.357 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:04:19 smithi003 systemd[1]: Started Ceph node-exporter.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:04:19.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:19.457 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:19.482 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:19.484 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:19.523 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:19.523 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:19.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:19.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:19.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:19.683 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:19.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:19.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:19.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:19.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:04:19.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:04:19.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:04:19.866 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:19.867 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:19.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:19.956 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:19.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:20.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:20.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:20.114 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:20.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:20.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:20.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:20.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:20.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:20.433 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:20.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:20.558 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:20.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:20.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:20.809 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[29576]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[29576]: Deploying daemon node-exporter.b on smithi118 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[40510]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:20.810 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:20 smithi003 ceph-mon[40510]: Deploying daemon node-exporter.b on smithi118 2024-03-21T14:04:20.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:20.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.835 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:20.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:20 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:04:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:20 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:04:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:20 smithi118 ceph-mon[33862]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:20 smithi118 ceph-mon[33862]: Deploying daemon node-exporter.b on smithi118 2024-03-21T14:04:20.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:20.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:20.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:21.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:21.085 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:21.117 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:21.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:21.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:21.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:21.338 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:21.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:21.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:21.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:21.680 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:21 smithi118 bash[65215]: Copying blob sha256:455fd88e5221bc1e278ef2d059cd70e4df99a24e5af050ede621534276f6cf9a 2024-03-21T14:04:21.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:21.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:21.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:21.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:21 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:04:21.846 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:21 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:04:21.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:21 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:04:21.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:21 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:04:21.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:21.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:21.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:22.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:22.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:22.160 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:22.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:21 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:04:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:21 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:04:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:22.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:22.308 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:22.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:22.434 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:22.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:22.536 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:22.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:22.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:22.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:22.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:22.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:22.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:22.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:22.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:22.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:23.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:22 smithi003 ceph-mon[29576]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:23.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:22 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:04:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:22 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:04:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:22 smithi003 ceph-mon[40510]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:22 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:04:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:22 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:04:23.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:23.047 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:22 smithi118 ceph-mon[33862]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:23.047 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:22 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:04:23.047 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:22 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:04:23.047 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:22 smithi118 bash[65215]: Copying config sha256:72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e 2024-03-21T14:04:23.048 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:22 smithi118 bash[65215]: Writing manifest to image destination 2024-03-21T14:04:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.064 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:23.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:23.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.189 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:23.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:23.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:23.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.314 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:23.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:23.407 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 podman[65215]: 2024-03-21 14:04:23.158549505 +0000 UTC m=+2.530286751 image pull 72c9c208898624938c9e4183d6686ea4a5fd3f912bc29bc3f00147924c521a3e quay.io/prometheus/node-exporter:v1.7.0 2024-03-21T14:04:23.407 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 podman[65215]: 2024-03-21 14:04:23.282736304 +0000 UTC m=+2.654473589 container create a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T14:04:23.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:23.438 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:23.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:23.563 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:23.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:23.664 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:23.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:23.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:23.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:23.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:23.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:23.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:23.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:23.931 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 podman[65215]: 2024-03-21 14:04:23.527005223 +0000 UTC m=+2.898742465 container init a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 podman[65215]: 2024-03-21 14:04:23.529838803 +0000 UTC m=+2.901576046 container start a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.533Z caller=node_exporter.go:192 level=info msg="Starting node_exporter" version="(version=1.7.0, branch=HEAD, revision=7333465abf9efba81876303bb57e6fadb946041b)" 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.533Z caller=node_exporter.go:193 level=info msg="Build context" build_context="(go=go1.21.4, platform=linux/amd64, user=root@35918982f6d8, date=20231112-23:53:35, tags=netgo osusergo static_build)" 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.533Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=arp 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-21T14:04:23.932 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-21T14:04:23.933 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-21T14:04:23.933 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-21T14:04:23.933 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-21T14:04:23.933 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-21T14:04:23.933 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=edac 2024-03-21T14:04:23.933 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.534Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-21T14:04:23.934 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=os 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=stat 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=time 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-21T14:04:23.935 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=uname 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9100 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b[65471]: ts=2024-03-21T14:04:23.535Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 bash[65215]: a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 2024-03-21T14:04:23.936 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:04:23 smithi118 systemd[1]: Started Ceph node-exporter.b for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:04:23.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:23.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:23.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:04:24.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:04:24.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:04:24.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:04:24.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:24.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:24.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:24.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:24.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:24.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:24.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:24.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:24.320 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:24.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:24.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:24.445 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:24.547 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:24.571 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:24.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:24.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:24.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:24.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:24.823 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:24.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:24.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:24.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:24 smithi118 ceph-mon[33862]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:24 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:04:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:24 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:04:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:24 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:04:24.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:24.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:24.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[29576]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:04:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:04:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:04:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[40510]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:04:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:04:25.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:04:25.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:24 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:04:25.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:25.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:25.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:25.202 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:25.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:25.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:25.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.329 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:25.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:25.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:25.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:25.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:25.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.583 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:25.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:25.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:25.809 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:25.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:25.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:25.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:25.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:26.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:26.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:26.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:26.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:26.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:26.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:26.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:26.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:26.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:26.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[29576]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-03-21T14:04:26.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[29576]: 6.a deep-scrub starts 2024-03-21T14:04:26.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[29576]: 6.a deep-scrub ok 2024-03-21T14:04:26.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:26.549 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:04:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:04:26.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[40510]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-03-21T14:04:26.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[40510]: 6.a deep-scrub starts 2024-03-21T14:04:26.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[40510]: 6.a deep-scrub ok 2024-03-21T14:04:26.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:26 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:26.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:26.570 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:26 smithi118 ceph-mon[33862]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 969 B/s rd, 0 op/s 2024-03-21T14:04:26.571 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:26 smithi118 ceph-mon[33862]: 6.a deep-scrub starts 2024-03-21T14:04:26.571 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:26 smithi118 ceph-mon[33862]: 6.a deep-scrub ok 2024-03-21T14:04:26.571 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:26 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:26.571 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:04:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:04:26.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.588 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:26.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:26.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:26.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:26.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:26.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:26.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:26.966 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:26.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:26.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:27.050 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:04:27.050 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:04:27.050 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-03-21T14:04:27.050 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:04:27.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:27.091 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:27.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:27.218 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:27.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:27.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:27.344 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:27.357 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T14:04:27.358 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' created 2024-03-21T14:04:27.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:27.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3326902894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/72920662' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:04:27.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:04:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:04:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3326902894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T14:04:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T14:04:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/72920662' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:04:27.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:27 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:04:27.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:27.597 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:27.662 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:04:27.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:04:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:04:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3326902894' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T14:04:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-21T14:04:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/72920662' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:04:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:27 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:04:27.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:04:27.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:27.701 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:04:27.722 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:04:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:27.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:27.743 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:04:27.763 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:04:27.784 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:04:27.804 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:04:27.805 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:04:27.806 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:04:27.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:27.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:27.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:27.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:27.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:28.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.100 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:28.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:28.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:28.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:28.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:28.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:28.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:28.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:28.367 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:04:28.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:04:28.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: 6.17 scrub starts 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: 6.17 scrub ok 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2363141896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:04:28.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1991538366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: 6.17 scrub starts 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: 6.17 scrub ok 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T14:04:28.460 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:04:28.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:04:28.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2363141896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:04:28.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1991538366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:04:28.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:04:28.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:28.461 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:28 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:28.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:28.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:28.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: 6.17 scrub starts 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: 6.17 scrub ok 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: osdmap e89: 8 total, 8 up, 8 in 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2363141896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1991538366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:04:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:04:28.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:28.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:28 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:28.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:28.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:28.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:28.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:28.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:28.985 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:28.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:29.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:29.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:29.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:29.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:29.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.236 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:29.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:29.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:29.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-21T14:04:29.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:29.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:29.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:29.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:29.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:29.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:29.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:29 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:04:29.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:29 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:04:29.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:29 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:04:29.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:29 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:04:29.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:29 smithi118 ceph-mon[33862]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T14:04:29.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:04:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:04:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:04:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:04:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[29576]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T14:04:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:04:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:04:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:04:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:04:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:29 smithi003 ceph-mon[40510]: osdmap e90: 8 total, 8 up, 8 in 2024-03-21T14:04:29.756 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:29.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:29.882 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:29.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:29.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:30.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:30.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.135 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:30.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:30.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:30.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:30.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:30.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:30.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:30 smithi003 ceph-mon[29576]: pgmap v53: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:30.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:30 smithi003 ceph-mon[29576]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:04:30.458 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:30 smithi003 ceph-mon[29576]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T14:04:30.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:30 smithi003 ceph-mon[40510]: pgmap v53: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:30.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:30 smithi003 ceph-mon[40510]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:04:30.458 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:30 smithi003 ceph-mon[40510]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T14:04:30.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:30.512 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:30.554 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:30.561 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:30.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:30.637 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:30 smithi118 ceph-mon[33862]: pgmap v53: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:30 smithi118 ceph-mon[33862]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:04:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:30 smithi118 ceph-mon[33862]: osdmap e91: 8 total, 8 up, 8 in 2024-03-21T14:04:30.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:30.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:30.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:30.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:30.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:30.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-21T14:04:30.990 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:31.014 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:31.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:31.140 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:31.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.267 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:31.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:31.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:31.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:31.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:31 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:31.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:31 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:31.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:31.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:31.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:31.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:31 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:31.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:31.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:31.769 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:31.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:31.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:04:31.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:31.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:31.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:31.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:32.021 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:32.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:32.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:32.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:32.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:32.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:32.399 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:32.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:32.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:04:32.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:32.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:32 smithi003 ceph-mon[29576]: pgmap v55: 148 pgs: 2 unknown, 146 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:04:32.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:32 smithi003 ceph-mon[40510]: pgmap v55: 148 pgs: 2 unknown, 146 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:04:32.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:32.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:32.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:32.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:32.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:32 smithi118 ceph-mon[33862]: pgmap v55: 148 pgs: 2 unknown, 146 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:04:32.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:32.776 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:32.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:32.902 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:32.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:33.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:33.027 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:33.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:33.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:33.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:33.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.280 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:33.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:33.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:33.404 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:33.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.531 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:33.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:33.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:33.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:33.694 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:33.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:33.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:33.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:33.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:33.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:33.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:34.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:34.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:34.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:34.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:34.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:34.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:34.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:34.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:34.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:34 smithi003 ceph-mon[29576]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:34.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:34 smithi003 ceph-mon[40510]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:34.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.545 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:34.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:34.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.673 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:34.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:34.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:34 smithi118 ceph-mon[33862]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:04:34.764 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:04:34.764 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:04:34.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:34.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:34.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:34.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:35.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:35.060 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:35.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:35.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.191 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:35.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:35.292 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:35.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:35.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:35.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:35.447 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:35.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:35.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:35.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:35.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:35.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:04:35.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:35.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:35.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:35.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:35.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:36.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:36.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:36.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:36.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:36.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-21T14:04:36.439 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:36.465 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:36.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:36.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:36.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:36.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:04:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:04:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:36 smithi118 ceph-mon[33862]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-21T14:04:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:36 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3973753939' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:04:36.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:36 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:04:36.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:36.720 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:36.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-mon[29576]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-21T14:04:36.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3973753939' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:04:36.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:04:36.738 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:04:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:04:36.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-mon[40510]: pgmap v57: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 647 B/s rd, 0 op/s 2024-03-21T14:04:36.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3973753939' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:04:36.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:36 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:04:36.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.850 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:36.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:36.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:36.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:36.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:37.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:37.107 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:37.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:37.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:37.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:37.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:37.363 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:37.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:37.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:37.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:37.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:37.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:37.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:37 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:04:37.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:37 smithi118 ceph-mon[33862]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T14:04:37.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:37.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:37 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:04:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:37 smithi003 ceph-mon[29576]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T14:04:37.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:37.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:37 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:04:37.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:37 smithi003 ceph-mon[40510]: osdmap e92: 8 total, 8 up, 8 in 2024-03-21T14:04:37.795 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:37.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:37.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:37.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:38.003 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:38.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:38.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:38.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:38.233 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:38.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:38.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:38.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:38.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:38.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:38.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:38.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:38.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:38 smithi118 ceph-mon[33862]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:38.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:38 smithi118 ceph-mon[33862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:04:38.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:38.747 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:38 smithi003 ceph-mon[29576]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:38 smithi003 ceph-mon[29576]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:04:38.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:38 smithi003 ceph-mon[40510]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:04:38.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:38 smithi003 ceph-mon[40510]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:04:38.774 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:38.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:38.876 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:38.903 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:38.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:38.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:39.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:39.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:39.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:39.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:39.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:39.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:39.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:39.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:39.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:39.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:39.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:39.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:39.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T14:04:39.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:04:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:39 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:39.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:39.636 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:39.664 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:39.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: osdmap e93: 8 total, 8 up, 8 in 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:04:39.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:39 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:39.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:39.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:39.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:39.916 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:39.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:40.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:40.042 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:40.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:40.167 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:40.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:40.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:40.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:40.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:40.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:40.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:40.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:40 smithi003 ceph-mon[29576]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T14:04:40.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:40 smithi003 ceph-mon[29576]: Reconfiguring daemon alertmanager.a on smithi003 2024-03-21T14:04:40.607 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:40 smithi003 ceph-mon[29576]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:40.607 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:40 smithi003 ceph-mon[40510]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T14:04:40.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:40 smithi003 ceph-mon[40510]: Reconfiguring daemon alertmanager.a on smithi003 2024-03-21T14:04:40.608 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:40 smithi003 ceph-mon[40510]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:40.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:40.667 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-21T14:04:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:40.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:40.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:40 smithi118 ceph-mon[33862]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-21T14:04:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:40 smithi118 ceph-mon[33862]: Reconfiguring daemon alertmanager.a on smithi003 2024-03-21T14:04:40.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:40 smithi118 ceph-mon[33862]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:04:40.712 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T14:04:40.712 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T14:04:40.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:40.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-03-21T14:04:40.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T14:04:40.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_80362 2024-03-21T14:04:40.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:04:40.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-03-21T14:04:40.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:40.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:40.947 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:40.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:40.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:41.049 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:41.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:41.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:41.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:41.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:41.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:41.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:41.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:41.442 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3189353622' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:04:41.449 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-03-21T14:04:41.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:41.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:41.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:41.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:41.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:41 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:04:41.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3189353622' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:04:41.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:41 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:04:41.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:41.778 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3094 3078 192.364 192.375 0.00526351 0.00518208 2024-03-21T14:04:41.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:41.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:41.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:41.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3189353622' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:04:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:41 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:04:41.942 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:41.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:41.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:42.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:42.075 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:42.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:42.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:42.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:42.322 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:42.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:42.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T14:04:42.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:42.485 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:42.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:42.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.618 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:42.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:42.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:42.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:42.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:42.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:42 smithi003 ceph-mon[40510]: pgmap v62: 148 pgs: 148 active+clean; 76 MiB data, 347 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.5 MiB/s wr, 152 op/s 2024-03-21T14:04:42.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:04:42.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:42 smithi003 ceph-mon[40510]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T14:04:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:42 smithi003 ceph-mon[29576]: pgmap v62: 148 pgs: 148 active+clean; 76 MiB data, 347 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.5 MiB/s wr, 152 op/s 2024-03-21T14:04:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:04:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:42 smithi003 ceph-mon[29576]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T14:04:42.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6208 6192 193.474 194.625 0.00491715 0.00515748 2024-03-21T14:04:42.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:42.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:42.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:42.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:42.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:42 smithi118 ceph-mon[33862]: pgmap v62: 148 pgs: 148 active+clean; 76 MiB data, 347 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 9.5 MiB/s wr, 152 op/s 2024-03-21T14:04:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:04:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:42 smithi118 ceph-mon[33862]: osdmap e94: 8 total, 8 up, 8 in 2024-03-21T14:04:42.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:43.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:43.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:43.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:43.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:43.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:43.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:43.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:43.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:43.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:43.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:43.778 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 8869 8853 184.415 166.312 0.00396706 0.00541484 2024-03-21T14:04:43.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:43.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:43.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:43.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:44.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:44.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:44.150 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:44.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:44.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:44.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:44.425 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:44.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:44.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:44.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:44.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:44.701 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:44.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:44 smithi003 ceph-mon[29576]: pgmap v64: 148 pgs: 148 active+clean; 88 MiB data, 367 MiB used, 715 GiB / 715 GiB avail; 747 B/s rd, 13 MiB/s wr, 205 op/s 2024-03-21T14:04:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:44 smithi003 ceph-mon[40510]: pgmap v64: 148 pgs: 148 active+clean; 88 MiB data, 367 MiB used, 715 GiB / 715 GiB avail; 747 B/s rd, 13 MiB/s wr, 205 op/s 2024-03-21T14:04:44.777 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 11692 11676 182.417 176.438 0.00190433 0.00546115 2024-03-21T14:04:44.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:44.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:44.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:44 smithi118 ceph-mon[33862]: pgmap v64: 148 pgs: 148 active+clean; 88 MiB data, 367 MiB used, 715 GiB / 715 GiB avail; 747 B/s rd, 13 MiB/s wr, 205 op/s 2024-03-21T14:04:44.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:44.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:44.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:44.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:45.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:45.120 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:45.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:45.254 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:45.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:45.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:45.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:45.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:45.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:45.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:45.675 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:45.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:45.777 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:45.778 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14387 14371 179.613 168.438 0.00344865 0.00555986 2024-03-21T14:04:45.810 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:45.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:45.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:45.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:46.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:46.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:46.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:46.200 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:46.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:46.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:46.366 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:04:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:46.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:46.410 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:04:46.464 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:04:46.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:46.510 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:04:46.512 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:46.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.14158.0:130 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'88, mlcod=21'85)", 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:00:17.677078+0000", 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 268.91398090899997, 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018280760000000001, 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677078+0000", 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677078+0000", 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677078+0000", 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-07 2024-03-21T14:04:46.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677085+0000", 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2559999999999996e-06 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677099+0000", 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3472000000000001e-05 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677106+0000", 2024-03-21T14:04:46.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9110000000000004e-06 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677205+0000", 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9386e-05 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.677222+0000", 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6668999999999999e-05 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678866+0000", 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001644118 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678906+0000", 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9722999999999999e-05 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.14158.0:132 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'90, mlcod=21'85)", 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:00:17.678341+0000", 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 268.91271719500003, 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001583744, 2024-03-21T14:04:46.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678341+0000", 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678341+0000", 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678342+0000", 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.39e-07 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678346+0000", 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7319999999999998e-06 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678353+0000", 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8630000000000002e-06 2024-03-21T14:04:46.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678363+0000", 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0064000000000001e-05 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678684+0000", 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032114600000000002 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.678704+0000", 2024-03-21T14:04:46.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9950999999999999e-05 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.679876+0000", 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011715059999999999 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.679925+0000", 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9042999999999998e-05 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.14158.0:136 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'94, mlcod=21'89)", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:00:17.681060+0000", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 268.90999831900001, 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002040591, 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681060+0000", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681060+0000", 2024-03-21T14:04:46.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681061+0000", 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5300000000000004e-07 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681069+0000", 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8420000000000005e-06 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681080+0000", 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1618e-05 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681087+0000", 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2429999999999999e-06 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681151+0000", 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3238999999999995e-05 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681167+0000", 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6466000000000001e-05 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.683051+0000", 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001883888 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.683101+0000", 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9641999999999997e-05 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.14158.0:137 1.0 e21/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 21'95, mlcod=21'89)", 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:00:17.681618+0000", 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 268.90944073399999, 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019526750000000001, 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681618+0000", 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681618+0000", 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681618+0000", 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0900000000000001e-07 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681642+0000", 2024-03-21T14:04:46.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3779999999999999e-05 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681653+0000", 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0883999999999999e-05 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681662+0000", 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1710000000000008e-06 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681705+0000", 2024-03-21T14:04:46.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3027e-05 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.681729+0000", 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3688000000000001e-05 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.683528+0000", 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017986969999999999 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:17.683570+0000", 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2719000000000002e-05 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.14445.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e64)", 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:33.393426+0000", 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 133.197632486, 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046719810000000001, 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.613 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.14445", 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/3739272181", 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 1 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393426+0000", 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393426+0000", 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393434+0000", 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5019999999999992e-06 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393440+0000", 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7150000000000003e-06 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393450+0000", 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7489999999999993e-06 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393459+0000", 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4560000000000006e-06 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393518+0000", 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8224999999999999e-05 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.393526+0000", 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5979999999999997e-06 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.440042+0000", 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046515869000000001 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.440146+0000", 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103696 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24326.0:2 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:33.436574+0000", 2024-03-21T14:04:46.617 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 133.15448489900001, 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003675981, 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24326", 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2605341271", 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 2 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.436574+0000", 2024-03-21T14:04:46.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.436574+0000", 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.436575+0000", 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.403e-06 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.436580+0000", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.203e-06 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.436591+0000", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1189000000000001e-05 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.436602+0000", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.065e-05 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.437165+0000", 2024-03-21T14:04:46.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056270599999999995 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.437175+0000", 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0538e-05 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.440172+0000", 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029972150000000001 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:33.440250+0000", 2024-03-21T14:04:46.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7076999999999994e-05 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24365.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 94595922646576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:37.471252+0000", 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 129.119806681, 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013214979999999999, 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.622 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:46.623 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24365", 2024-03-21T14:04:46.623 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2761509089", 2024-03-21T14:04:46.623 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 57 2024-03-21T14:04:46.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.623 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471252+0000", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471252+0000", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471253+0000", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0200000000000001e-07 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471257+0000", 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3270000000000002e-06 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471265+0000", 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5010000000000004e-06 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471273+0000", 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3039999999999999e-06 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471379+0000", 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106407 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471449+0000", 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9435000000000002e-05 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471597+0000", 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000147999 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472275+0000", 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00067828599999999995 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472318+0000", 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2843999999999998e-05 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472426+0000", 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010786299999999999 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472486+0000", 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0854999999999999e-05 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472573+0000", 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6874999999999997e-05 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:57 5.3 5:c609908c:::notify.5:head [watch watch cookie 94450053596288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:37.471542+0000", 2024-03-21T14:04:46.628 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 129.11951634299999, 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001906589, 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 57 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471542+0000", 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471542+0000", 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471543+0000", 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2200000000000002e-07 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471557+0000", 2024-03-21T14:04:46.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4047e-05 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471569+0000", 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1829e-05 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.471578+0000", 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6309999999999992e-06 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472033+0000", 2024-03-21T14:04:46.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00045580300000000001 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for rw locks", 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472148+0000", 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114969 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472609+0000", 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046109299999999998 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472678+0000", 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8813000000000007e-05 2024-03-21T14:04:46.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.472777+0000", 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8754999999999999e-05 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.473116+0000", 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033897199999999999 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.473323+0000", 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020728399999999999 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.473381+0000", 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7571999999999997e-05 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.473411+0000", 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0144000000000001e-05 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:37.473449+0000", 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7755e-05 2024-03-21T14:04:46.634 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24350.0:206 4.19 e71/66 4:9a6dd8c0:gc::gc.0:head v 71'1, mlcod=0'0)", 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.463768+0000", 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.127290284, 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014503700000000001, 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463768+0000", 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463768+0000", 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.636 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463770+0000", 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.595e-06 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463781+0000", 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1382e-05 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463800+0000", 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8981e-05 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463810+0000", 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9369999999999999e-06 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463918+0000", 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108063 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.463951+0000", 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3090999999999999e-05 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.465167+0000", 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012160820000000001 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.465219+0000", 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1239000000000002e-05 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:208 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.469003+0000", 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.122055737, 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038534709999999998, 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 208 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469003+0000", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469003+0000", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469005+0000", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2579999999999998e-06 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469014+0000", 2024-03-21T14:04:46.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3209999999999994e-06 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469035+0000", 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0749000000000001e-05 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469046+0000", 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0831e-05 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469116+0000", 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9739000000000004e-05 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.469120+0000", 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2980000000000002e-06 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.471480+0000", 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023602139999999998 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.471650+0000", 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170011 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.471937+0000", 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028654400000000001 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.472493+0000", 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055665499999999998 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.472734+0000", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024078599999999999 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.472780+0000", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6385999999999999e-05 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.472820+0000", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9932999999999999e-05 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.472856+0000", 2024-03-21T14:04:46.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5746e-05 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:317 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.523575+0000", 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.067483924, 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0018338079999999999, 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 317 2024-03-21T14:04:46.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523575+0000", 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523575+0000", 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523577+0000", 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3420000000000002e-06 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523588+0000", 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1432e-05 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523610+0000", 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1586999999999998e-05 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523621+0000", 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0791000000000001e-05 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523697+0000", 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5963000000000004e-05 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.523824+0000", 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127384 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.524260+0000", 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043600399999999998 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.524709+0000", 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044868799999999998 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525045+0000", 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033643600000000002 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525318+0000", 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027331300000000002 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525368+0000", 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9795e-05 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525408+0000", 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0073000000000001e-05 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:323 4.3 4:c3a682d4:reshard::reshard.0000000004:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.651 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.525918+0000", 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.065140625, 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013439179999999999, 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 323 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525918+0000", 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525918+0000", 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525919+0000", 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7400000000000002e-07 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525924+0000", 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8559999999999998e-06 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525949+0000", 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.563e-05 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.525957+0000", 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.038e-06 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.526001+0000", 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3823999999999999e-05 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.526069+0000", 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8194999999999993e-05 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.526214+0000", 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014460699999999999 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.526827+0000", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00061316300000000001 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527005+0000", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017846600000000001 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527173+0000", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000167271 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527223+0000", 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0263999999999999e-05 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527262+0000", 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8729999999999997e-05 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:329 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.527425+0000", 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.06363373400001, 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001521199, 2024-03-21T14:04:46.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 329 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527425+0000", 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527425+0000", 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527426+0000", 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.387e-06 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527432+0000", 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6570000000000002e-06 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527444+0000", 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2469999999999999e-05 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527452+0000", 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.943e-06 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527510+0000", 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8208000000000002e-05 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527619+0000", 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108517 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.527783+0000", 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016425200000000001 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.528199+0000", 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041552499999999998 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.528503+0000", 2024-03-21T14:04:46.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030457299999999999 2024-03-21T14:04:46.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.528851+0000", 2024-03-21T14:04:46.663 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:04:46.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000348241 2024-03-21T14:04:46.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.528896+0000", 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4817999999999997e-05 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.528946+0000", 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9607999999999997e-05 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:333 4.6 4:66924383:reshard::reshard.0000000005:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.664 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.529469+0000", 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.061589021, 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019569399999999999, 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 333 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529469+0000", 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529469+0000", 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529470+0000", 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1100000000000004e-07 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529477+0000", 2024-03-21T14:04:46.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0879999999999997e-06 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529492+0000", 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4993e-05 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529503+0000", 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0871e-05 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529553+0000", 2024-03-21T14:04:46.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9815000000000003e-05 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529627+0000", 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3769999999999993e-05 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.529830+0000", 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020293000000000001 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.530201+0000", 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000371591 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.530540+0000", 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033868 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531316+0000", 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00077568999999999999 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531373+0000", 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7499e-05 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531426+0000", 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3102000000000001e-05 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.670 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:335 4.16 4:6dc3b799:reshard::reshard.0000000006:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.531597+0000", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.05946121700001, 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001577059, 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 335 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531597+0000", 2024-03-21T14:04:46.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531597+0000", 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531599+0000", 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4309999999999999e-06 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531606+0000", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1289999999999998e-06 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531622+0000", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6342e-05 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531630+0000", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0989999999999999e-06 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531694+0000", 2024-03-21T14:04:46.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3295999999999998e-05 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.531829+0000", 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013594 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.532032+0000", 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020224699999999999 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.532393+0000", 2024-03-21T14:04:46.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000361768 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.532795+0000", 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040157399999999997 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.533090+0000", 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029475699999999998 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.533136+0000", 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5803000000000002e-05 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.533174+0000", 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8673000000000001e-05 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:378 4.6 4:64760185:::obj_delete_at_hint.0000000010:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.676 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.546305+0000", 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[29576]: pgmap v65: 148 pgs: 148 active+clean; 88 MiB data, 367 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 11 MiB/s wr, 175 op/s 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1115439317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[40510]: pgmap v65: 148 pgs: 148 active+clean; 88 MiB data, 367 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 11 MiB/s wr, 175 op/s 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:46.677 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1115439317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:04:46.678 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:04:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:04:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.044753688, 2024-03-21T14:04:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015775679999999999, 2024-03-21T14:04:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.678 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 378 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546305+0000", 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546305+0000", 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546306+0000", 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4670000000000001e-06 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546321+0000", 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4421e-05 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546336+0000", 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4960000000000001e-05 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546344+0000", 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2139999999999996e-06 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546404+0000", 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0622999999999999e-05 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546532+0000", 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012742100000000001 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.546723+0000", 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019148599999999999 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.547086+0000", 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036274799999999998 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.547643+0000", 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055679999999999998 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.547783+0000", 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013989800000000001 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.547838+0000", 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4863e-05 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.547882+0000", 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4666999999999998e-05 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:443 4.b 4:d242335b:gc::gc.2:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.565570+0000", 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 126.02548859300001, 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014428939999999999, 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 443 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565570+0000", 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565570+0000", 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565572+0000", 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.902e-06 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565582+0000", 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0332e-05 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565602+0000", 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9983000000000002e-05 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565615+0000", 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.306e-05 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565691+0000", 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6071999999999999e-05 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565787+0000", 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5872000000000006e-05 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.565969+0000", 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018177699999999999 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.566315+0000", 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000346592 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.566621+0000", 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030551800000000001 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.566907+0000", 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028567499999999999 2024-03-21T14:04:46.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.566967+0000", 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0229999999999998e-05 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.567013+0000", 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5881000000000003e-05 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.689 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24350.0:600 4.16 4:6849fd06:gc::gc.17:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.624223+0000", 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.966835484, 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001400711, 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24350", 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 600 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624223+0000", 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624223+0000", 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624224+0000", 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2999999999999999e-07 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624229+0000", 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2599999999999996e-06 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624243+0000", 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4399000000000001e-05 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624251+0000", 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8229999999999995e-06 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624299+0000", 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7695000000000003e-05 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624365+0000", 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5641000000000004e-05 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624471+0000", 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106109 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.624758+0000", 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028775900000000001 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.625144+0000", 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00038526700000000001 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.625559+0000", 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041483999999999998 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.625587+0000", 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8629e-05 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.625624+0000", 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6458999999999998e-05 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24379.0:54 5.3 5:c609908c:::notify.5:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-03-21T14:04:46.695 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:02:40.630467+0000", 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.960591869, 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001538202, 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24379", 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/781640920", 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 54 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630467+0000", 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630467+0000", 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630468+0000", 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4100000000000001e-06 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630473+0000", 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9130000000000002e-06 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630487+0000", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4090000000000001e-05 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630495+0000", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3869999999999995e-06 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630555+0000", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.961e-05 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630630+0000", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4949000000000004e-05 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.630734+0000", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010378999999999999 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.631686+0000", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00095199599999999998 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.631733+0000", 2024-03-21T14:04:46.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7122e-05 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.631887+0000", 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015423800000000001 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.631949+0000", 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.211e-05 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:02:40.632005+0000", 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5587000000000002e-05 2024-03-21T14:04:46.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDScrubReserve(5.e,QREQUEST,e:85,from:4,reservation_nonce:2)", 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:03:42.994360+0000", 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 63.596698662999998, 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023280890000000002, 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994360+0000", 2024-03-21T14:04:46.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994360+0000", 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994362+0000", 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0590000000000001e-06 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:04:46.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994370+0000", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6389999999999998e-06 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994394+0000", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3059999999999999e-05 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994405+0000", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1498000000000001e-05 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994492+0000", 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7187000000000004e-05 2024-03-21T14:04:46.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:04:46.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:46.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.994502+0000", 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7110000000000007e-06 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.996641+0000", 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0021393900000000001 2024-03-21T14:04:46.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.996655+0000", 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3777e-05 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:03:42.996688+0000", 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2768000000000002e-05 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:04:46.706 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:04:46.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:04:46.708 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:04:46.708 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:04:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:04:46.708 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:46 smithi118 ceph-mon[33862]: pgmap v65: 148 pgs: 148 active+clean; 88 MiB data, 367 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 11 MiB/s wr, 175 op/s 2024-03-21T14:04:46.708 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:04:46.708 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:04:46.708 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:46 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1115439317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:04:46.778 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 16553 16537 172.236 135.375 0.0102439 0.00579825 2024-03-21T14:04:46.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:46.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:46.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:46.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:46.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:47.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:47.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:47.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:47.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:47.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:47.392 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:47.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:47.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:47.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:47.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:47.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:47.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:47.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:47.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:47.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:47.778 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 18974 18958 169.245 151.312 0.0056611 0.00590066 2024-03-21T14:04:47.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:47.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:47.911 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:47.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:47.942 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:47.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:48.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:48.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:48.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:48.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:48.329 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:48.352 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:48.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:48.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:48.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:48.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:48.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:48.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:48.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:48 smithi003 ceph-mon[29576]: pgmap v66: 148 pgs: 148 active+clean; 967 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-21T14:04:48.750 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:48 smithi003 ceph-mon[40510]: pgmap v66: 148 pgs: 148 active+clean; 967 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-21T14:04:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:48.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:48.778 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 21211 21195 165.556 139.812 0.00550807 0.00603398 2024-03-21T14:04:48.871 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:48.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:48.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:48.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:48 smithi118 ceph-mon[33862]: pgmap v66: 148 pgs: 148 active+clean; 967 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-03-21T14:04:48.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:49.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:49.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:49.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:49.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:49.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:49.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:49.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:49.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:49.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:49.595 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:49.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:49.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:49.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:49.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:49 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:49.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:49 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:49.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3670878416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:04:49.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:49 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:49.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:49 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:49.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3670878416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:04:49.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 22912 22896 158.973 106.312 0.00237647 0.00628492 2024-03-21T14:04:49.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:49.854 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:49.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:49.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:04:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:49 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:49 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3670878416' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:04:49.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:50.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:50.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:50.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:50.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:50.299 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:50.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:50.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:50.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:50.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:50.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:50.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:50.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:50 smithi118 ceph-mon[33862]: pgmap v67: 148 pgs: 148 active+clean; 967 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-21T14:04:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:50 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:04:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3670878416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:04:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:50 smithi118 ceph-mon[33862]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T14:04:50.699 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:50.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[29576]: pgmap v67: 148 pgs: 148 active+clean; 967 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-21T14:04:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:04:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3670878416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:04:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[29576]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T14:04:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[40510]: pgmap v67: 148 pgs: 148 active+clean; 967 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-21T14:04:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:04:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3670878416' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:04:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:50 smithi003 ceph-mon[40510]: osdmap e95: 8 total, 8 up, 8 in 2024-03-21T14:04:50.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 24714 24698 154.337 112.625 0.00204092 0.00646433 2024-03-21T14:04:50.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:50.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:50.925 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:50.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:50.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:51.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:51.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:51.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:51.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:51.218 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:51.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:51.352 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:51.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:51.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:51.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:51.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:51.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:51.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:51 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:51.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:51 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:51.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:51 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:51.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:51 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:51.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:51.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 26698 26682 151.579 124 0.0096647 0.00659085 2024-03-21T14:04:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:04:51.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:51.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:51.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:51.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:51 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:51 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:52.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:52.052 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:52.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:52.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:52.188 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:52.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:52.322 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:52.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:52.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:52.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:52.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:52.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:52.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[29576]: pgmap v69: 148 pgs: 148 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[29576]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[40510]: pgmap v69: 148 pgs: 148 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-21T14:04:52.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:52 smithi003 ceph-mon[40510]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T14:04:52.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 28723 28707 149.493 126.562 0.00250452 0.0066829 2024-03-21T14:04:52.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:52.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:52.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:52 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:52 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:52 smithi118 ceph-mon[33862]: pgmap v69: 148 pgs: 148 active+clean; 1.5 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-21T14:04:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:52 smithi118 ceph-mon[33862]: osdmap e96: 8 total, 8 up, 8 in 2024-03-21T14:04:52.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:53.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:53.159 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:53.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:53.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:53.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:53.442 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:53.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:53.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:53.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:53.720 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:53.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 30822 30806 148.084 131.188 0.00455213 0.0067464 2024-03-21T14:04:53.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:53.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:53.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:53.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:53.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:54.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:54.128 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:54.230 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:54.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:54.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:54.392 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:54.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:54.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:04:54.476 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:54.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:54.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:54.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:04:54.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:54.655 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:54.670 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:54 smithi003 ceph-mon[40510]: pgmap v71: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 189 MiB/s wr, 3.02k op/s 2024-03-21T14:04:54.670 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:54 smithi003 ceph-mon[29576]: pgmap v71: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 189 MiB/s wr, 3.02k op/s 2024-03-21T14:04:54.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:54.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 32691 32675 145.85 116.812 0.022466 0.00685 2024-03-21T14:04:54.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:54.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:54.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:54 smithi118 ceph-mon[33862]: pgmap v71: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 189 MiB/s wr, 3.02k op/s 2024-03-21T14:04:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:54.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:54.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:55.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:55.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:55.196 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:04:55 smithi003 systemd[1]: Stopping Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:04:55.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:55.257 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:55.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:55.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:55.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:55.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:55.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:55.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:55.666 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:55 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:04:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:55 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:04:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:55 smithi003 ceph-mon[29576]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T14:04:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:55 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:04:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:55 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:04:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:55 smithi003 ceph-mon[40510]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T14:04:55.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:55.779 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 34395 34379 143.226 106.5 0.00831542 0.00697608 2024-03-21T14:04:55.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:55.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:55.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:55.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:55 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:04:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:55 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:04:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:55 smithi118 ceph-mon[33862]: osdmap e97: 8 total, 8 up, 8 in 2024-03-21T14:04:55.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:55.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:56.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:56.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:56.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:56.211 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:56.216 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:56.220 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:56.317 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:56.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:56.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:56.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:56.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:56.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:56.628 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:56.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:56 smithi118 ceph-mon[33862]: pgmap v73: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T14:04:56.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:04:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:04:56.697 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:56 smithi003 ceph-mon[29576]: pgmap v73: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T14:04:56.697 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:04:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:04:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:04:56.697 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:56 smithi003 ceph-mon[40510]: pgmap v73: 148 pgs: 148 active+clean; 1.6 GiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T14:04:56.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:56.745 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:04:56.756 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:56.780 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 36276 36260 141.615 117.562 0.00779155 0.00705439 2024-03-21T14:04:56.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:56.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:57.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:57.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:57.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:57.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:57.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:57.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:57.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:57.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.594 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:57.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:57.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:57.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:57.780 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 15 38125 38110 140.085 115.625 0.00603868 0.00713272 2024-03-21T14:04:57.799 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:57.799 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:57.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:57.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:57.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:57.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:57.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:04:57.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:04:58.003 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:58.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:04:58.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:58.136 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:58.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:58.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:58.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:58.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:58.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:58.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:58.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:04:58.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:04:58.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:04:58 smithi003 ceph-mon[29576]: pgmap v74: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-03-21T14:04:58.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:04:58 smithi003 ceph-mon[40510]: pgmap v74: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-03-21T14:04:58.780 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 39964 39948 138.684 114.875 0.0130359 0.00720566 2024-03-21T14:04:58.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:58.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:58.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:04:58 smithi118 ceph-mon[33862]: pgmap v74: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 159 MiB/s wr, 2.54k op/s 2024-03-21T14:04:58.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:04:58.981 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:58.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:58.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:04:59.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:59.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:59.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:04:59.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:59.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:59.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:59.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:04:59.404 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:59.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:04:59.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:04:59.536 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:04:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:04:59.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:04:59.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:04:59.780 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 41519 41503 136.5 97.1875 0.00194559 0.00731205 2024-03-21T14:04:59.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:04:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:59.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:04:59.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:04:59.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:04:59.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:04:59.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:00.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:00.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:00.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:00.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:00.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:00.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:00.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:00.388 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:00.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:00.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:00.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:00.633 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:00.663 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:00.765 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:00.780 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:05:00.781762+0000 min lat: 0.00127999 max lat: 0.103614 avg lat: 0.00739543 2024-03-21T14:05:00.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:05:00.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 43259 43243 135.112 108.75 0.0119729 0.00739543 2024-03-21T14:05:00.794 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:00.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:00.861 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:00 smithi003 ceph-mon[29576]: pgmap v75: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-21T14:05:00.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:00 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:00.862 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:00 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[75627]: ts=2024-03-21T14:05:00.733Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T14:05:00.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:00 smithi003 ceph-mon[40510]: pgmap v75: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-21T14:05:00.862 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:00 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:00.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:00.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:00 smithi118 ceph-mon[33862]: pgmap v75: 148 pgs: 148 active+clean; 2.2 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-21T14:05:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:00 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:00.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:00.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:01.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:01.077 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:01.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:01.222 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:01 smithi003 podman[82141]: 2024-03-21 14:05:01.017567497 +0000 UTC m=+2.613724826 container died 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:01.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:01.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:01.357 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:01.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:01.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:01.601 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:01.633 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:01.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:01.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:01.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 44890 44874 133.532 101.938 0.00545524 0.0074819 2024-03-21T14:05:01.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:01.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:01.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:02.005 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:01 smithi003 podman[82141]: 2024-03-21 14:05:01.754886173 +0000 UTC m=+3.351043495 container cleanup 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:02.005 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:01 smithi003 bash[82141]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T14:05:02.005 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:01 smithi003 podman[82403]: 2024-03-21 14:05:01.965981115 +0000 UTC m=+1.176287557 container remove 3f6bdcbc65d102aa39a4036aa72f6bccafd0417a1405e3d623ef4cb751f16c67 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:02.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:02.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:02.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:02.164 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:02.195 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:02.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:02.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:02.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:02.504 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:02 smithi003 podman[82403]: 2024-03-21 14:05:02.058073127 +0000 UTC m=+1.268379573 volume remove 2ccb05a7c3b0e6a5b0ca56068b13c737bcf1a638295a2d0a459ee821d770f406 2024-03-21T14:05:02.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:02.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:02.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:02 smithi118 ceph-mon[33862]: pgmap v76: 148 pgs: 148 active+clean; 2.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T14:05:02.681 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:05:02 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=infra.usagestats t=2024-03-21T14:05:02.392834361Z level=info msg="Usage stats are ready to report" 2024-03-21T14:05:02.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:02.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:02.782 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 46566 46550 132.223 104.75 0.0105254 0.00755831 2024-03-21T14:05:02.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:02 smithi003 ceph-mon[29576]: pgmap v76: 148 pgs: 148 active+clean; 2.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T14:05:02.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:02 smithi003 ceph-mon[40510]: pgmap v76: 148 pgs: 148 active+clean; 2.6 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T14:05:02.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:02.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:02.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:02.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:02.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:03.021 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:03.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:03.050 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:05:03.050 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:05:03.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:46919 7.b 7:dc733094:::benchmark_data_smithi003_80362_object46918:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:03.046855+0000", 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0026055340000000001, 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026407760000000001, 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 46919 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:03.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046855+0000", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046855+0000", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046857+0000", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.855e-06 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046887+0000", 2024-03-21T14:05:03.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9674000000000001e-05 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046903+0000", 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6538e-05 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046913+0000", 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0012e-05 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.046994+0000", 2024-03-21T14:05:03.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0742999999999996e-05 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.047111+0000", 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117421 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.047236+0000", 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012455699999999999 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:46917 7.4 e97/89 7:2f64cb53:::benchmark_data_smithi003_80362_object46916:head v 97'2984, mlcod=97'2981)", 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:03.048192+0000", 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001268744, 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013876369999999999, 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048192+0000", 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:03.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048192+0000", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048193+0000", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7400000000000002e-07 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048215+0000", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2396e-05 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048229+0000", 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3567000000000001e-05 2024-03-21T14:05:03.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048236+0000", 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0849999999999999e-06 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049078+0000", 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000841988 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049096+0000", 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8340999999999999e-05 2024-03-21T14:05:03.057 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:46926 7.b 7:dbba7757:::benchmark_data_smithi003_80362_object46925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:03.048936+0000", 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00052444000000000004, 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070907800000000003, 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:05:03.058 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 46926 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048936+0000", 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048936+0000", 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:03.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048938+0000", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4309999999999999e-06 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048979+0000", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1193999999999999e-05 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048990+0000", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.163e-05 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.048998+0000", 2024-03-21T14:05:03.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6760000000000007e-06 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049058+0000", 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9784000000000002e-05 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049183+0000", 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012558099999999999 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049324+0000", 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000140545 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.061 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:46923 7.0 e97/89 7:05d6ae52:::benchmark_data_smithi003_80362_object46922:head v 97'2981, mlcod=97'2979)", 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:03.049016+0000", 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00044405399999999999, 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070227600000000005, 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049016+0000", 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049016+0000", 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049017+0000", 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8000000000000005e-07 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049077+0000", 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0109999999999999e-05 2024-03-21T14:05:03.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049086+0000", 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.149e-06 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049093+0000", 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6329999999999999e-06 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049158+0000", 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5048999999999993e-05 2024-03-21T14:05:03.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049176+0000", 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7929999999999999e-05 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.24692.0:46919 7.b e97/89)", 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:03.049299+0000", 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00016131300000000001, 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047681999999999998, 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:05:03.065 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049299+0000", 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049299+0000", 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049300+0000", 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1399999999999997e-07 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049304+0000", 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1450000000000001e-06 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049311+0000", 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3869999999999997e-06 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049318+0000", 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.241e-06 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:03.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.24692.0:46919 7.b e97/89)", 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:03.049356+0000", 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.000104727, 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00046493400000000002, 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049356+0000", 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049356+0000", 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049357+0000", 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9100000000000003e-07 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049360+0000", 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6289999999999999e-06 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049375+0000", 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5023e-05 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:03.049381+0000", 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8490000000000002e-06 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 6 2024-03-21T14:05:03.070 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:05:03.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:03.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:03.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:03.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:03.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:03.423 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:03.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:03.549 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T14:05:03.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:03.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:03.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:03.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:03.777 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:03 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2693797236' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:05:03.777 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:03 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2693797236' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:05:03.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 15 48003 47988 130.382 89.875 0.0115114 0.0076634 2024-03-21T14:05:03.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:03.862 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:03.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:03.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:03 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2693797236' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:05:03.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:04.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:04.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:04.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:04.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:04.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:04.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:04.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:04.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:05:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:04.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:04.548 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:04.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:04.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:04.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:04.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 49528 49512 128.918 95.25 0.00957946 0.00775158 2024-03-21T14:05:04.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:04.832 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:04.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:04 smithi118 ceph-mon[33862]: pgmap v77: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-21T14:05:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2693797236' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:05:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:04 smithi118 ceph-mon[33862]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T14:05:04.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:04.963 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:04.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:04 smithi003 ceph-mon[29576]: pgmap v77: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-21T14:05:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2693797236' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:05:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:04 smithi003 ceph-mon[29576]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T14:05:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:04 smithi003 ceph-mon[40510]: pgmap v77: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-21T14:05:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2693797236' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:05:05.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:04 smithi003 ceph-mon[40510]: osdmap e98: 8 total, 8 up, 8 in 2024-03-21T14:05:05.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:05.095 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:05.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:05.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:05.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:05.340 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:05.366 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:05.468 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:05.487 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:05:05.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:05.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:05.559 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:05:05.603 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:05:05.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:05.642 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:05.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:05.662 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:05:05.662 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:05 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/90638484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:05.663 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:05 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/90638484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:05.706 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:05:05.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:05.747 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:05:05.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:05.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 50778 50762 126.886 78.125 0.0181068 0.00787476 2024-03-21T14:05:05.792 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:05:05.836 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:05:05.841 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:05:05.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:05.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:05.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:05 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/90638484' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:05.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T14:05:06.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:06.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:06.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:06.209 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:06.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:06.341 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:06.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:06.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:06.561 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:05:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:05:06.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:05:06.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:06.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:05:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:05:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:06 smithi118 ceph-mon[33862]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-21T14:05:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:06 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2153970800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:05:06.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:06.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:06.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 52024 52008 125.001 77.875 0.0108647 0.00799498 2024-03-21T14:05:06.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:06.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:06.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:07.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:06 smithi003 ceph-mon[29576]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-21T14:05:07.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:06 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2153970800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:05:07.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:06 smithi003 ceph-mon[40510]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-21T14:05:07.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:06 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2153970800' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:05:07.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:07.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:07.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:07.186 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:07.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:07.311 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:07.315 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:07.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:07.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:07.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:07.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:07.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:07.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:07.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:07.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:07.781 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 53410 53394 123.579 86.625 0.00545076 0.00808653 2024-03-21T14:05:07.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:07.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:07.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:05:07.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2153970800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:05:07.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[29576]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T14:05:07.888 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:07 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T14:05:07.888 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:07 smithi003 systemd[1]: Stopped Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:05:07.888 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:07 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.183s CPU time. 2024-03-21T14:05:07.889 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:07 smithi003 systemd[1]: Starting Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:05:07.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:07.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:05:07.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2153970800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:05:07.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:07 smithi003 ceph-mon[40510]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T14:05:07.893 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:07.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:07 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:05:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2153970800' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:05:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:07 smithi118 ceph-mon[33862]: osdmap e99: 8 total, 8 up, 8 in 2024-03-21T14:05:07.995 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:08.026 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:08.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:08.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:08.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:08.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:08.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:08.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:08.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:08.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:08.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:08.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:08.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:08.782 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 54938 54922 122.573 95.5 0.00589751 0.00815325 2024-03-21T14:05:08.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:08.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:05:08.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:05:08.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[29576]: pgmap v81: 148 pgs: 148 active+clean; 3.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-03-21T14:05:08.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:05:08.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:05:08.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:05:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:05:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[40510]: pgmap v81: 148 pgs: 148 active+clean; 3.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-03-21T14:05:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:05:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:08 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:05:08.860 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:08.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:08 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:05:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:08 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:05:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:08 smithi118 ceph-mon[33862]: pgmap v81: 148 pgs: 148 active+clean; 3.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 125 MiB/s wr, 2.01k op/s 2024-03-21T14:05:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:08 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:05:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:08 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:05:08.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:08.994 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:08.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:08.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:09.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:09.128 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:09.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:09.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:09.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:09.249 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:09.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:09.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:09.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:05:09.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:09.446 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:09.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:09.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:09.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:09.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:09.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:09.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:09.729 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:09.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:09.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:05:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:05:09.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:05:09.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:05:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:05:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:05:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:05:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:09 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:05:09.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 56313 56297 121.307 85.9375 0.0274988 0.0082372 2024-03-21T14:05:09.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:09.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:09.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:09.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:09 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:05:09.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:09 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:05:09.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:09 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:05:09.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:09 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:05:09.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:10.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:10.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:10.105 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:09 smithi003 podman[83473]: 2024-03-21 14:05:09.894285555 +0000 UTC m=+0.116480537 image pull 11f11916f8cdfb17c1da74b20deb2075d778357b4d60474d82453303f8d461f5 quay.io/prometheus/alertmanager:v0.27.0 2024-03-21T14:05:10.106 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:05:10.106 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:05:10.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:10.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:10.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:10.271 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:10.303 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:10.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:05:10.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:10.420 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:10 smithi003 podman[83473]: 2024-03-21 14:05:10.124161409 +0000 UTC m=+0.346356389 volume create fa10807a53d89fae3f8570e351cf8ad500f1c4d5f1cc1de085603a0636b966c1 2024-03-21T14:05:10.433 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:10.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:10.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:10.567 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:10.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:10.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: pgmap v82: 148 pgs: 148 active+clean; 3.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:05:10.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: pgmap v82: 148 pgs: 148 active+clean; 3.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:05:10.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:10 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:05:10.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 57648 57632 120.044 83.4375 0.00651985 0.00832472 2024-03-21T14:05:10.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:10.852 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:10.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:05:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:05:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: pgmap v82: 148 pgs: 148 active+clean; 3.1 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-21T14:05:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:05:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:05:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:05:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:10 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:05:10.954 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:10.984 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:10.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:10.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:11.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:11.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:11.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:11.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:11.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:11.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:11.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:11.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:11.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:11.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:11.657 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:11.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:11.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 58763 58747 118.42 69.6875 0.00438347 0.00843884 2024-03-21T14:05:11.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:11.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:11.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:11.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:11.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:11.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:11 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:05:11.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:11 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:05:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:11 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:05:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:11 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:05:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:11 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:05:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:11 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:05:12.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:12.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:12.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:12.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:12.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:12.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:12.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:12.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:12.593 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:12.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:12.607 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:12 smithi003 podman[83473]: 2024-03-21 14:05:12.344560729 +0000 UTC m=+2.566755719 container create da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:12.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:12.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:12.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:12.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 60016 60000 117.166 78.3125 0.0172265 0.00852964 2024-03-21T14:05:12.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:12.868 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:12.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: pgmap v83: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:05:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:12 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:05:12.970 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:12.994 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:12.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:05:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: pgmap v83: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: pgmap v83: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:05:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:12 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:05:13.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:13.126 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:13.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:13.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:13.367 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:13.397 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:13.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:13.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.523 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:13.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:13.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:13.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:13.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:13.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:13.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 61046 61030 115.567 64.375 0.00853488 0.00864771 2024-03-21T14:05:13.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:13.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:13.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:13.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.4 scrub starts 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.4 scrub ok 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:05:13.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:13 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:05:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:05:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:05:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:05:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.4 scrub starts 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.4 scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.4 scrub starts 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.4 scrub ok 2024-03-21T14:05:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:05:14.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:05:14.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:05:14.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:13 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:05:14.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:14.062 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:14.164 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:14.193 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:14.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:14.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:14.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:14.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:14.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:14.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:14.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:14.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:14.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 62395 62379 114.647 84.3125 0.00913196 0.00871731 2024-03-21T14:05:14.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:14.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:14.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: pgmap v84: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:05:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:14 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: pgmap v84: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:05:14.992 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: pgmap v84: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:05:14.993 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:14 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:05:14.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:15.021 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:15.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:15.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:15.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:15.254 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 podman[83473]: 2024-03-21 14:05:15.101628052 +0000 UTC m=+5.323823042 container init da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:15.254 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 podman[83473]: 2024-03-21 14:05:15.109478599 +0000 UTC m=+5.331673579 container start da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.131Z caller=main.go:181 level=info msg="Starting Alertmanager" version="(version=0.27.0, branch=HEAD, revision=0aa3c2aad14cff039931923ab16b26b7481783b5)" 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.132Z caller=main.go:182 level=info build_context="(go=go1.21.7, platform=linux/amd64, user=root@22cd11f671e9, date=20240228-11:51:20, tags=netgo)" 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.133Z caller=cluster.go:186 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.3 port=9094 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.135Z caller=cluster.go:683 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.186Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.191Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.194Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093 2024-03-21T14:05:15.255 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:15.194Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-21T14:05:15.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:15.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:15.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:15.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:15.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:15.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:15.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:15.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:15.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:15.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:15.783 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 63738 63722 113.77 83.9375 0.00543999 0.00878406 2024-03-21T14:05:15.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:15.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:15.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:15.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:16.004 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 bash[83473]: da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 2024-03-21T14:05:16.004 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:15 smithi003 systemd[1]: Started Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:05:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:16.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:16.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:16.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:16.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:16.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:16.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:16.311 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:16.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:16.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:05:16.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:16.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:16.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:16.598 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:16.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:16.637 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: pgmap v85: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:05:16.640 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:05:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:05:16.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:16.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:16.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:16.786 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 64963 64947 112.735 76.5625 0.00837696 0.00886533 2024-03-21T14:05:16.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:16.886 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:16.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:16.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:05:16.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:05:16.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: pgmap v85: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:05:16.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:16 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:05:16.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: pgmap v85: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:05:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:16 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:05:16.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:17.016 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:17.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:17.087 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:17.118 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:17.151 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:17.173 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:17.135Z caller=cluster.go:708 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000050492s 2024-03-21T14:05:17.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:17.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:17.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:17.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:17.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:17.574 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:17.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:17.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:17.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:17.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 66219 66203 111.81 78.5 0.0167608 0.0089379 2024-03-21T14:05:17.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:17.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:17.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: Reconfiguring daemon prometheus.a on smithi118 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:05:17.919 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:17 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:05:17.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:17.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:17.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:17.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: Reconfiguring daemon prometheus.a on smithi118 2024-03-21T14:05:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:05:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:05:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: Reconfiguring daemon prometheus.a on smithi118 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: osdmap e100: 8 total, 8 up, 8 in 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:05:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:05:18.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:17 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:05:18.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:18.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:18.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:18.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:18.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.408 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:18.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:18.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:18.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:18.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:18.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:18 smithi118 ceph-mon[33862]: pgmap v87: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:05:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:18 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:05:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:18 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:05:18.692 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:18.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 67376 67360 110.771 72.3125 0.0448887 0.00902135 2024-03-21T14:05:18.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:18.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:18.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:18 smithi003 ceph-mon[29576]: pgmap v87: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:05:18.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:18 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:05:18.849 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:18 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:05:18.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:18 smithi003 ceph-mon[40510]: pgmap v87: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:05:18.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:18 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:05:18.849 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:18 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:05:18.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:18.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:18.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:19.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:19.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:19.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:19.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:19.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:19.277 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:19.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:19.373 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:19.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:19.501 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:19.603 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:19.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T14:05:19.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:19.763 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:19.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:19.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 68593 68577 109.881 76.0625 0.0108264 0.00909599 2024-03-21T14:05:19.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:19.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:19.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:19.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:19.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:05:19.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/90254252' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/90254252' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:05:19.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:19 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:05:19.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:19 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:05:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:19 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:05:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:19 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:05:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:19 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:05:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:19 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/90254252' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:05:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:19 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:05:20.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:20.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:20.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:20.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:20.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:20.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:20.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:20.431 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 systemd[1]: Stopping Ceph prometheus.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:05:20.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:20.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:20.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:20.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:20.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:20.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:05:20.785110+0000 min lat: 0.00127999 max lat: 0.200511 avg lat: 0.00915372 2024-03-21T14:05:20.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:05:20.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 69905 69889 109.183 82 0.00668681 0.00915372 2024-03-21T14:05:20.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:20.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:20.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: pgmap v88: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:05:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:05:20.931 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.522Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.523Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.524Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.524Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[62864]: ts=2024-03-21T14:05:20.524Z caller=main.go:1273 level=info msg="See you next time!" 2024-03-21T14:05:20.932 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:20 smithi118 podman[69865]: 2024-03-21 14:05:20.533156531 +0000 UTC m=+0.136426791 container died e8a4069f98a3775b5621197d342b9c8059baf26684b9d7442a6ac35949c0a839 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:20.980 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: pgmap v88: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:05:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: pgmap v88: 148 pgs: 148 active+clean; 3.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: osdmap e101: 8 total, 8 up, 8 in 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:05:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:20 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:05:21.010 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:21.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:21.112 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:21.142 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:21.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:21.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:21.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:21.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:21.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:21.430 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:21 smithi118 podman[69865]: 2024-03-21 14:05:21.065050049 +0000 UTC m=+0.668320305 container cleanup e8a4069f98a3775b5621197d342b9c8059baf26684b9d7442a6ac35949c0a839 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:21.431 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:21 smithi118 bash[69865]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T14:05:21.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:21.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.554 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:21.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:21.655 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:21.684 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:21.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 71174 71158 108.455 79.3125 0.0101728 0.00921433 2024-03-21T14:05:21.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:21.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:21.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:21.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:21.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:22.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:05:22.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:05:22.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:05:22.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:05:22.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:05:22.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:05:22.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:05:22.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:21 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:05:22.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:21 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:05:22.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:21 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:05:22.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:21 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:05:22.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:21 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:05:22.067 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:22.101 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:22.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:22.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:22.308 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 podman[69893]: 2024-03-21 14:05:22.025755292 +0000 UTC m=+1.491653181 container remove e8a4069f98a3775b5621197d342b9c8059baf26684b9d7442a6ac35949c0a839 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:22.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:22.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:22.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:22.578 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-21T14:05:22.579 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 systemd[1]: Stopped Ceph prometheus.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:05:22.579 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.224s CPU time. 2024-03-21T14:05:22.579 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 systemd[1]: Starting Ceph prometheus.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:05:22.638 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.672 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:22.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:22.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:22.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 72403 72387 107.701 76.8125 0.0128354 0.00927974 2024-03-21T14:05:22.805 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:22.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:22.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:22.921 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:05:22.921 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:05:22.921 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:05:22.921 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:05:22.921 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-mon[33862]: pgmap v90: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T14:05:22.922 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 podman[70080]: 2024-03-21 14:05:22.483611042 +0000 UTC m=+0.028953778 image pull 1d3b7f56885b6dd623f1785be963aa9c195f86bc256ea454e8d02a7980b79c53 quay.io/prometheus/prometheus:v2.51.0 2024-03-21T14:05:22.922 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 podman[70080]: 2024-03-21 14:05:22.643676813 +0000 UTC m=+0.189019547 container create e24138720969ffa80f9d4030b8922678b7a904857108845f0b9dacd0b05f7d53 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:22.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:05:22.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:05:22.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:05:22.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:05:22.925 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[29576]: pgmap v90: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T14:05:22.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:05:22.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:05:22.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:05:22.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:05:22.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:22 smithi003 ceph-mon[40510]: pgmap v90: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-21T14:05:22.926 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:05:22.926 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:05:22.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:22.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:23.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:23.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 podman[70080]: 2024-03-21 14:05:22.922114899 +0000 UTC m=+0.467457654 container init e24138720969ffa80f9d4030b8922678b7a904857108845f0b9dacd0b05f7d53 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 podman[70080]: 2024-03-21 14:05:22.92614266 +0000 UTC m=+0.471485405 container start e24138720969ffa80f9d4030b8922678b7a904857108845f0b9dacd0b05f7d53 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.959Z caller=main.go:617 level=info msg="Starting Prometheus Server" mode=server version="(version=2.51.0, branch=HEAD, revision=c05c15512acb675e3f6cd662a6727854e93fc024)" 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.959Z caller=main.go:622 level=info build_context="(go=go1.22.1, platform=linux/amd64, user=root@b5723e458358, date=20240319-10:54:45, tags=netgo,builtinassets,stringlabels)" 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.959Z caller=main.go:623 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi118 (none))" 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.959Z caller=main.go:624 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.959Z caller=main.go:625 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.962Z caller=web.go:568 level=info component=web msg="Start listening for connections" address=:9095 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.962Z caller=main.go:1129 level=info msg="Starting TSDB ..." 2024-03-21T14:05:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.964Z caller=tls_config.go:313 level=info component=web msg="Listening on" address=[::]:9095 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.964Z caller=tls_config.go:316 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.970Z caller=head.go:616 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.970Z caller=head.go:698 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.53µs 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.970Z caller=head.go:706 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.971Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.971Z caller=head.go:778 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.971Z caller=head.go:815 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=38.308µs wal_replay_duration=639.586µs wbl_replay_duration=116ns total_replay_duration=892.547µs 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.973Z caller=main.go:1150 level=info fs_type=XFS_SUPER_MAGIC 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.973Z caller=main.go:1153 level=info msg="TSDB started" 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:22.973Z caller=main.go:1335 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:23.005Z caller=main.go:1372 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=32.254799ms db_storage=1.161µs remote_storage=1.162µs web_handler=466ns query_engine=718ns scrape=789.104µs scrape_sd=151.904µs notify=15.432µs notify_sd=11.789µs rules=30.774742ms tracing=8.806µs 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:23.005Z caller=main.go:1114 level=info msg="Server is ready to receive web requests." 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:05:23.005Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..." 2024-03-21T14:05:23.183 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:23 smithi118 bash[70080]: e24138720969ffa80f9d4030b8922678b7a904857108845f0b9dacd0b05f7d53 2024-03-21T14:05:23.184 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:05:23 smithi118 systemd[1]: Started Ceph prometheus.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:05:23.184 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:23.188 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STOPPING 2024-03-21T14:05:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.217 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:23.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:23.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:23.356 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:23.458 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:23.491 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:23.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:05:23.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STOPPED 2024-03-21T14:05:23.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STARTING 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Serving on http://:::9283 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STARTED 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STOPPING 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STOPPED 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STARTING 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Serving on http://:::9283 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STARTED 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STOPPING 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STOPPED 2024-03-21T14:05:23.505 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STARTING 2024-03-21T14:05:23.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:23.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:23.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:23.777 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:23.784 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 73555 73539 106.871 72 0.0189281 0.0093521 2024-03-21T14:05:23.880 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:23.881 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Serving on http://:::9283 2024-03-21T14:05:23.881 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:05:23] ENGINE Bus STARTED 2024-03-21T14:05:23.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: 3.0 scrub starts 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: 3.0 scrub ok 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: 6.1c scrub starts 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: 6.1c scrub ok 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: 3.0 scrub starts 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: 3.0 scrub ok 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: 6.1c scrub starts 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: 6.1c scrub ok 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:23.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:05:23.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:23.915 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:23.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:23.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T14:05:24.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:24.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.051 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:24.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:24.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: 3.0 scrub starts 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: 3.0 scrub ok 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: 6.1c scrub starts 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: 6.1c scrub ok 2024-03-21T14:05:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:05:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:24.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:24.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:24.332 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:24.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:24.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:24.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:24.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:24.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:24.786 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 74691 74675 106.05 71 0.0139817 0.009424 2024-03-21T14:05:24.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:24.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:05:24.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: pgmap v91: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:05:24.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: pgmap v91: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:05:24.857 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:24 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:05:24.890 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:24.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:05:24.923 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:24.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:25.025 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:25.055 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:25.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:25.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi003.front.sepia.ceph.com:9093"}]: dispatch 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi118.front.sepia.ceph.com:9095"}]: dispatch 2024-03-21T14:05:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi118.front.sepia.ceph.com:3000"}]: dispatch 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: pgmap v91: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:05:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:24 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:05:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:25.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:25.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:25.436 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:05:25.138Z caller=cluster.go:700 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002656358s 2024-03-21T14:05:25.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:25.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:25.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:25.642 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:25.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:05:25.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:05:25.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:05:25.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:05:25.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:05:25.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:05:25.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:05:25.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:25.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.775 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:25.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:25.787 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 75769 75753 105.189 67.375 0.00836357 0.00950115 2024-03-21T14:05:25.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:05:25.843 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:05:25.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:25.877 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:05:25.897 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:05:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:25.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:25.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:25 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:25 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:25 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:25 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:25 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:05:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:25 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:05:25.933 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:05:25.972 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:25.982 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:05:25.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:05:25.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:05:25.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:05:25.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:05:25.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:25 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:05:26.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:26.028 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:26.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:26.030 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:05:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:26.097 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:05:26.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:05:26.107 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:05:26.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:26.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:26.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:26.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:26.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:26.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:26.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:26.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:26.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:26.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:05:26.729 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:26.740 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:05:26.755 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:26.788 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 15 76724 76709 104.201 59.75 0.0158823 0.00959166 2024-03-21T14:05:26.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:26.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:26.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:26.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:26.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[29576]: pgmap v92: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 592 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-21T14:05:26.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2961601613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:26.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:05:26.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:05:26.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4139445427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:05:26.935 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:05:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[40510]: pgmap v92: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 592 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-21T14:05:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2961601613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:05:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:05:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4139445427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:05:26.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:26 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:05:26.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:27.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:27.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-mon[33862]: pgmap v92: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 592 B/s rd, 45 MiB/s wr, 719 op/s 2024-03-21T14:05:27.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2961601613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:27.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:05:27.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:05:27.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4139445427' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:05:27.099 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:26 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:05:27.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:27.165 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:27.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:27.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:27.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:27.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:27.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:27.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:27.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:27.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:27.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:27.790 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 77938 77922 103.597 75.8125 0.00731538 0.00964732 2024-03-21T14:05:27.840 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:27.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:27.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:27.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:05:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:05:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:05:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:05:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T14:05:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:05:28.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:27 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:05:28.022 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:28.080 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:28.080 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:05:28.080 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:05:28.080 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:05:28.080 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:05:28.080 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:05:28.081 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: osdmap e102: 8 total, 8 up, 8 in 2024-03-21T14:05:28.081 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:05:28.081 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:27 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:05:28.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:28.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:28.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:28.290 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:28.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:28.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:28.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:28.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:28.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:28.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:28.715 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:28.789 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 79128 79112 102.986 74.375 0.00760104 0.00970543 2024-03-21T14:05:28.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:28.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:28.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:28.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:28.993 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:28 smithi118 ceph-mon[33862]: pgmap v94: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-21T14:05:28.994 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:28 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:05:28.994 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:28 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:05:29.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:29.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:28 smithi003 ceph-mon[29576]: pgmap v94: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-21T14:05:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:28 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:05:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:28 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:05:29.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:28 smithi003 ceph-mon[40510]: pgmap v94: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-21T14:05:29.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:28 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:05:29.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:28 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:05:29.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:29.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:29.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.291 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:29.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:29.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:29.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:29.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:29.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:29.565 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:29.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:29.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:05:29.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.703 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:29.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:29.793 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 80231 80215 102.288 68.9375 0.013109 0.00977065 2024-03-21T14:05:29.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:29.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:29.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:29.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:30.019 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:05:30.019 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[29576]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.020 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:29 smithi003 ceph-mon[40510]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T14:05:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:30.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:30.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:30.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:05:30.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:05:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:05:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:05:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:29 smithi118 ceph-mon[33862]: osdmap e103: 8 total, 8 up, 8 in 2024-03-21T14:05:30.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:30.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:30.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:30.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:30.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:30.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:30.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:30.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:30.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:30.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:30.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:30.793 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 81208 81192 101.463 61.0625 0.00568954 0.00985079 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: pgmap v95: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: pgmap v95: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:30.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:30 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:30.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:30.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:31.049 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.078 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:31.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:31.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:05:31.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: pgmap v95: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:30 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:31.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:31.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:31.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:31.503 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:31.534 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:31.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:31.675 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:31.774 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T14:05:31.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:31.791 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 82218 82202 100.711 63.125 0.0278121 0.00992282 2024-03-21T14:05:31.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:31.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:31.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:31.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:31.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: 3.1a scrub starts 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: 3.1a scrub ok 2024-03-21T14:05:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2632735693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: 3.1a scrub starts 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: 3.1a scrub ok 2024-03-21T14:05:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:31 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2632735693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:05:32.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:32.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: 3.1a scrub starts 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: 3.1a scrub ok 2024-03-21T14:05:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:31 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2632735693' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:05:32.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:32.273 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:32.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:32.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:32.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:32.405 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:32.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:32.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:32.507 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:32.539 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:32.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:32.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:32.670 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:32.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:32.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:32.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:32.792 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 83344 83328 100.127 70.375 0.00474535 0.00998196 2024-03-21T14:05:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:32.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:32.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:32.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T14:05:32.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:33.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: pgmap v97: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 956 op/s 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2632735693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: 5.18 deep-scrub starts 2024-03-21T14:05:33.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[40510]: 5.18 deep-scrub ok 2024-03-21T14:05:33.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: pgmap v97: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 956 op/s 2024-03-21T14:05:33.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2632735693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:05:33.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T14:05:33.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:05:33.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:05:33.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: 5.18 deep-scrub starts 2024-03-21T14:05:33.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:32 smithi003 ceph-mon[29576]: 5.18 deep-scrub ok 2024-03-21T14:05:33.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:33.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: pgmap v97: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 956 op/s 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2632735693' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: osdmap e104: 8 total, 8 up, 8 in 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: 5.18 deep-scrub starts 2024-03-21T14:05:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:32 smithi118 ceph-mon[33862]: 5.18 deep-scrub ok 2024-03-21T14:05:33.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:33.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:33.409 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:33.438 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:33.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:33.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:33.626 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:05:33.626 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:05:33.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.626 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:84156 7.3 e104/89 7:cab23feb:::benchmark_data_smithi003_80362_object84155:head v 104'5278, mlcod=104'5276)", 2024-03-21T14:05:33.626 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:33.622791+0000", 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0028900559999999998, 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029237260000000002, 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.622791+0000", 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.622791+0000", 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:05:33.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.622792+0000", 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4499999999999995e-07 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.622815+0000", 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3476e-05 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:05:33.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.622841+0000", 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5395e-05 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.622849+0000", 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5390000000000007e-06 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.623115+0000", 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026547000000000002 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:05:33.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:33.623171+0000", 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6437999999999997e-05 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 1 2024-03-21T14:05:33.630 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:05:33.636 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:33.636 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:33.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:33.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:33.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:33.795 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 84314 84298 99.3771 60.625 0.0099732 0.0100579 2024-03-21T14:05:33.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:33.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:33.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:33.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[29576]: 5.d deep-scrub starts 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[29576]: 5.d deep-scrub ok 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:05:34.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[40510]: 5.d deep-scrub starts 2024-03-21T14:05:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[40510]: 5.d deep-scrub ok 2024-03-21T14:05:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:05:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:05:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:05:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:33 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:05:34.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:34.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:34.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:34.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:34.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:33 smithi118 ceph-mon[33862]: 5.d deep-scrub starts 2024-03-21T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:33 smithi118 ceph-mon[33862]: 5.d deep-scrub ok 2024-03-21T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:33 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:33 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:33 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:05:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:33 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:05:34.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:34.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:34.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:34.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:34.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:34.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:34.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:34.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:34.710 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:34.794 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 85443 85427 98.8435 70.5625 0.0104927 0.010112 2024-03-21T14:05:34.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:34.839 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:34.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:34.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:34.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:34.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:34.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[29576]: pgmap v99: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 780 B/s rd, 52 MiB/s wr, 838 op/s 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[29576]: 6.9 deep-scrub starts 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[29576]: 6.9 deep-scrub ok 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[29576]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[40510]: pgmap v99: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 780 B/s rd, 52 MiB/s wr, 838 op/s 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:05:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:05:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[40510]: 6.9 deep-scrub starts 2024-03-21T14:05:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[40510]: 6.9 deep-scrub ok 2024-03-21T14:05:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:34 smithi003 ceph-mon[40510]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T14:05:35.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:35.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:34 smithi118 ceph-mon[33862]: pgmap v99: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 780 B/s rd, 52 MiB/s wr, 838 op/s 2024-03-21T14:05:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:34 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:05:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:34 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:05:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:34 smithi118 ceph-mon[33862]: 6.9 deep-scrub starts 2024-03-21T14:05:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:34 smithi118 ceph-mon[33862]: 6.9 deep-scrub ok 2024-03-21T14:05:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:34 smithi118 ceph-mon[33862]: osdmap e105: 8 total, 8 up, 8 in 2024-03-21T14:05:35.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:35.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:35.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.386 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:35.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:35.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:35.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:35.647 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:35.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:35.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:35.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:35.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:35.796 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 86407 86391 98.1386 60.25 0.0564525 0.0101779 2024-03-21T14:05:35.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:35.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:35.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:36.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:36.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:36.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:35 smithi118 ceph-mon[33862]: 7.e scrub starts 2024-03-21T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:35 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:35 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:35 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:05:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:35 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:05:36.196 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[29576]: 7.e scrub starts 2024-03-21T14:05:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:05:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:05:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:05:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:05:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[40510]: 7.e scrub starts 2024-03-21T14:05:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:05:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:05:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:05:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:35 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:05:36.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:36.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:36.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:36.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:36.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:36.579 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:36.609 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:36.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:05:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:05:36.711 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:36.735 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:05:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:05:36.796 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 87256 87240 97.3337 53.0625 0.0147395 0.0102689 2024-03-21T14:05:36.836 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:36.870 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:36.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:36.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:36.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:37.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:37.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:37.112 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:37.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:37.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:36 smithi118 ceph-mon[33862]: pgmap v101: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 57 MiB/s wr, 916 op/s 2024-03-21T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:36 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:36 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:36 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:05:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:36 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:05:37.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[29576]: pgmap v101: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 57 MiB/s wr, 916 op/s 2024-03-21T14:05:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:05:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:05:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:05:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:05:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[40510]: pgmap v101: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 57 MiB/s wr, 916 op/s 2024-03-21T14:05:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:05:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:05:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:05:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:36 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:05:37.272 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:37.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:37.402 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:37.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:37.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:37.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:37.664 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:37.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:37.796 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 88292 88276 96.7621 64.75 0.00412432 0.0103222 2024-03-21T14:05:37.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:37.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:38.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:38.055 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:38.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:38.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 7.e scrub ok 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:05:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:05:38.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:37 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:05:38.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 7.e scrub ok 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:05:38.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 7.e scrub ok 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:05:38.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:05:38.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:05:38.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:05:38.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:05:38.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:05:38.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:05:38.222 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:37 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:05:38.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:38.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:38.494 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:38.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:38.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:38.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:38.754 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:38.796 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 89399 89383 96.2867 69.1875 0.00784624 0.0103802 2024-03-21T14:05:38.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:38.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:38.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:38.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:39.022 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:39.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:38 smithi118 ceph-mon[33862]: pgmap v102: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T14:05:39.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:38 smithi118 ceph-mon[33862]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T14:05:39.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:38 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:05:39.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:38 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:05:39.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:38 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:05:39.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:38 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:05:39.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:39.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[29576]: pgmap v102: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T14:05:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[29576]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T14:05:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:05:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:05:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:05:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:05:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[40510]: pgmap v102: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T14:05:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[40510]: osdmap e106: 8 total, 8 up, 8 in 2024-03-21T14:05:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:05:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:05:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:05:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:38 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:05:39.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:39.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:39.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:39.406 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:39.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T14:05:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:39.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:39.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:39.565 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:39.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:39.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:39.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:39.796 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 90312 90296 95.6216 57.0625 0.0128466 0.0104529 2024-03-21T14:05:39.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:39.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:39.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:39.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:05:39.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:05:39.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:05:39.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:05:39.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[29576]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:05:39.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:39 smithi003 ceph-mon[40510]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T14:05:39.966 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:39.995 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:39.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:39.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:40.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:40.126 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:40.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:05:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:39 smithi118 ceph-mon[33862]: osdmap e107: 8 total, 8 up, 8 in 2024-03-21T14:05:40.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:40.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:40.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:40.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:40.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:40.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:40.539 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:40.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:40.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:40.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:40.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:40.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:05:40.798938+0000 min lat: 0.00127999 max lat: 0.200511 avg lat: 0.0105261 2024-03-21T14:05:40.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:05:40.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 91201 91185 94.952 55.5625 0.00947157 0.0105261 2024-03-21T14:05:40.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:40.837 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:40.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:40.966 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:40.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:40 smithi003 ceph-mon[40510]: pgmap v104: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 680 B/s rd, 43 MiB/s wr, 689 op/s 2024-03-21T14:05:41.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:40 smithi003 ceph-mon[29576]: pgmap v104: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 680 B/s rd, 43 MiB/s wr, 689 op/s 2024-03-21T14:05:41.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:41.096 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:40 smithi118 ceph-mon[33862]: pgmap v104: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 680 B/s rd, 43 MiB/s wr, 689 op/s 2024-03-21T14:05:41.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:41.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:41.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:41.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:41.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:41.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:41.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:41.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:41.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:41.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:41.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:41.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 92208 92192 94.4271 62.9375 0.0107523 0.0105853 2024-03-21T14:05:41.890 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:05:41.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:41.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:41.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:41.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:41.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2480219366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2480219366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:05:41.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:41 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:05:42.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:42.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:42.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:42.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:42.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:41 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:05:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:41 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:05:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2480219366' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:05:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:41 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:05:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:42.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:42.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:42.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:42.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:42.601 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:42.628 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:42.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:42.760 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:42.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:42.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 93302 93286 94.007 68.375 0.00823062 0.0106323 2024-03-21T14:05:42.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:42.891 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:42.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:42.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:43.024 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:43.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:43.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:43.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:43.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:42 smithi118 ceph-mon[33862]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T14:05:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:42 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:05:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:05:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:42 smithi118 ceph-mon[33862]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T14:05:43.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[29576]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T14:05:43.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:05:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:05:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[29576]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T14:05:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[40510]: pgmap v106: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.4 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-21T14:05:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:05:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:05:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:42 smithi003 ceph-mon[40510]: osdmap e108: 8 total, 8 up, 8 in 2024-03-21T14:05:43.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:43.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:43.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:43.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:43.412 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:43.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:43.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:43.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:43.672 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:43.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:43.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 15 94409 94394 93.614 69.25 0.00738649 0.0106768 2024-03-21T14:05:43.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:43.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:43.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:43.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:43.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:44.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:44.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:44.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:44.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:44.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:44.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:44.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:44.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:44.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:44.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:44.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:44.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:44.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:44.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:44.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:44.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 95393 95377 93.1109 61.4375 0.0358764 0.0107348 2024-03-21T14:05:44.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:44.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:44.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:44.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:44.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:05:44.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:44 smithi003 ceph-mon[29576]: pgmap v108: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-21T14:05:44.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:44 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:44.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:44 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:44.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:44 smithi003 ceph-mon[40510]: pgmap v108: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-21T14:05:44.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:44 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:44.941 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:44 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:44.995 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:05:45.011 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:45.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:45.065 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:05:45.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:45.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:45.132 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:05:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:44 smithi118 ceph-mon[33862]: pgmap v108: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 50 MiB/s wr, 805 op/s 2024-03-21T14:05:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:44 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:44 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:45.181 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:05:45.192 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:45.218 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:45.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:45.227 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:05:45.277 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:05:45.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:45.322 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:05:45.334 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:05:45.347 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:45.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:45.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:45.476 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:45.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:45.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:45.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:45.754 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:45.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 96264 96248 92.5159 54.4375 0.0155371 0.010804 2024-03-21T14:05:45.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:45.883 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:45.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:45.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:46.012 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:46.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:46.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2832535780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:46.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:46.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:46.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/606277021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:05:46.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2832535780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:46.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:46.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:46.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/606277021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:05:46.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:46.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:46.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:46.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2832535780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:05:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/606277021' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:05:46.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:46.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:46.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:46.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:46.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:46.530 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T14:05:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.555 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:46.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:46.656 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:46.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:05:46.689 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:46.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:05:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:05:46.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:46.798 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 97305 97289 92.1 65.0625 0.0248434 0.0108522 2024-03-21T14:05:46.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:46.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:46.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:46.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:47.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:47.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:47.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-mon[33862]: pgmap v109: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 686 B/s rd, 40 MiB/s wr, 647 op/s 2024-03-21T14:05:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:05:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:05:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/606277021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:05:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:46 smithi118 ceph-mon[33862]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T14:05:47.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[29576]: pgmap v109: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 686 B/s rd, 40 MiB/s wr, 647 op/s 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/606277021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[29576]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[40510]: pgmap v109: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 686 B/s rd, 40 MiB/s wr, 647 op/s 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/606277021' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:05:47.208 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:46 smithi003 ceph-mon[40510]: osdmap e109: 8 total, 8 up, 8 in 2024-03-21T14:05:47.236 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:47.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:47.366 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:47.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:47.469 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:47.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:05:47.501 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:47.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T14:05:47.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:47.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:47.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:47.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:47.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 98360 98344 91.7095 65.9375 0.0111628 0.0108978 2024-03-21T14:05:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:47.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:47.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:47.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:47.976 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:47.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:48.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:48.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:48.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:48.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:48.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:48.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:48.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:48.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:48.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:05:48.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:48.517 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:48.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:48.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:48.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:48.744 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:48.772 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:48.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 99194 99178 91.1275 52.125 0.0129928 0.0109691 2024-03-21T14:05:48.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:48.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:48.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:48.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:48.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:49.031 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:49.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:49.155 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:48 smithi118 ceph-mon[33862]: pgmap v111: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T14:05:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:48 smithi003 ceph-mon[29576]: pgmap v111: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T14:05:49.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:48 smithi003 ceph-mon[40510]: pgmap v111: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-21T14:05:49.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:49.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:49.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:49.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:49.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:49.580 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:49.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:49.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:49.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:49.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 100097 100081 90.6247 56.4375 0.0230636 0.0110294 2024-03-21T14:05:49.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:49.846 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:49.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:49.976 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:49.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:49.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:50.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:50.103 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:50.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:50.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:50.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:49 smithi003 ceph-mon[29576]: pgmap v112: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T14:05:50.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:49 smithi003 ceph-mon[40510]: pgmap v112: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T14:05:50.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:50.282 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:49 smithi118 ceph-mon[33862]: pgmap v112: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T14:05:50.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:50.406 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:50.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:50.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:50.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:50.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:50.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:50.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:50.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 101042 101026 90.1739 59.0625 0.0197249 0.0110846 2024-03-21T14:05:50.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:50.835 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:50.920 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:05:50.920 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:05:50.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:50.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:50.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:51.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:51.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:51.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:51.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:51.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:51.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:51.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:51.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:51.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:51.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:51.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:51.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:51.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 102103 102087 89.8378 66.3125 0.0117822 0.0111266 2024-03-21T14:05:51.808 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:51.910 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:51.944 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:51.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:52.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:52.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:52.074 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:52.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:52.204 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:52.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:52.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:52.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:52.471 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:52.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:52 smithi003 ceph-mon[29576]: pgmap v113: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 734 op/s 2024-03-21T14:05:52.520 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:52 smithi003 ceph-mon[40510]: pgmap v113: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 734 op/s 2024-03-21T14:05:52.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T14:05:52.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:52.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:52.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:52.658 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:52.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:52 smithi118 ceph-mon[33862]: pgmap v113: 148 pgs: 148 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 46 MiB/s wr, 734 op/s 2024-03-21T14:05:52.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:52.799 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 103127 103111 89.4789 64 0.010216 0.0111698 2024-03-21T14:05:52.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:52.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:52.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:52.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:52.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:53.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:53.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:53.215 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:53.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:53.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:53.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:53.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:53.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:53.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:53.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:53.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 104224 104208 89.191 68.5625 0.0141038 0.0112065 2024-03-21T14:05:53.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:53.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:53.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:53.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:54.041 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:54.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:54.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:54.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:54.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:54.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:54.303 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:54.334 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:54.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:54.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:54.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:54.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:54.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:54.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:54.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:54 smithi118 ceph-mon[33862]: pgmap v114: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-21T14:05:54.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:54 smithi003 ceph-mon[29576]: pgmap v114: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-21T14:05:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:54 smithi003 ceph-mon[40510]: pgmap v114: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-21T14:05:54.757 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:05:54.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:54.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:54.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 105198 105182 88.8084 60.875 0.014019 0.011255 2024-03-21T14:05:54.807 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:05:54.859 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:05:54.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:54.901 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:05:54.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:54.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:54.943 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:05:54.984 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:05:55.024 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:05:55.031 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:55.048 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:05:55.049 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:05:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:05:55.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:55.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:55.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:55.186 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:55.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:55.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:55.316 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:55.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:55.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:55.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:55.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:55.567 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:55.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:55.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:55 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/206875646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:55.699 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:55.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:55 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/206875646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:55 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/206875646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:05:55.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 106142 106126 88.4109 59 0.0142599 0.0113056 2024-03-21T14:05:55.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:55.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:55.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:55.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:55.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:56.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:56.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:56.248 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:56.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:56.380 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:56.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:56.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:56 smithi003 ceph-mon[29576]: pgmap v115: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-21T14:05:56.492 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:56 smithi003 ceph-mon[40510]: pgmap v115: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-21T14:05:56.512 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:56.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:56.645 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:56.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:56 smithi118 ceph-mon[33862]: pgmap v115: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-21T14:05:56.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:05:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:05:56.747 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:05:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:05:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:05:56.776 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:56.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:56.800 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 107224 107208 88.1374 67.625 0.00564705 0.0113405 2024-03-21T14:05:56.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:56.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:56.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:57.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:57.045 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:57.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:57.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:57.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:57.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:57.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:57.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:57.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:57.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:57.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:57.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:57.713 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:57.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:57.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:57.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:57.745 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:05:57.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 108273 108257 87.8435 65.5625 0.0234614 0.0113785 2024-03-21T14:05:57.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:57.874 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:57.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:57.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:57.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:05:58.005 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:58.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:05:58.039 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:05:58.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:58.135 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:58.237 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:05:58.266 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:05:58.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:58.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:58.497 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:58.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:58.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:58.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:05:58.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:58 smithi003 ceph-mon[29576]: pgmap v116: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 950 B/s rd, 57 MiB/s wr, 920 op/s 2024-03-21T14:05:58.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:58 smithi003 ceph-mon[40510]: pgmap v116: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 950 B/s rd, 57 MiB/s wr, 920 op/s 2024-03-21T14:05:58.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:05:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:58 smithi118 ceph-mon[33862]: pgmap v116: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 950 B/s rd, 57 MiB/s wr, 920 op/s 2024-03-21T14:05:58.778 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:58.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 109247 109231 87.4978 60.875 0.0118517 0.0114241 2024-03-21T14:05:58.807 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:05:58.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:58.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:58.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:59.049 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:59.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:59.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:05:59.220 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:05:59.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:05:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:59.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:05:59.414 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T14:05:59.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:59.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:59.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:05:59.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:05:59.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/607112632' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:05:59.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:59 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:05:59.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:05:59 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:05:59.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:05:59.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/607112632' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:05:59.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:59 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:05:59.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:05:59 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:05:59.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/607112632' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:05:59.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:59 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:05:59.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:05:59 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:05:59.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:05:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:05:59.801 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 110230 110214 87.1679 61.4375 0.0111814 0.0114671 2024-03-21T14:05:59.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:05:59.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:05:59.928 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:05:59.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:05:59.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:00.029 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:00.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:00.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:00.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:00.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:00.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T14:06:00.341 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:00.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:00.442 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:06:00.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:00.467 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:00.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:00.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:00.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:00.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:00 smithi118 ceph-mon[33862]: pgmap v117: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-21T14:06:00.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/607112632' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:06:00.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:00 smithi118 ceph-mon[33862]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T14:06:00.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/141485433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:00.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:00 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:00.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[29576]: pgmap v117: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-21T14:06:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/607112632' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:06:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[29576]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T14:06:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/141485433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:00.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:00.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[40510]: pgmap v117: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 825 op/s 2024-03-21T14:06:00.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/607112632' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:06:00.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[40510]: osdmap e110: 8 total, 8 up, 8 in 2024-03-21T14:06:00.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/141485433' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:00.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:00 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:00.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:00.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:06:00.804135+0000 min lat: 0.00127999 max lat: 0.200511 avg lat: 0.0115125 2024-03-21T14:06:00.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:06:00.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 111185 111169 86.8229 59.6875 0.0333395 0.0115125 2024-03-21T14:06:00.868 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:00.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:00.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:00.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:00.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:01.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:01.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:01.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:01.167 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:01.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:01.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:01.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:01.450 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:01.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:01.574 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:01.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:01.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:01 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:01.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:01 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:01.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:01 smithi118 ceph-mon[33862]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T14:06:01.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:01.707 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[29576]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T14:06:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[40510]: osdmap e111: 8 total, 8 up, 8 in 2024-03-21T14:06:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:01.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 112196 112180 86.5311 63.1875 0.016671 0.0115506 2024-03-21T14:06:01.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:01.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:01.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:02.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:02.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:02.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:02.261 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:02.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:02.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:02.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:21333 7.e 7:70a08e40:::benchmark_data_smithi003_80362_object21332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-03-21T14:06:02.374 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:04:48.834193+0000", 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 73.533358770000007, 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099850314999999995, 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 21333 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834193+0000", 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834193+0000", 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834195+0000", 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.387e-06 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834218+0000", 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3037e-05 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834230+0000", 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1819e-05 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834237+0000", 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5750000000000003e-06 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834288+0000", 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0476999999999998e-05 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834425+0000", 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013718899999999999 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.834566+0000", 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000141629 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.835602+0000", 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010354489999999999 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.835780+0000", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000177985 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.933879+0000", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.098099454000000003 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.933992+0000", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112862 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:48.934044+0000", 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1452000000000001e-05 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:21610 7.e 7:7c0eb773:::benchmark_data_smithi003_80362_object21609:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:04:49.045600+0000", 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 73.321951949999999, 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099735013999999997, 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.380 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 21610 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045600+0000", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045600+0000", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045601+0000", 2024-03-21T14:06:02.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1200000000000001e-06 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045622+0000", 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0458999999999998e-05 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045633+0000", 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1156e-05 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045640+0000", 2024-03-21T14:06:02.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3529999999999997e-06 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045810+0000", 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169419 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.045941+0000", 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131364 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.046076+0000", 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013495400000000001 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.047717+0000", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00164145 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.047800+0000", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2893999999999996e-05 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.144988+0000", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.09718794 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.145283+0000", 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000294245 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:49.145335+0000", 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2660000000000001e-05 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:24437 7.e 7:77ac0c79:::benchmark_data_smithi003_80362_object24436:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:04:50.583324+0000", 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 71.784228573999997, 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070672100000000002, 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.385 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 24437 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583324+0000", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583324+0000", 2024-03-21T14:06:02.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583325+0000", 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.302e-06 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583344+0000", 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9559e-05 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583354+0000", 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9410000000000002e-06 2024-03-21T14:06:02.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583361+0000", 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.72e-06 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583423+0000", 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2266000000000003e-05 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583535+0000", 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111747 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.583621+0000", 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6289e-05 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.584405+0000", 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00078401700000000003 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.584702+0000", 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029634300000000002 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.653886+0000", 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.069183837999999998 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.653951+0000", 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5303999999999997e-05 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:04:50.653996+0000", 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4774000000000002e-05 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:85522 7.e 7:7321d840:::benchmark_data_smithi003_80362_object85521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:34.867389+0000", 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.500163427, 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070123995999999994, 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85522 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.867389+0000", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.867389+0000", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.867391+0000", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3889999999999999e-06 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.867417+0000", 2024-03-21T14:06:02.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.582e-05 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.867428+0000", 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1255e-05 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.867437+0000", 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7630000000000006e-06 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.879330+0000", 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011893317 2024-03-21T14:06:02.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.879459+0000", 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012857100000000001 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.879612+0000", 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015305000000000001 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.883280+0000", 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036681050000000001 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.934029+0000", 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050748957999999997 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.934069+0000", 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0370999999999997e-05 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.937451+0000", 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033818870000000001 2024-03-21T14:06:02.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:34.937513+0000", 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1509999999999999e-05 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:85740 7.e 7:7dec6feb:::benchmark_data_smithi003_80362_object85739:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.100987+0000", 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 27.266564844000001, 2024-03-21T14:06:02.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083141825000000003, 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 85740 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.100987+0000", 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.100987+0000", 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.100988+0000", 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4200000000000005e-07 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.101009+0000", 2024-03-21T14:06:02.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0953000000000001e-05 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.101025+0000", 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5733999999999999e-05 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.101033+0000", 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7449999999999995e-06 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.112721+0000", 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011688726 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.112856+0000", 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013517299999999999 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.113010+0000", 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000153284 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.116399+0000", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033888820000000002 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.116435+0000", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6189999999999997e-05 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.184010+0000", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.067574706999999998 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.184069+0000", 2024-03-21T14:06:02.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9698999999999999e-05 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.184129+0000", 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9889999999999997e-05 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86362 7.e 7:776382ae:::benchmark_data_smithi003_80362_object86361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.723042+0000", 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.644510407999999, 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.075603154000000006, 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86362 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.723042+0000", 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.723042+0000", 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.723044+0000", 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.03e-06 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.723065+0000", 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1645999999999999e-05 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.723083+0000", 2024-03-21T14:06:02.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7207999999999999e-05 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.723091+0000", 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8519999999999993e-06 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.724016+0000", 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092485600000000005 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.724165+0000", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000148877 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.724329+0000", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163537 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.728819+0000", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044907269999999999 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.733241+0000", 2024-03-21T14:06:02.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044217680000000004 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.798501+0000", 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.065259836000000002 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.798562+0000", 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0760000000000001e-05 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.798645+0000", 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3057000000000001e-05 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86374 7.e 7:70863d7b:::benchmark_data_smithi003_80362_object86373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.737224+0000", 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.630328036999998, 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.081324901000000005, 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86374 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.737224+0000", 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.737224+0000", 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.737226+0000", 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.42e-06 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.737249+0000", 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3822999999999999e-05 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.737261+0000", 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1970999999999999e-05 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.737270+0000", 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7220000000000005e-06 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.798687+0000", 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.061417126000000002 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.798840+0000", 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015266800000000001 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.799010+0000", 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169727 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.808638+0000", 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096285350000000006 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.808710+0000", 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2362999999999998e-05 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818170+0000", 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094592259999999994 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818374+0000", 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020404 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818549+0000", 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017527999999999999 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86377 7.e 7:7cbced0e:::benchmark_data_smithi003_80362_object86376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.740601+0000", 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.626950790999999, 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.078237709000000003, 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.416 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86377 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740601+0000", 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740601+0000", 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740603+0000", 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5370000000000001e-06 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740628+0000", 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4896999999999999e-05 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740639+0000", 2024-03-21T14:06:02.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1531e-05 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740648+0000", 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9639999999999992e-06 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.801746+0000", 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.061097624000000003 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.801920+0000", 2024-03-21T14:06:02.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000173917 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.802131+0000", 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000210824 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.808677+0000", 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065466150000000004 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.808767+0000", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9580999999999995e-05 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818567+0000", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098002590000000004 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818780+0000", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021323100000000001 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818839+0000", 2024-03-21T14:06:02.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8729e-05 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86382 7.e 7:76aa0f2e:::benchmark_data_smithi003_80362_object86381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.740662+0000", 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.626890425999999, 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.078767004000000002, 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.422 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86382 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740662+0000", 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740662+0000", 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740662+0000", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6199999999999998e-07 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740685+0000", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2830999999999999e-05 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740693+0000", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5769999999999996e-06 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.740699+0000", 2024-03-21T14:06:02.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7329999999999997e-06 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.804855+0000", 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064155718 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.804989+0000", 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133635 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.805129+0000", 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013979199999999999 2024-03-21T14:06:02.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.426 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:02.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.812882+0000", 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077535089999999996 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.816453+0000", 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035709439999999999 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.818953+0000", 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002499608 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.819372+0000", 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00041960499999999998 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.819429+0000", 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6490000000000003e-05 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.428 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86390 7.e 7:7b0e6a2b:::benchmark_data_smithi003_80362_object86389:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.747483+0000", 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.620068882999998, 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.074209068000000003, 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86390 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.429 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.747483+0000", 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.747483+0000", 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.747485+0000", 2024-03-21T14:06:02.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.658e-06 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.747508+0000", 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3549000000000001e-05 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.793770+0000", 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046261548 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.793791+0000", 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1362999999999999e-05 2024-03-21T14:06:02.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.808828+0000", 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015036493 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.809015+0000", 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000187426 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.809163+0000", 2024-03-21T14:06:02.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014812899999999999 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.815857+0000", 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066939419999999996 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.816492+0000", 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063442900000000005 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.819437+0000", 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029455359999999999 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.819703+0000", 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026591499999999998 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.821692+0000", 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0019890799999999998 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:86534 7.a e105/89 7:5f0fa08f:::benchmark_data_smithi003_80362_object86533:head v 105'5396, mlcod=105'5394)", 2024-03-21T14:06:02.434 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:35.921159+0000", 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.446392646, 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070401200999999997, 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.921159+0000", 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.921159+0000", 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.921161+0000", 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.161e-06 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.921192+0000", 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1266999999999997e-05 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.921201+0000", 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8880000000000003e-06 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.921208+0000", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0949999999999998e-06 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.983237+0000", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.062029075000000003 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.983259+0000", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2515e-05 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.991472+0000", 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082123249999999995 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:35.991561+0000", 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8875000000000005e-05 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86672 7.e 7:7b8698f6:::benchmark_data_smithi003_80362_object86671:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.095006+0000", 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.272545648000001, 2024-03-21T14:06:02.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083385015000000007, 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86672 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.095006+0000", 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.095006+0000", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.095009+0000", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0200000000000001e-06 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.095034+0000", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5258000000000001e-05 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.095046+0000", 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2116999999999999e-05 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.095054+0000", 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1929999999999993e-06 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.104279+0000", 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092248009999999995 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.104402+0000", 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123044 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.104546+0000", 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000144382 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.108883+0000", 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004336225 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.112645+0000", 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037627509999999999 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.178277+0000", 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.065631361999999999 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.178335+0000", 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8479e-05 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.178392+0000", 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6382999999999999e-05 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:86696 7.e 7:7c8b1194:::benchmark_data_smithi003_80362_object86695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.116785+0000", 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 26.250767118999999, 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.072297553000000001, 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86696 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.116785+0000", 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.116785+0000", 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.116787+0000", 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7430000000000001e-06 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.116810+0000", 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3317999999999999e-05 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.116822+0000", 2024-03-21T14:06:02.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1999e-05 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.116831+0000", 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.037e-06 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.178568+0000", 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.061737357999999999 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.178686+0000", 2024-03-21T14:06:02.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117433 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.178826+0000", 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139655 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.188825+0000", 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0099994839999999995 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.188838+0000", 2024-03-21T14:06:02.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2666999999999999e-05 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.188947+0000", 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109083 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.189000+0000", 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2849999999999997e-05 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.189083+0000", 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2925999999999999e-05 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.449 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:87190 7.a e105/89 7:5ac35abf:::benchmark_data_smithi003_80362_object87189:head v 105'5447, mlcod=105'5444)", 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.655493+0000", 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.712058751000001, 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099646245999999994, 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.655493+0000", 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.655493+0000", 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.655494+0000", 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7199999999999999e-07 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.655514+0000", 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0098999999999998e-05 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.655522+0000", 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7989999999999994e-06 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.655527+0000", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5979999999999996e-06 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657281+0000", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017535879999999999 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657300+0000", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9259999999999999e-05 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.754338+0000", 2024-03-21T14:06:02.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.097037290999999998 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.755140+0000", 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00080203900000000005 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:87195 7.a e105/89 7:59999b48:::benchmark_data_smithi003_80362_object87194:head v 105'5448, mlcod=105'5444)", 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.658489+0000", 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.709062875000001, 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10853417899999999, 2024-03-21T14:06:02.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658489+0000", 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658489+0000", 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658490+0000", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.161e-06 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658516+0000", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5854e-05 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658527+0000", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0628000000000001e-05 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658535+0000", 2024-03-21T14:06:02.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7759999999999997e-06 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.659870+0000", 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013349729999999999 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.659888+0000", 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8291000000000002e-05 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.766983+0000", 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.107094734 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.767023+0000", 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0762e-05 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:87198 7.2 e105/89 7:4ff725ac:::benchmark_data_smithi003_80362_object87197:head v 105'5495, mlcod=105'5492)", 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.662640+0000", 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.704912290999999, 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.107421207, 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.458 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.662640+0000", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.662640+0000", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.662641+0000", 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0640000000000001e-06 2024-03-21T14:06:02.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.662660+0000", 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9375999999999999e-05 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.753351+0000", 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.090690755999999997 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.753367+0000", 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5574e-05 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.755169+0000", 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001802784 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.755189+0000", 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9448000000000001e-05 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.769839+0000", 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014650471 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.770061+0000", 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00022173399999999999 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:87203 7.8 7:1c4e752b:::benchmark_data_smithi003_80362_object87202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.663644+0000", 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 25.703907866000002, 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.102370321, 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.462 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 87203 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.663644+0000", 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.663644+0000", 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.663647+0000", 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2790000000000001e-06 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.663669+0000", 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2129000000000001e-05 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.756058+0000", 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.09238884 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.756070+0000", 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2713e-05 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.759386+0000", 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003316032 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.759561+0000", 2024-03-21T14:06:02.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017468900000000001 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.759706+0000", 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014530399999999999 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.765418+0000", 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057119609999999998 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.765471+0000", 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2871999999999998e-05 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.765913+0000", 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00044201800000000002 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.765975+0000", 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2267000000000005e-05 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.766015+0000", 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9217000000000001e-05 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:88274 7.e 7:796b03ba:::benchmark_data_smithi003_80362_object88273:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:37.743716+0000", 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.623836504, 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073961879999999994, 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.468 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 88274 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743716+0000", 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743716+0000", 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743718+0000", 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.097e-06 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743739+0000", 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1489e-05 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743781+0000", 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2147999999999997e-05 2024-03-21T14:06:02.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743799+0000", 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8003999999999999e-05 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.743884+0000", 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4134000000000005e-05 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.744020+0000", 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013643100000000001 2024-03-21T14:06:02.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.746051+0000", 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020313900000000001 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.749313+0000", 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003261711 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.749332+0000", 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8533999999999999e-05 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.817545+0000", 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.068213663999999993 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.817622+0000", 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7236000000000004e-05 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:37.817678+0000", 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5041999999999999e-05 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.473 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:90081 7.e 7:7f9b09da:::benchmark_data_smithi003_80362_object90080:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:39.446419+0000", 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 22.921133537999999, 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.077344072, 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.474 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 90081 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446419+0000", 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446419+0000", 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446421+0000", 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0329999999999998e-06 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446445+0000", 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4561000000000001e-05 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446460+0000", 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5180999999999999e-05 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446470+0000", 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.679e-06 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446536+0000", 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5998000000000001e-05 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446658+0000", 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012231899999999999 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.446845+0000", 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018642499999999999 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.458910+0000", 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012064956999999999 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.459509+0000", 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00059875599999999998 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.523684+0000", 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064175385000000001 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.523729+0000", 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.477e-05 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.523763+0000", 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4007999999999997e-05 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.479 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:90082 7.e 7:7d828caf:::benchmark_data_smithi003_80362_object90081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:39.447568+0000", 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 22.919983958, 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.085887521999999994, 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 90082 2024-03-21T14:06:02.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.447568+0000", 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.447568+0000", 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:06:02.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.447569+0000", 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5300000000000002e-07 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.447588+0000", 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9249000000000002e-05 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.447599+0000", 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0447e-05 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.447607+0000", 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9070000000000007e-06 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.455880+0000", 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082729959999999995 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.456037+0000", 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015731999999999999 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.456173+0000", 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013602300000000001 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.459514+0000", 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033410509999999998 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.462777+0000", 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032626399999999998 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.533128+0000", 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070351669000000006 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.533322+0000", 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019384 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:39.533456+0000", 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013342700000000001 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.485 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:06:02.486 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:06:02.486 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:06:02.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:02.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:02.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:02 smithi003 ceph-mon[29576]: pgmap v120: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 872 op/s 2024-03-21T14:06:02.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:02 smithi003 ceph-mon[40510]: pgmap v120: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 872 op/s 2024-03-21T14:06:02.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:02.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:02 smithi118 ceph-mon[33862]: pgmap v120: 148 pgs: 148 active+clean; 6.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 872 op/s 2024-03-21T14:06:02.686 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:02.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:02.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 113172 113156 86.2197 61 0.0124267 0.011593 2024-03-21T14:06:02.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:02.930 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:02.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:02.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:02.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:02.983 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:03.062 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:03.095 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:03.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:06:03.196 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:03.222 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:03.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:03.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:03.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:03.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:03.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:03.672 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:03.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:03.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:03.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 114195 114179 85.9513 63.9375 0.0200039 0.0116291 2024-03-21T14:06:03.805 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:03.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:03.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:03.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:03.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:04.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:04.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:04.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:04.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:04.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:04.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:04.465 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:04.494 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:04.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:06:04.541 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:06:04.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[29576]: pgmap v121: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-21T14:06:04.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[29576]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T14:06:04.542 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:06:04.542 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:06:04.542 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[40510]: pgmap v121: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-21T14:06:04.542 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:04 smithi003 ceph-mon[40510]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T14:06:04.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:04.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:04.729 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:04.758 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:04.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 15 115224 115209 85.6944 64.375 0.0117997 0.0116648 2024-03-21T14:06:04.860 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:04.891 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:04.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:04 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:06:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:04 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:06:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:04 smithi118 ceph-mon[33862]: pgmap v121: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-21T14:06:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:04 smithi118 ceph-mon[33862]: osdmap e112: 8 total, 8 up, 8 in 2024-03-21T14:06:04.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:05.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:05.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:05.175 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:05.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:05.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:05.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:05.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:05.444 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:05.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:05.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:05.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:05.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:06:05.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:05.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:05 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:06:05.680 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:05 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:06:05.681 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:05 smithi003 ceph-mon[29576]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T14:06:05.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:05 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:06:05.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:05 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:06:05.681 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:05 smithi003 ceph-mon[40510]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T14:06:05.693 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:06:05.705 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:05.735 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:06:05.778 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:06:05.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 116246 116230 85.437 63.8125 0.0169018 0.0116996 2024-03-21T14:06:05.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:05.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:06:05.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:05.864 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:06:05.909 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:06:05.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:05 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:06:05.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:05 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:06:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:05 smithi118 ceph-mon[33862]: osdmap e113: 8 total, 8 up, 8 in 2024-03-21T14:06:05.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:05.950 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:06:05.957 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:06:05.968 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:05.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:06.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:06.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T14:06:06.099 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:06.201 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:06.229 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:06.289 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:06.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:06.359 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:06.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:06.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:06.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:06.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:06.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:06.650 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:06.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:06 smithi118 ceph-mon[33862]: pgmap v124: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 868 B/s rd, 54 MiB/s wr, 859 op/s 2024-03-21T14:06:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:06 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1252556184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:06.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:06:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:06:06.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:06 smithi003 ceph-mon[29576]: pgmap v124: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 868 B/s rd, 54 MiB/s wr, 859 op/s 2024-03-21T14:06:06.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:06 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1252556184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:06.729 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:06:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:06:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:06:06.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:06 smithi003 ceph-mon[40510]: pgmap v124: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 868 B/s rd, 54 MiB/s wr, 859 op/s 2024-03-21T14:06:06.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:06 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1252556184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:06.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:06.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:06.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:06.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 117263 117247 85.1826 63.5625 0.0340633 0.0117346 2024-03-21T14:06:06.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:06.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:06.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:06.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:07.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:07.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:07.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:07.317 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:07.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:07.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:07.450 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:07.479 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:07.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:07.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:07.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:07.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:07.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:07.803 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 118337 118321 84.975 67.125 0.0330607 0.0117631 2024-03-21T14:06:07.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:07.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:07.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:07.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:07.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:07.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:08.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:08.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:08.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:08.302 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:08.333 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:08.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:08.463 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:08.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:08.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:08.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:08.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:08 smithi003 ceph-mon[29576]: pgmap v125: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:06:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:08 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:06:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:08 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:06:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:08 smithi003 ceph-mon[40510]: pgmap v125: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:06:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:08 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:06:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:08 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:06:08.763 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:08.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:06:08.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 119412 119396 84.7726 67.1875 0.0150363 0.0117912 2024-03-21T14:06:08.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:08 smithi118 ceph-mon[33862]: pgmap v125: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:06:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:08 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:06:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:08 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:06:08.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:08.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:09.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:09.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:09.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:09.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:09.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:09.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:09.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:09.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:09.508 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:09.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:09.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:09 smithi003 ceph-mon[29576]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T14:06:09.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:09 smithi003 ceph-mon[40510]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T14:06:09.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:09.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 120226 120210 84.3918 50.875 0.0141017 0.011844 2024-03-21T14:06:09.831 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:09 smithi118 ceph-mon[33862]: osdmap e114: 8 total, 8 up, 8 in 2024-03-21T14:06:09.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:09.916 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:09.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:09.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:10.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:10.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:10.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:10.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:10.192 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:10.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:10.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:10.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:10.464 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T14:06:10.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:10.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:10.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:10.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0077 2024-03-21T14:06:10.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 121014 2024-03-21T14:06:10.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-03-21T14:06:10.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-03-21T14:06:10.804 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 84.0303 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 32.3808 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 194.625 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 50.875 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1344 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 518.094 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3114 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 814 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0118964 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.00902197 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.200511 2024-03-21T14:06:10.805 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00127999 2024-03-21T14:06:10.808 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T14:06:10.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:10.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:10.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:10.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:10 smithi118 ceph-mon[33862]: pgmap v127: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 837 op/s 2024-03-21T14:06:10.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:10 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2410164211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:06:10.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:10 smithi003 ceph-mon[29576]: pgmap v127: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 837 op/s 2024-03-21T14:06:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:10 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2410164211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:06:11.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:10 smithi003 ceph-mon[40510]: pgmap v127: 148 pgs: 148 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 837 op/s 2024-03-21T14:06:11.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:10 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2410164211' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:06:11.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:11.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:11.144 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:11.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:11.269 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:11.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:11.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:11.401 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:11.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:11.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:11.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:11.642 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:11.673 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:11.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:11.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:11.809 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:11.910 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:11.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:11 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2410164211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:06:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:11 smithi118 ceph-mon[33862]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T14:06:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:11 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3143757588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:06:11.938 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:11.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:11.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:11 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2410164211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:06:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:11 smithi003 ceph-mon[29576]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T14:06:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:11 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3143757588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:06:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:11 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2410164211' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:06:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:11 smithi003 ceph-mon[40510]: osdmap e115: 8 total, 8 up, 8 in 2024-03-21T14:06:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:11 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3143757588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:06:12.040 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.070 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:12.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:12.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:12.200 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:12.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:12.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:12.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:12.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:12.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:12.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:12.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.597 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:12.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:12.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:12.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:12.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:12.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:12.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[29576]: pgmap v129: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 61 MiB/s wr, 995 op/s 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3143757588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[29576]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[40510]: pgmap v129: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 61 MiB/s wr, 995 op/s 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3143757588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[40510]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:06:12.869 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:12 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:06:12.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:12.870 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:12.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:12 smithi118 ceph-mon[33862]: pgmap v129: 148 pgs: 148 active+clean; 7.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 61 MiB/s wr, 995 op/s 2024-03-21T14:06:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:12 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3143757588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:06:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:12 smithi118 ceph-mon[33862]: osdmap e116: 8 total, 8 up, 8 in 2024-03-21T14:06:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:12 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:06:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:12 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:06:12.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:13.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:13.112 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.140 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:13.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:13.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:13.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:13.330 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T14:06:13.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:06:13.382 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:13.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:13.458 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:06:13.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:13.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:13.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:13.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:13.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:13.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:13.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:13.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:13 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:06:13.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:13 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:06:13.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:13.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:13.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:13 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:06:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:13 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:06:14.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:13 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:06:14.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:13 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:06:14.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:14.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:14.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:14.264 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:14.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:14.309 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:06:14.309 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:06:14.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:14.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:14.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:14.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:14.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:14.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:14.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:14 smithi003 ceph-mon[29576]: pgmap v131: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 44 MiB/s wr, 784 op/s 2024-03-21T14:06:14.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:14 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:06:14.732 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:14 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:06:14.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:14 smithi003 ceph-mon[40510]: pgmap v131: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 44 MiB/s wr, 784 op/s 2024-03-21T14:06:14.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:14 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:06:14.733 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:14 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:06:14.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:14.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:14.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:14.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:14 smithi118 ceph-mon[33862]: pgmap v131: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 44 MiB/s wr, 784 op/s 2024-03-21T14:06:14.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:14 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:06:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:14 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:06:14.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:14.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:15.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:15.058 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:15.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T14:06:15.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:15.127 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:15.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:15.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:15.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:15.380 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:15.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:15.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:15.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:15.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:06:15.644 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:15 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:06:15.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:15.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:15.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:15.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:15.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:15.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:15 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:06:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:15 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:06:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:15 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:06:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:15 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:06:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:15.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:15.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:16.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:16.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:16.195 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:16.223 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:16.325 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:16.353 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:16.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:16.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:06:16.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:16.482 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:16.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:16.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:16.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:16.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:16.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: pgmap v132: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 903 B/s rd, 39 MiB/s wr, 692 op/s 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:06:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:16 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:06:16.729 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:16.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:16.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: pgmap v132: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 903 B/s rd, 39 MiB/s wr, 692 op/s 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:06:16.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:06:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: pgmap v132: 148 pgs: 148 active+clean; 7.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 903 B/s rd, 39 MiB/s wr, 692 op/s 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:06:16.759 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:16 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:06:16.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:16.886 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:16.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:16.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:17.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:17.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.159 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:17.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:17.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:17.285 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:17.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:17.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:17.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:17.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:17.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:17.537 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:17.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:17.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:17.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:17.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:17.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:17.924 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:06:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:17 smithi118 ceph-mon[33862]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T14:06:17.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:17.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:06:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:06:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:06:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:06:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:06:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:06:18.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:06:18.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:06:18.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:18.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:06:18.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:06:18.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[29576]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T14:06:18.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:06:18.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:06:18.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:06:18.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:06:18.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:06:18.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:06:18.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:06:18.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:06:18.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:18.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:06:18.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:06:18.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:17 smithi003 ceph-mon[40510]: osdmap e117: 8 total, 8 up, 8 in 2024-03-21T14:06:18.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:18.082 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:18.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:18.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:18.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:18.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:18.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:18.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:18.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:18.623 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:18.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:18.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:18.860 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:18.887 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:18.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: 6.2 scrub starts 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: 6.2 scrub ok 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: pgmap v134: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 39 MiB/s wr, 1.48k op/s 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:06:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:18 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:06:18.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: 6.2 scrub starts 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: 6.2 scrub ok 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: pgmap v134: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 39 MiB/s wr, 1.48k op/s 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: 6.2 scrub starts 2024-03-21T14:06:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: 6.2 scrub ok 2024-03-21T14:06:19.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: pgmap v134: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.6 KiB/s rd, 39 MiB/s wr, 1.48k op/s 2024-03-21T14:06:19.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:06:19.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:18 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:06:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:19.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:19.164 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:19.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:19.297 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:19.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:19.426 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:19.528 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:19.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:19.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:19.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:19.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:19.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:19.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:19.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 7.4 scrub starts 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:06:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:19 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:06:19.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:19.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:19.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:19.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 7.4 scrub starts 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:06:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 7.4 scrub starts 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:06:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:19 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:06:20.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:20.099 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:20.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:20.119 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:20.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:20.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:20.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:20.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:20.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:20.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:20.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:20.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: 7.8 scrub starts 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: pgmap v135: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 20 MiB/s wr, 1.03k op/s 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: 7.d scrub starts 2024-03-21T14:06:20.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: 7.8 scrub starts 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: pgmap v135: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 20 MiB/s wr, 1.03k op/s 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: 7.d scrub starts 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:06:20.669 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:20 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:06:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:20.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:20.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:20.807 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:20.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: 7.8 scrub starts 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: pgmap v135: 148 pgs: 148 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 20 MiB/s wr, 1.03k op/s 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: osdmap e118: 8 total, 8 up, 8 in 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: 7.d scrub starts 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:06:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:20 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:06:20.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:20.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:21.015 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:06:21.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:21.060 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:06:21.081 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:21.111 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:06:21.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:21.186 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:06:21.208 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:21.270 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:06:21.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:21.320 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:06:21.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:21.363 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:06:21.403 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:06:21.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:06:21.416 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:06:21.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:21.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:21.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:21.603 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:21.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:21.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:21.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:21.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:21.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:21 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:06:21.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:21 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:06:21.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/260199982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:21.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:21 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:06:21.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:21 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:06:21.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:21 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/260199982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:21.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:21.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:21.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:21 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:06:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:21 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:06:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:21 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/260199982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:22.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:22.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:22.142 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:22.171 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:22.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:22.304 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:22.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:22.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:22.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:22.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:22.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:22.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:22.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:22.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.859 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:22.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 7.4 scrub ok 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: pgmap v137: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 869 op/s 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 7.8 scrub ok 2024-03-21T14:06:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:22 smithi118 ceph-mon[33862]: 7.d scrub ok 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 7.4 scrub ok 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: pgmap v137: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 869 op/s 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 7.8 scrub ok 2024-03-21T14:06:22.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[29576]: 7.d scrub ok 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 7.4 scrub ok 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: pgmap v137: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 869 op/s 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 7.8 scrub ok 2024-03-21T14:06:22.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:22 smithi003 ceph-mon[40510]: 7.d scrub ok 2024-03-21T14:06:22.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:22.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:22.988 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:22.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:23.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:23.116 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:23.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:23.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:23.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:23.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:23.393 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:23.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:23.468 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:06:23.476 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:06:23.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:23.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:23.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:23.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:23.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:23.799 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:23.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:23.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:23.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:23.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:24.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:24.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:24.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:24.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:24.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:06:24.327 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:24.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:24.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.487 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:24.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:24.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:24.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:24.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:24.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:24.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:24.882 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:24.912 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:24.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:24 smithi118 ceph-mon[33862]: pgmap v138: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:06:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:24 smithi118 ceph-mon[33862]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:24 smithi118 ceph-mon[33862]: 7.0 scrub starts 2024-03-21T14:06:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:24 smithi118 ceph-mon[33862]: 7.b scrub starts 2024-03-21T14:06:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[29576]: pgmap v138: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:06:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[29576]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[29576]: 7.0 scrub starts 2024-03-21T14:06:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[29576]: 7.b scrub starts 2024-03-21T14:06:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[40510]: pgmap v138: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:06:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[40510]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:25.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[40510]: 7.0 scrub starts 2024-03-21T14:06:25.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:24 smithi003 ceph-mon[40510]: 7.b scrub starts 2024-03-21T14:06:25.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:25.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:25.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:25.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:25.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.338 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:25.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:25.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:25.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:25.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:25.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:25.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[29576]: 7.0 scrub ok 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:06:25.751 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:25 smithi003 ceph-mon[40510]: 7.0 scrub ok 2024-03-21T14:06:25.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:25.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:25.892 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:25.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:25.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:25 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:06:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:25 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:06:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:25 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:06:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:25 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:06:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:25 smithi118 ceph-mon[33862]: 7.0 scrub ok 2024-03-21T14:06:26.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:26.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:26.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.193 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:26.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:26.294 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:26.323 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:26.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:26.371 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:06:26.372 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:06:26.372 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:06:26.372 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:06:26.372 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:06:26.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:26.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:26.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:26.640 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2024-03-21T14:06:23.305539+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7264278976,"num_objects":111034,"num_object_clones":0,"num_object_copies":333102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111034,"num_whiteouts":0,"num_read":1156,"num_read_kb":1002,"num_write":131809,"num_write_kb":7747358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90410,"ondisk_log_size":90410,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22013308,"kb_used_data":21359912,"kb_used_omap":12,"kb_used_meta":653299,"kb_avail":727947908,"statfs":{"total":767960285184,"available":745418657792,"internally_reserved":0,"allocated":21872549888,"data_stored":21869256800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":668978736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-636092416,"num_objects":-9706,"num_object_clones":0,"num_object_copies":-29118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9706,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9706,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002038"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620596+0000","last_change":"2024-03-21T14:05:24.217692+0000","last_active":"2024-03-21T14:06:19.620596+0000","last_peered":"2024-03-21T14:06:19.620596+0000","last_clean":"2024-03-21T14:06:19.620596+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:06:19.620596+0000","last_undegraded":"2024-03-21T14:06:19.620596+0000","last_fullsized":"2024-03-21T14:06:19.620596+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.217564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.217564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:26.145158+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632388+0000","last_change":"2024-03-21T14:05:13.317268+0000","last_active":"2024-03-21T14:06:19.632388+0000","last_peered":"2024-03-21T14:06:19.632388+0000","last_clean":"2024-03-21T14:06:19.632388+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:06:19.632388+0000","last_undegraded":"2024-03-21T14:06:19.632388+0000","last_fullsized":"2024-03-21T14:06:19.632388+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:13.316964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:59.788675+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.316964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:27.201258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633537+0000","last_change":"2024-03-21T14:05:37.038645+0000","last_active":"2024-03-21T14:06:19.633537+0000","last_peered":"2024-03-21T14:06:19.633537+0000","last_clean":"2024-03-21T14:06:19.633537+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:06:19.633537+0000","last_undegraded":"2024-03-21T14:06:19.633537+0000","last_fullsized":"2024-03-21T14:06:19.633537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:37.038531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.038531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.964335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620549+0000","last_change":"2024-03-21T14:04:25.952183+0000","last_active":"2024-03-21T14:06:19.620549+0000","last_peered":"2024-03-21T14:06:19.620549+0000","last_clean":"2024-03-21T14:06:19.620549+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:06:19.620549+0000","last_undegraded":"2024-03-21T14:06:19.620549+0000","last_fullsized":"2024-03-21T14:06:19.620549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:04:25.952054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:04:25.952054+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631430+0000","last_change":"2024-03-21T14:05:09.665071+0000","last_active":"2024-03-21T14:06:19.631430+0000","last_peered":"2024-03-21T14:06:19.631430+0000","last_clean":"2024-03-21T14:06:19.631430+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:06:19.631430+0000","last_undegraded":"2024-03-21T14:06:19.631430+0000","last_fullsized":"2024-03-21T14:06:19.631430+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:09.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.664919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.367732+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620681+0000","last_change":"2024-03-21T14:05:32.256870+0000","last_active":"2024-03-21T14:06:19.620681+0000","last_peered":"2024-03-21T14:06:19.620681+0000","last_clean":"2024-03-21T14:06:19.620681+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:06:19.620681+0000","last_undegraded":"2024-03-21T14:06:19.620681+0000","last_fullsized":"2024-03-21T14:06:19.620681+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.256757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:51.002652+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631304+0000","last_change":"2024-03-21T14:05:32.643259+0000","last_active":"2024-03-21T14:06:19.631304+0000","last_peered":"2024-03-21T14:06:19.631304+0000","last_clean":"2024-03-21T14:06:19.631304+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:06:19.631304+0000","last_undegraded":"2024-03-21T14:06:19.631304+0000","last_fullsized":"2024-03-21T14:06:19.631304+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.643133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.643133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:00.637058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":162,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632079+0000","last_change":"2024-03-21T14:05:24.664374+0000","last_active":"2024-03-21T14:06:19.632079+0000","last_peered":"2024-03-21T14:06:19.632079+0000","last_clean":"2024-03-21T14:06:19.632079+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:06:19.632079+0000","last_undegraded":"2024-03-21T14:06:19.632079+0000","last_fullsized":"2024-03-21T14:06:19.632079+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:05:24.664099+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.664099+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.878115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630762+0000","last_change":"2024-03-21T14:05:15.176172+0000","last_active":"2024-03-21T14:06:19.630762+0000","last_peered":"2024-03-21T14:06:19.630762+0000","last_clean":"2024-03-21T14:06:19.630762+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:06:19.630762+0000","last_undegraded":"2024-03-21T14:06:19.630762+0000","last_fullsized":"2024-03-21T14:06:19.630762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:15.176047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.176047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:23.380634+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631225+0000","last_change":"2024-03-21T14:06:14.360406+0000","last_active":"2024-03-21T14:06:19.631225+0000","last_peered":"2024-03-21T14:06:19.631225+0000","last_clean":"2024-03-21T14:06:19.631225+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:06:19.631225+0000","last_undegraded":"2024-03-21T14:06:19.631225+0000","last_fullsized":"2024-03-21T14:06:19.631225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.360300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.360300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.401246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631482+0000","last_change":"2024-03-21T14:05:39.329035+0000","last_active":"2024-03-21T14:06:19.631482+0000","last_peered":"2024-03-21T14:06:19.631482+0000","last_clean":"2024-03-21T14:06:19.631482+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:06:19.631482+0000","last_undegraded":"2024-03-21T14:06:19.631482+0000","last_fullsized":"2024-03-21T14:06:19.631482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:39.328812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:39.328812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.855477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620712+0000","last_change":"2024-03-21T14:05:19.183943+0000","last_active":"2024-03-21T14:06:19.620712+0000","last_peered":"2024-03-21T14:06:19.620712+0000","last_clean":"2024-03-21T14:06:19.620712+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:06:19.620712+0000","last_undegraded":"2024-03-21T14:06:19.620712+0000","last_fullsized":"2024-03-21T14:06:19.620712+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:05:19.183805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.183805+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.082634+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630967+0000","last_change":"2024-03-21T14:05:26.392444+0000","last_active":"2024-03-21T14:06:19.630967+0000","last_peered":"2024-03-21T14:06:19.630967+0000","last_clean":"2024-03-21T14:06:19.630967+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:06:19.630967+0000","last_undegraded":"2024-03-21T14:06:19.630967+0000","last_fullsized":"2024-03-21T14:06:19.630967+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:26.392328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:26.392328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:46.561081+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.641917+0000","last_change":"2024-03-21T14:05:12.880436+0000","last_active":"2024-03-21T14:06:19.641917+0000","last_peered":"2024-03-21T14:06:19.641917+0000","last_clean":"2024-03-21T14:06:19.641917+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:06:19.641917+0000","last_undegraded":"2024-03-21T14:06:19.641917+0000","last_fullsized":"2024-03-21T14:06:19.641917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:12.880320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.880320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631398+0000","last_change":"2024-03-21T14:05:14.427489+0000","last_active":"2024-03-21T14:06:19.631398+0000","last_peered":"2024-03-21T14:06:19.631398+0000","last_clean":"2024-03-21T14:06:19.631398+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:06:19.631398+0000","last_undegraded":"2024-03-21T14:06:19.631398+0000","last_fullsized":"2024-03-21T14:06:19.631398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:14.427346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.427346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.303573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":132,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621073+0000","last_change":"2024-03-21T14:05:13.203259+0000","last_active":"2024-03-21T14:06:19.621073+0000","last_peered":"2024-03-21T14:06:19.621073+0000","last_clean":"2024-03-21T14:06:19.621073+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:06:19.621073+0000","last_undegraded":"2024-03-21T14:06:19.621073+0000","last_fullsized":"2024-03-21T14:06:19.621073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:05:13.203132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.203132+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:31.940513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621612+0000","last_change":"2024-03-21T14:05:21.139383+0000","last_active":"2024-03-21T14:06:19.621612+0000","last_peered":"2024-03-21T14:06:19.621612+0000","last_clean":"2024-03-21T14:06:19.621612+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:06:19.621612+0000","last_undegraded":"2024-03-21T14:06:19.621612+0000","last_fullsized":"2024-03-21T14:06:19.621612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:21.139263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:21.139263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.485769+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.635538+0000","last_change":"2024-03-21T14:06:17.620381+0000","last_active":"2024-03-21T14:06:19.635538+0000","last_peered":"2024-03-21T14:06:19.635538+0000","last_clean":"2024-03-21T14:06:19.635538+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:06:19.635538+0000","last_undegraded":"2024-03-21T14:06:19.635538+0000","last_fullsized":"2024-03-21T14:06:19.635538+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.620252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.620252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:21.968626+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632715+0000","last_change":"2024-03-21T14:05:23.078226+0000","last_active":"2024-03-21T14:06:19.632715+0000","last_peered":"2024-03-21T14:06:19.632715+0000","last_clean":"2024-03-21T14:06:19.632715+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:06:19.632715+0000","last_undegraded":"2024-03-21T14:06:19.632715+0000","last_fullsized":"2024-03-21T14:06:19.632715+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.078104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.078104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:23.943453+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":163,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.626266+0000","last_change":"2024-03-21T14:06:16.097848+0000","last_active":"2024-03-21T14:06:19.626266+0000","last_peered":"2024-03-21T14:06:19.626266+0000","last_clean":"2024-03-21T14:06:19.626266+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:06:19.626266+0000","last_undegraded":"2024-03-21T14:06:19.626266+0000","last_fullsized":"2024-03-21T14:06:19.626266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:06:16.097602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.097602+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.783390+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632774+0000","last_change":"2024-03-21T14:04:14.595422+0000","last_active":"2024-03-21T14:06:19.632774+0000","last_peered":"2024-03-21T14:06:19.632774+0000","last_clean":"2024-03-21T14:06:19.632774+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:06:19.632774+0000","last_undegraded":"2024-03-21T14:06:19.632774+0000","last_fullsized":"2024-03-21T14:06:19.632774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:14.595340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:14.595340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621280+0000","last_change":"2024-03-21T14:05:22.170101+0000","last_active":"2024-03-21T14:06:19.621280+0000","last_peered":"2024-03-21T14:06:19.621280+0000","last_clean":"2024-03-21T14:06:19.621280+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:06:19.621280+0000","last_undegraded":"2024-03-21T14:06:19.621280+0000","last_fullsized":"2024-03-21T14:06:19.621280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.169947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.169947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:46.686385+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620738+0000","last_change":"2024-03-21T14:05:31.213796+0000","last_active":"2024-03-21T14:06:19.620738+0000","last_peered":"2024-03-21T14:06:19.620738+0000","last_clean":"2024-03-21T14:06:19.620738+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:06:19.620738+0000","last_undegraded":"2024-03-21T14:06:19.620738+0000","last_fullsized":"2024-03-21T14:06:19.620738+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.213713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.213713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.560680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":145,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631425+0000","last_change":"2024-03-21T14:04:08.642482+0000","last_active":"2024-03-21T14:06:19.631425+0000","last_peered":"2024-03-21T14:06:19.631425+0000","last_clean":"2024-03-21T14:06:19.631425+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:06:19.631425+0000","last_undegraded":"2024-03-21T14:06:19.631425+0000","last_fullsized":"2024-03-21T14:06:19.631425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:04:08.642384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:04:08.642384+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642382+0000","last_change":"2024-03-21T14:05:22.883709+0000","last_active":"2024-03-21T14:06:19.642382+0000","last_peered":"2024-03-21T14:06:19.642382+0000","last_clean":"2024-03-21T14:06:19.642382+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:06:19.642382+0000","last_undegraded":"2024-03-21T14:06:19.642382+0000","last_fullsized":"2024-03-21T14:06:19.642382+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:22.883594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.883594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.619002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":186,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.626007+0000","last_change":"2024-03-21T14:05:09.894660+0000","last_active":"2024-03-21T14:06:19.626007+0000","last_peered":"2024-03-21T14:06:19.626007+0000","last_clean":"2024-03-21T14:06:19.626007+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:06:19.626007+0000","last_undegraded":"2024-03-21T14:06:19.626007+0000","last_fullsized":"2024-03-21T14:06:19.626007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:09.894577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.894577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.107501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632495+0000","last_change":"2024-03-21T14:05:08.269919+0000","last_active":"2024-03-21T14:06:19.632495+0000","last_peered":"2024-03-21T14:06:19.632495+0000","last_clean":"2024-03-21T14:06:19.632495+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:06:19.632495+0000","last_undegraded":"2024-03-21T14:06:19.632495+0000","last_fullsized":"2024-03-21T14:06:19.632495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:05:08.269793+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:05:08.269793+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:21.929564+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":145,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634123+0000","last_change":"2024-03-21T14:05:18.144792+0000","last_active":"2024-03-21T14:06:19.634123+0000","last_peered":"2024-03-21T14:06:19.634123+0000","last_clean":"2024-03-21T14:06:19.634123+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:06:19.634123+0000","last_undegraded":"2024-03-21T14:06:19.634123+0000","last_fullsized":"2024-03-21T14:06:19.634123+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:18.144307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.144307+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.897107+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634623+0000","last_change":"2024-03-21T14:05:35.356518+0000","last_active":"2024-03-21T14:06:19.634623+0000","last_peered":"2024-03-21T14:06:19.634623+0000","last_clean":"2024-03-21T14:06:19.634623+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:06:19.634623+0000","last_undegraded":"2024-03-21T14:06:19.634623+0000","last_fullsized":"2024-03-21T14:06:19.634623+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.356273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.356273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:00.889025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634484+0000","last_change":"2024-03-21T14:05:16.155310+0000","last_active":"2024-03-21T14:06:19.634484+0000","last_peered":"2024-03-21T14:06:19.634484+0000","last_clean":"2024-03-21T14:06:19.634484+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:06:19.634484+0000","last_undegraded":"2024-03-21T14:06:19.634484+0000","last_fullsized":"2024-03-21T14:06:19.634484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.155028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.155028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.502395+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634475+0000","last_change":"2024-03-21T14:05:28.539817+0000","last_active":"2024-03-21T14:06:19.634475+0000","last_peered":"2024-03-21T14:06:19.634475+0000","last_clean":"2024-03-21T14:06:19.634475+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:06:19.634475+0000","last_undegraded":"2024-03-21T14:06:19.634475+0000","last_fullsized":"2024-03-21T14:06:19.634475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.539674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.539674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.618118+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":149,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.625986+0000","last_change":"2024-03-21T14:05:12.953038+0000","last_active":"2024-03-21T14:06:19.625986+0000","last_peered":"2024-03-21T14:06:19.625986+0000","last_clean":"2024-03-21T14:06:19.625986+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:06:19.625986+0000","last_undegraded":"2024-03-21T14:06:19.625986+0000","last_fullsized":"2024-03-21T14:06:19.625986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:12.952946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.952946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.647609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.641810+0000","last_change":"2024-03-21T14:05:27.971209+0000","last_active":"2024-03-21T14:06:19.641810+0000","last_peered":"2024-03-21T14:06:19.641810+0000","last_clean":"2024-03-21T14:06:19.641810+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:06:19.641810+0000","last_undegraded":"2024-03-21T14:06:19.641810+0000","last_fullsized":"2024-03-21T14:06:19.641810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.971081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.971081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:52.888530+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.625660+0000","last_change":"2024-03-21T14:05:35.015627+0000","last_active":"2024-03-21T14:06:19.625660+0000","last_peered":"2024-03-21T14:06:19.625660+0000","last_clean":"2024-03-21T14:06:19.625660+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:06:19.625660+0000","last_undegraded":"2024-03-21T14:06:19.625660+0000","last_fullsized":"2024-03-21T14:06:19.625660+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.015517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.015517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.197434+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633797+0000","last_change":"2024-03-21T14:05:31.027870+0000","last_active":"2024-03-21T14:06:19.633797+0000","last_peered":"2024-03-21T14:06:19.633797+0000","last_clean":"2024-03-21T14:06:19.633797+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:06:19.633797+0000","last_undegraded":"2024-03-21T14:06:19.633797+0000","last_fullsized":"2024-03-21T14:06:19.633797+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.027754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.027754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:33.511018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":136,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.638549+0000","last_change":"2024-03-21T14:05:36.765565+0000","last_active":"2024-03-21T14:06:19.638549+0000","last_peered":"2024-03-21T14:06:19.638549+0000","last_clean":"2024-03-21T14:06:19.638549+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:06:19.638549+0000","last_undegraded":"2024-03-21T14:06:19.638549+0000","last_fullsized":"2024-03-21T14:06:19.638549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:05:36.765228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.765228+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.122075+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632213+0000","last_change":"2024-03-21T14:04:05.667110+0000","last_active":"2024-03-21T14:06:19.632213+0000","last_peered":"2024-03-21T14:06:19.632213+0000","last_clean":"2024-03-21T14:06:19.632213+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:06:19.632213+0000","last_undegraded":"2024-03-21T14:06:19.632213+0000","last_fullsized":"2024-03-21T14:06:19.632213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:05.666988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:05.666988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.641988+0000","last_change":"2024-03-21T14:05:07.887048+0000","last_active":"2024-03-21T14:06:19.641988+0000","last_peered":"2024-03-21T14:06:19.641988+0000","last_clean":"2024-03-21T14:06:19.641988+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:06:19.641988+0000","last_undegraded":"2024-03-21T14:06:19.641988+0000","last_fullsized":"2024-03-21T14:06:19.641988+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:07.886924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:07.886924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:21.006088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":140,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632735+0000","last_change":"2024-03-21T14:05:17.362797+0000","last_active":"2024-03-21T14:06:19.632735+0000","last_peered":"2024-03-21T14:06:19.632735+0000","last_clean":"2024-03-21T14:06:19.632735+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:06:19.632735+0000","last_undegraded":"2024-03-21T14:06:19.632735+0000","last_fullsized":"2024-03-21T14:06:19.632735+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:17.362665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:17.362665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:27.772347+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":149,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631976+0000","last_change":"2024-03-21T14:05:38.851695+0000","last_active":"2024-03-21T14:06:19.631976+0000","last_peered":"2024-03-21T14:06:19.631976+0000","last_clean":"2024-03-21T14:06:19.631976+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:06:19.631976+0000","last_undegraded":"2024-03-21T14:06:19.631976+0000","last_fullsized":"2024-03-21T14:06:19.631976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.849289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.849289+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:59.036012+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630471+0000","last_change":"2024-03-21T14:04:13.003296+0000","last_active":"2024-03-21T14:06:19.630471+0000","last_peered":"2024-03-21T14:06:19.630471+0000","last_clean":"2024-03-21T14:06:19.630471+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:06:19.630471+0000","last_undegraded":"2024-03-21T14:06:19.630471+0000","last_fullsized":"2024-03-21T14:06:19.630471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:13.003207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:13.003207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631340+0000","last_change":"2024-03-21T14:03:53.547267+0000","last_active":"2024-03-21T14:06:19.631340+0000","last_peered":"2024-03-21T14:06:19.631340+0000","last_clean":"2024-03-21T14:06:19.631340+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:06:19.631340+0000","last_undegraded":"2024-03-21T14:06:19.631340+0000","last_fullsized":"2024-03-21T14:06:19.631340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:53.547160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:03:53.547160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642543+0000","last_change":"2024-03-21T14:04:03.822268+0000","last_active":"2024-03-21T14:06:19.642543+0000","last_peered":"2024-03-21T14:06:19.642543+0000","last_clean":"2024-03-21T14:06:19.642543+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:06:19.642543+0000","last_undegraded":"2024-03-21T14:06:19.642543+0000","last_fullsized":"2024-03-21T14:06:19.642543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:03.822149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:03.822149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":146,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630401+0000","last_change":"2024-03-21T14:05:14.291620+0000","last_active":"2024-03-21T14:06:19.630401+0000","last_peered":"2024-03-21T14:06:19.630401+0000","last_clean":"2024-03-21T14:06:19.630401+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:06:19.630401+0000","last_undegraded":"2024-03-21T14:06:19.630401+0000","last_fullsized":"2024-03-21T14:06:19.630401+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:05:14.291294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.291294+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631638+0000","last_change":"2024-03-21T14:04:27.572389+0000","last_active":"2024-03-21T14:06:19.631638+0000","last_peered":"2024-03-21T14:06:19.631638+0000","last_clean":"2024-03-21T14:06:19.631638+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:06:19.631638+0000","last_undegraded":"2024-03-21T14:06:19.631638+0000","last_fullsized":"2024-03-21T14:06:19.631638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.572309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.572309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630150+0000","last_change":"2024-03-21T14:05:38.341843+0000","last_active":"2024-03-21T14:06:19.630150+0000","last_peered":"2024-03-21T14:06:19.630150+0000","last_clean":"2024-03-21T14:06:19.630150+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:06:19.630150+0000","last_undegraded":"2024-03-21T14:06:19.630150+0000","last_fullsized":"2024-03-21T14:06:19.630150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.341705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.341705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.933155+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621139+0000","last_change":"2024-03-21T14:05:36.278697+0000","last_active":"2024-03-21T14:06:19.621139+0000","last_peered":"2024-03-21T14:06:19.621139+0000","last_clean":"2024-03-21T14:06:19.621139+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:06:19.621139+0000","last_undegraded":"2024-03-21T14:06:19.621139+0000","last_fullsized":"2024-03-21T14:06:19.621139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.278592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.856682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":167,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621173+0000","last_change":"2024-03-21T14:05:38.294588+0000","last_active":"2024-03-21T14:06:19.621173+0000","last_peered":"2024-03-21T14:06:19.621173+0000","last_clean":"2024-03-21T14:06:19.621173+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:06:19.621173+0000","last_undegraded":"2024-03-21T14:06:19.621173+0000","last_fullsized":"2024-03-21T14:06:19.621173+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.294441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.294441+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:01.276451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633354+0000","last_change":"2024-03-21T14:05:38.046118+0000","last_active":"2024-03-21T14:06:19.633354+0000","last_peered":"2024-03-21T14:06:19.633354+0000","last_clean":"2024-03-21T14:06:19.633354+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:06:19.633354+0000","last_undegraded":"2024-03-21T14:06:19.633354+0000","last_fullsized":"2024-03-21T14:06:19.633354+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.045998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.045998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:52.031731+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":180,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631533+0000","last_change":"2024-03-21T14:05:13.668961+0000","last_active":"2024-03-21T14:06:19.631533+0000","last_peered":"2024-03-21T14:06:19.631533+0000","last_clean":"2024-03-21T14:06:19.631533+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:06:19.631533+0000","last_undegraded":"2024-03-21T14:06:19.631533+0000","last_fullsized":"2024-03-21T14:06:19.631533+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:13.668830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.668830+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:24.307265+0000","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642117+0000","last_change":"2024-03-21T14:04:17.788255+0000","last_active":"2024-03-21T14:06:19.642117+0000","last_peered":"2024-03-21T14:06:19.642117+0000","last_clean":"2024-03-21T14:06:19.642117+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:06:19.642117+0000","last_undegraded":"2024-03-21T14:06:19.642117+0000","last_fullsized":"2024-03-21T14:06:19.642117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:04:17.788113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":137,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634709+0000","last_change":"2024-03-21T14:05:14.713009+0000","last_active":"2024-03-21T14:06:19.634709+0000","last_peered":"2024-03-21T14:06:19.634709+0000","last_clean":"2024-03-21T14:06:19.634709+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:06:19.634709+0000","last_undegraded":"2024-03-21T14:06:19.634709+0000","last_fullsized":"2024-03-21T14:06:19.634709+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.712091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.712091+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:44.503021+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.622090+0000","last_change":"2024-03-21T14:04:26.959485+0000","last_active":"2024-03-21T14:06:19.622090+0000","last_peered":"2024-03-21T14:06:19.622090+0000","last_clean":"2024-03-21T14:06:19.622090+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:06:19.622090+0000","last_undegraded":"2024-03-21T14:06:19.622090+0000","last_fullsized":"2024-03-21T14:06:19.622090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:26.959343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:26.959343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":180,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:20.079241+0000","last_change":"2024-03-21T14:06:20.079291+0000","last_active":"2024-03-21T14:06:20.079241+0000","last_peered":"2024-03-21T14:06:20.079241+0000","last_clean":"2024-03-21T14:06:20.079241+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:06:20.079241+0000","last_undegraded":"2024-03-21T14:06:20.079241+0000","last_fullsized":"2024-03-21T14:06:20.079241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:20.079142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.079142+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.032560+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:20.884295+0000","last_change":"2024-03-21T14:06:20.884332+0000","last_active":"2024-03-21T14:06:20.884295+0000","last_peered":"2024-03-21T14:06:20.884295+0000","last_clean":"2024-03-21T14:06:20.884295+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:06:20.884295+0000","last_undegraded":"2024-03-21T14:06:20.884295+0000","last_fullsized":"2024-03-21T14:06:20.884295+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:20.884210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.884210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.916333+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":146,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:20.431354+0000","last_change":"2024-03-21T14:06:20.431407+0000","last_active":"2024-03-21T14:06:20.431354+0000","last_peered":"2024-03-21T14:06:20.431354+0000","last_clean":"2024-03-21T14:06:20.431354+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:06:20.431354+0000","last_undegraded":"2024-03-21T14:06:20.431354+0000","last_fullsized":"2024-03-21T14:06:20.431354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:20.431279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.431279+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.061041+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.627038+0000","last_change":"2024-03-21T14:05:29.391237+0000","last_active":"2024-03-21T14:06:19.627038+0000","last_peered":"2024-03-21T14:06:19.627038+0000","last_clean":"2024-03-21T14:06:19.627038+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:06:19.627038+0000","last_undegraded":"2024-03-21T14:06:19.627038+0000","last_fullsized":"2024-03-21T14:06:19.627038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.391091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.391091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.808745+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633279+0000","last_change":"2024-03-21T14:04:22.564083+0000","last_active":"2024-03-21T14:06:19.633279+0000","last_peered":"2024-03-21T14:06:19.633279+0000","last_clean":"2024-03-21T14:06:19.633279+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:06:19.633279+0000","last_undegraded":"2024-03-21T14:06:19.633279+0000","last_fullsized":"2024-03-21T14:06:19.633279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:22.563986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:22.563986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.641803+0000","last_change":"2024-03-21T14:05:16.811057+0000","last_active":"2024-03-21T14:06:19.641803+0000","last_peered":"2024-03-21T14:06:19.641803+0000","last_clean":"2024-03-21T14:06:19.641803+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:06:19.641803+0000","last_undegraded":"2024-03-21T14:06:19.641803+0000","last_fullsized":"2024-03-21T14:06:19.641803+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.810928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.810928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:26.117133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":146,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634337+0000","last_change":"2024-03-21T14:06:16.830628+0000","last_active":"2024-03-21T14:06:19.634337+0000","last_peered":"2024-03-21T14:06:19.634337+0000","last_clean":"2024-03-21T14:06:19.634337+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:06:19.634337+0000","last_undegraded":"2024-03-21T14:06:19.634337+0000","last_fullsized":"2024-03-21T14:06:19.634337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:16.830503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.830503+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:17.447722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642481+0000","last_change":"2024-03-21T14:05:15.857954+0000","last_active":"2024-03-21T14:06:19.642481+0000","last_peered":"2024-03-21T14:06:19.642481+0000","last_clean":"2024-03-21T14:06:19.642481+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:06:19.642481+0000","last_undegraded":"2024-03-21T14:06:19.642481+0000","last_fullsized":"2024-03-21T14:06:19.642481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:15.857799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.857799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:42.408109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632751+0000","last_change":"2024-03-21T14:05:28.362497+0000","last_active":"2024-03-21T14:06:19.632751+0000","last_peered":"2024-03-21T14:06:19.632751+0000","last_clean":"2024-03-21T14:06:19.632751+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:06:19.632751+0000","last_undegraded":"2024-03-21T14:06:19.632751+0000","last_fullsized":"2024-03-21T14:06:19.632751+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.362424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.362424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.774731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.625407+0000","last_change":"2024-03-21T14:05:08.883023+0000","last_active":"2024-03-21T14:06:19.625407+0000","last_peered":"2024-03-21T14:06:19.625407+0000","last_clean":"2024-03-21T14:06:19.625407+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:06:19.625407+0000","last_undegraded":"2024-03-21T14:06:19.625407+0000","last_fullsized":"2024-03-21T14:06:19.625407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:08.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:08.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:20.019324+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":131,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633107+0000","last_change":"2024-03-21T14:05:15.725635+0000","last_active":"2024-03-21T14:06:19.633107+0000","last_peered":"2024-03-21T14:06:19.633107+0000","last_clean":"2024-03-21T14:06:19.633107+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:06:19.633107+0000","last_undegraded":"2024-03-21T14:06:19.633107+0000","last_fullsized":"2024-03-21T14:06:19.633107+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:05:15.725275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.725275+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"118'8263","reported_seq":8331,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.323924+0000","last_change":"2024-03-21T14:04:28.368834+0000","last_active":"2024-03-21T14:06:21.323924+0000","last_peered":"2024-03-21T14:06:21.323924+0000","last_clean":"2024-03-21T14:06:21.323924+0000","last_became_active":"2024-03-21T14:04:28.368403+0000","last_became_peered":"2024-03-21T14:04:28.368403+0000","last_unstale":"2024-03-21T14:06:21.323924+0000","last_undegraded":"2024-03-21T14:06:21.323924+0000","last_fullsized":"2024-03-21T14:06:21.323924+0000","mapping_epoch":89,"log_start":"98'3200","ondisk_log_start":"98'3200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5063,"log_dups_size":0,"ondisk_log_size":5063,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457768960,"num_objects":6985,"num_object_clones":0,"num_object_copies":20955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8263,"num_write_kb":487936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631289+0000","last_change":"2024-03-21T14:06:16.410140+0000","last_active":"2024-03-21T14:06:19.631289+0000","last_peered":"2024-03-21T14:06:19.631289+0000","last_clean":"2024-03-21T14:06:19.631289+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:06:19.631289+0000","last_undegraded":"2024-03-21T14:06:19.631289+0000","last_fullsized":"2024-03-21T14:06:19.631289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:16.409991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.409991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.498428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":138,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.645659+0000","last_change":"2024-03-21T14:04:06.838826+0000","last_active":"2024-03-21T14:06:19.645659+0000","last_peered":"2024-03-21T14:06:19.645659+0000","last_clean":"2024-03-21T14:06:19.645659+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:06:19.645659+0000","last_undegraded":"2024-03-21T14:06:19.645659+0000","last_fullsized":"2024-03-21T14:06:19.645659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:04:06.838694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:06.838694+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":155,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630958+0000","last_change":"2024-03-21T14:05:40.337551+0000","last_active":"2024-03-21T14:06:19.630958+0000","last_peered":"2024-03-21T14:06:19.630958+0000","last_clean":"2024-03-21T14:06:19.630958+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:06:19.630958+0000","last_undegraded":"2024-03-21T14:06:19.630958+0000","last_fullsized":"2024-03-21T14:06:19.630958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:40.337278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:40.337278+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.381891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":172,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642061+0000","last_change":"2024-03-21T14:04:27.810344+0000","last_active":"2024-03-21T14:06:19.642061+0000","last_peered":"2024-03-21T14:06:19.642061+0000","last_clean":"2024-03-21T14:06:19.642061+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:06:19.642061+0000","last_undegraded":"2024-03-21T14:06:19.642061+0000","last_fullsized":"2024-03-21T14:06:19.642061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:04:27.810260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.810260+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"118'8233","reported_seq":8301,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.400581+0000","last_change":"2024-03-21T14:04:28.366763+0000","last_active":"2024-03-21T14:06:21.400581+0000","last_peered":"2024-03-21T14:06:21.400581+0000","last_clean":"2024-03-21T14:06:21.400581+0000","last_became_active":"2024-03-21T14:04:28.366208+0000","last_became_peered":"2024-03-21T14:04:28.366208+0000","last_unstale":"2024-03-21T14:06:21.400581+0000","last_undegraded":"2024-03-21T14:06:21.400581+0000","last_fullsized":"2024-03-21T14:06:21.400581+0000","mapping_epoch":89,"log_start":"99'3600","ondisk_log_start":"99'3600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4633,"log_dups_size":0,"ondisk_log_size":4633,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455409664,"num_objects":6949,"num_object_clones":0,"num_object_copies":20847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8233,"num_write_kb":485824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631663+0000","last_change":"2024-03-21T14:05:23.704653+0000","last_active":"2024-03-21T14:06:19.631663+0000","last_peered":"2024-03-21T14:06:19.631663+0000","last_clean":"2024-03-21T14:06:19.631663+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:06:19.631663+0000","last_undegraded":"2024-03-21T14:06:19.631663+0000","last_fullsized":"2024-03-21T14:06:19.631663+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.703365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.703365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:31.525757+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642498+0000","last_change":"2024-03-21T14:05:19.908957+0000","last_active":"2024-03-21T14:06:19.642498+0000","last_peered":"2024-03-21T14:06:19.642498+0000","last_clean":"2024-03-21T14:06:19.642498+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:06:19.642498+0000","last_undegraded":"2024-03-21T14:06:19.642498+0000","last_fullsized":"2024-03-21T14:06:19.642498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.908814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.908814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:40.177906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":145,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.622343+0000","last_change":"2024-03-21T14:05:12.169866+0000","last_active":"2024-03-21T14:06:19.622343+0000","last_peered":"2024-03-21T14:06:19.622343+0000","last_clean":"2024-03-21T14:06:19.622343+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:06:19.622343+0000","last_undegraded":"2024-03-21T14:06:19.622343+0000","last_fullsized":"2024-03-21T14:06:19.622343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:05:12.169735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.169735+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.683299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.640235+0000","last_change":"2024-03-21T14:05:21.248283+0000","last_active":"2024-03-21T14:06:19.640235+0000","last_peered":"2024-03-21T14:06:19.640235+0000","last_clean":"2024-03-21T14:06:19.640235+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:06:19.640235+0000","last_undegraded":"2024-03-21T14:06:19.640235+0000","last_fullsized":"2024-03-21T14:06:19.640235+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:21.248165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:21.248165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:29.883952+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"118'8210","reported_seq":8278,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.350803+0000","last_change":"2024-03-21T14:04:28.368993+0000","last_active":"2024-03-21T14:06:21.350803+0000","last_peered":"2024-03-21T14:06:21.350803+0000","last_clean":"2024-03-21T14:06:21.350803+0000","last_became_active":"2024-03-21T14:04:28.367553+0000","last_became_peered":"2024-03-21T14:04:28.367553+0000","last_unstale":"2024-03-21T14:06:21.350803+0000","last_undegraded":"2024-03-21T14:06:21.350803+0000","last_fullsized":"2024-03-21T14:06:21.350803+0000","mapping_epoch":89,"log_start":"97'2600","ondisk_log_start":"97'2600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447217664,"num_objects":6824,"num_object_clones":0,"num_object_copies":20472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8210,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.641626+0000","last_change":"2024-03-21T14:03:58.825134+0000","last_active":"2024-03-21T14:06:19.641626+0000","last_peered":"2024-03-21T14:06:19.641626+0000","last_clean":"2024-03-21T14:06:19.641626+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:06:19.641626+0000","last_undegraded":"2024-03-21T14:06:19.641626+0000","last_fullsized":"2024-03-21T14:06:19.641626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:58.825002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:03:58.825002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.626620+0000","last_change":"2024-03-21T14:05:20.225369+0000","last_active":"2024-03-21T14:06:19.626620+0000","last_peered":"2024-03-21T14:06:19.626620+0000","last_clean":"2024-03-21T14:06:19.626620+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:06:19.626620+0000","last_undegraded":"2024-03-21T14:06:19.626620+0000","last_fullsized":"2024-03-21T14:06:19.626620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:20.225234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.225234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.070460+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642653+0000","last_change":"2024-03-21T14:06:19.057440+0000","last_active":"2024-03-21T14:06:19.642653+0000","last_peered":"2024-03-21T14:06:19.642653+0000","last_clean":"2024-03-21T14:06:19.642653+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:06:19.642653+0000","last_undegraded":"2024-03-21T14:06:19.642653+0000","last_fullsized":"2024-03-21T14:06:19.642653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:19.057319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:19.057319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.818134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":153,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.625567+0000","last_change":"2024-03-21T14:04:15.977228+0000","last_active":"2024-03-21T14:06:19.625567+0000","last_peered":"2024-03-21T14:06:19.625567+0000","last_clean":"2024-03-21T14:06:19.625567+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:06:19.625567+0000","last_undegraded":"2024-03-21T14:06:19.625567+0000","last_fullsized":"2024-03-21T14:06:19.625567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:04:15.977079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:04:15.977079+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"118'8222","reported_seq":8363,"reported_epoch":118,"state":"active+clean+scrubbing","last_fresh":"2024-03-21T14:06:21.421715+0000","last_change":"2024-03-21T14:06:19.133227+0000","last_active":"2024-03-21T14:06:21.421715+0000","last_peered":"2024-03-21T14:06:21.421715+0000","last_clean":"2024-03-21T14:06:21.421715+0000","last_became_active":"2024-03-21T14:04:28.360990+0000","last_became_peered":"2024-03-21T14:04:28.360990+0000","last_unstale":"2024-03-21T14:06:21.421715+0000","last_undegraded":"2024-03-21T14:06:21.421715+0000","last_fullsized":"2024-03-21T14:06:21.421715+0000","mapping_epoch":89,"log_start":"94'1400","ondisk_log_start":"94'1400","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":6849,"log_size":6822,"log_dups_size":0,"ondisk_log_size":6822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453246976,"num_objects":6916,"num_object_clones":0,"num_object_copies":20748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":484416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633141+0000","last_change":"2024-03-21T14:05:34.000829+0000","last_active":"2024-03-21T14:06:19.633141+0000","last_peered":"2024-03-21T14:06:19.633141+0000","last_clean":"2024-03-21T14:06:19.633141+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:06:19.633141+0000","last_undegraded":"2024-03-21T14:06:19.633141+0000","last_fullsized":"2024-03-21T14:06:19.633141+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.000689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.026990+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633822+0000","last_change":"2024-03-21T14:05:09.245728+0000","last_active":"2024-03-21T14:06:19.633822+0000","last_peered":"2024-03-21T14:06:19.633822+0000","last_clean":"2024-03-21T14:06:19.633822+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:06:19.633822+0000","last_undegraded":"2024-03-21T14:06:19.633822+0000","last_fullsized":"2024-03-21T14:06:19.633822+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:09.245624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.245624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:25.145892+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631477+0000","last_change":"2024-03-21T14:05:23.381853+0000","last_active":"2024-03-21T14:06:19.631477+0000","last_peered":"2024-03-21T14:06:19.631477+0000","last_clean":"2024-03-21T14:06:19.631477+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:06:19.631477+0000","last_undegraded":"2024-03-21T14:06:19.631477+0000","last_fullsized":"2024-03-21T14:06:19.631477+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.381726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.381726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.801210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":162,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632834+0000","last_change":"2024-03-21T14:06:15.846648+0000","last_active":"2024-03-21T14:06:19.632834+0000","last_peered":"2024-03-21T14:06:19.632834+0000","last_clean":"2024-03-21T14:06:19.632834+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:06:19.632834+0000","last_undegraded":"2024-03-21T14:06:19.632834+0000","last_fullsized":"2024-03-21T14:06:19.632834+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:15.846504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.846504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.520598+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"118'8041","reported_seq":8109,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.273368+0000","last_change":"2024-03-21T14:04:28.369386+0000","last_active":"2024-03-21T14:06:21.273368+0000","last_peered":"2024-03-21T14:06:21.273368+0000","last_clean":"2024-03-21T14:06:21.273368+0000","last_became_active":"2024-03-21T14:04:28.369020+0000","last_became_peered":"2024-03-21T14:04:28.369020+0000","last_unstale":"2024-03-21T14:06:21.273368+0000","last_undegraded":"2024-03-21T14:06:21.273368+0000","last_fullsized":"2024-03-21T14:06:21.273368+0000","mapping_epoch":89,"log_start":"96'1700","ondisk_log_start":"96'1700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446365696,"num_objects":6811,"num_object_clones":0,"num_object_copies":20433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8041,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631999+0000","last_change":"2024-03-21T14:05:16.720441+0000","last_active":"2024-03-21T14:06:19.631999+0000","last_peered":"2024-03-21T14:06:19.631999+0000","last_clean":"2024-03-21T14:06:19.631999+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:06:19.631999+0000","last_undegraded":"2024-03-21T14:06:19.631999+0000","last_fullsized":"2024-03-21T14:06:19.631999+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.720331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.720331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.562402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":145,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631157+0000","last_change":"2024-03-21T14:05:11.492743+0000","last_active":"2024-03-21T14:06:19.631157+0000","last_peered":"2024-03-21T14:06:19.631157+0000","last_clean":"2024-03-21T14:06:19.631157+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:06:19.631157+0000","last_undegraded":"2024-03-21T14:06:19.631157+0000","last_fullsized":"2024-03-21T14:06:19.631157+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:05:11.492609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.492609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:26.850596+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620966+0000","last_change":"2024-03-21T14:05:18.141165+0000","last_active":"2024-03-21T14:06:19.620966+0000","last_peered":"2024-03-21T14:06:19.620966+0000","last_clean":"2024-03-21T14:06:19.620966+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:06:19.620966+0000","last_undegraded":"2024-03-21T14:06:19.620966+0000","last_fullsized":"2024-03-21T14:06:19.620966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:18.140955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.140955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.081222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":178,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631123+0000","last_change":"2024-03-21T14:05:12.482027+0000","last_active":"2024-03-21T14:06:19.631123+0000","last_peered":"2024-03-21T14:06:19.631123+0000","last_clean":"2024-03-21T14:06:19.631123+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:06:19.631123+0000","last_undegraded":"2024-03-21T14:06:19.631123+0000","last_fullsized":"2024-03-21T14:06:19.631123+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:05:12.481864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.481864+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.106831+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633186+0000","last_change":"2024-03-21T14:06:14.840624+0000","last_active":"2024-03-21T14:06:19.633186+0000","last_peered":"2024-03-21T14:06:19.633186+0000","last_clean":"2024-03-21T14:06:19.633186+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:06:19.633186+0000","last_undegraded":"2024-03-21T14:06:19.633186+0000","last_fullsized":"2024-03-21T14:06:19.633186+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.840509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.840509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.244776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"118'8383","reported_seq":8531,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.391028+0000","last_change":"2024-03-21T14:06:21.025026+0000","last_active":"2024-03-21T14:06:21.391028+0000","last_peered":"2024-03-21T14:06:21.391028+0000","last_clean":"2024-03-21T14:06:21.391028+0000","last_became_active":"2024-03-21T14:04:28.367387+0000","last_became_peered":"2024-03-21T14:04:28.367387+0000","last_unstale":"2024-03-21T14:06:21.391028+0000","last_undegraded":"2024-03-21T14:06:21.391028+0000","last_fullsized":"2024-03-21T14:06:21.391028+0000","mapping_epoch":89,"log_start":"97'2800","ondisk_log_start":"97'2800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8368","last_scrub_stamp":"2024-03-21T14:06:21.024886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.024886+0000","objects_scrubbed":7201,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.840880+0000","scrub_duration":3116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464191488,"num_objects":7083,"num_object_clones":0,"num_object_copies":21249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8383,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642746+0000","last_change":"2024-03-21T14:06:11.955196+0000","last_active":"2024-03-21T14:06:19.642746+0000","last_peered":"2024-03-21T14:06:19.642746+0000","last_clean":"2024-03-21T14:06:19.642746+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:06:19.642746+0000","last_undegraded":"2024-03-21T14:06:19.642746+0000","last_fullsized":"2024-03-21T14:06:19.642746+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:11.955082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:11.955082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.410006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631345+0000","last_change":"2024-03-21T14:05:22.226038+0000","last_active":"2024-03-21T14:06:19.631345+0000","last_peered":"2024-03-21T14:06:19.631345+0000","last_clean":"2024-03-21T14:06:19.631345+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:06:19.631345+0000","last_undegraded":"2024-03-21T14:06:19.631345+0000","last_fullsized":"2024-03-21T14:06:19.631345+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.225815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.225815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:48.284109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":151,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.626471+0000","last_change":"2024-03-21T14:06:12.171064+0000","last_active":"2024-03-21T14:06:19.626471+0000","last_peered":"2024-03-21T14:06:19.626471+0000","last_clean":"2024-03-21T14:06:19.626471+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:06:19.626471+0000","last_undegraded":"2024-03-21T14:06:19.626471+0000","last_fullsized":"2024-03-21T14:06:19.626471+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:12.170919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:12.170919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.011481+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":162,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630457+0000","last_change":"2024-03-21T14:05:11.256911+0000","last_active":"2024-03-21T14:06:19.630457+0000","last_peered":"2024-03-21T14:06:19.630457+0000","last_clean":"2024-03-21T14:06:19.630457+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:06:19.630457+0000","last_undegraded":"2024-03-21T14:06:19.630457+0000","last_fullsized":"2024-03-21T14:06:19.630457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:05:11.256782+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.256782+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:29.753889+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632946+0000","last_change":"2024-03-21T14:05:13.199758+0000","last_active":"2024-03-21T14:06:19.632946+0000","last_peered":"2024-03-21T14:06:19.632946+0000","last_clean":"2024-03-21T14:06:19.632946+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:06:19.632946+0000","last_undegraded":"2024-03-21T14:06:19.632946+0000","last_fullsized":"2024-03-21T14:06:19.632946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:13.199466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.199466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:26.574284+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"118'8220","reported_seq":8288,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.367275+0000","last_change":"2024-03-21T14:04:28.364561+0000","last_active":"2024-03-21T14:06:21.367275+0000","last_peered":"2024-03-21T14:06:21.367275+0000","last_clean":"2024-03-21T14:06:21.367275+0000","last_became_active":"2024-03-21T14:04:28.363923+0000","last_became_peered":"2024-03-21T14:04:28.363923+0000","last_unstale":"2024-03-21T14:06:21.367275+0000","last_undegraded":"2024-03-21T14:06:21.367275+0000","last_fullsized":"2024-03-21T14:06:21.367275+0000","mapping_epoch":89,"log_start":"97'2600","ondisk_log_start":"97'2600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5620,"log_dups_size":0,"ondisk_log_size":5620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454557696,"num_objects":6936,"num_object_clones":0,"num_object_copies":20808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8220,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631588+0000","last_change":"2024-03-21T14:06:15.375779+0000","last_active":"2024-03-21T14:06:19.631588+0000","last_peered":"2024-03-21T14:06:19.631588+0000","last_clean":"2024-03-21T14:06:19.631588+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:06:19.631588+0000","last_undegraded":"2024-03-21T14:06:19.631588+0000","last_fullsized":"2024-03-21T14:06:19.631588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:15.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.375667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.848357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":387,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.641742+0000","last_change":"2024-03-21T14:05:31.964155+0000","last_active":"2024-03-21T14:06:19.641742+0000","last_peered":"2024-03-21T14:06:19.641742+0000","last_clean":"2024-03-21T14:06:19.641742+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:06:19.641742+0000","last_undegraded":"2024-03-21T14:06:19.641742+0000","last_fullsized":"2024-03-21T14:06:19.641742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-03-21T14:05:31.964052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.964052+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:51.267636+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632084+0000","last_change":"2024-03-21T14:04:03.733569+0000","last_active":"2024-03-21T14:06:19.632084+0000","last_peered":"2024-03-21T14:06:19.632084+0000","last_clean":"2024-03-21T14:06:19.632084+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:06:19.632084+0000","last_undegraded":"2024-03-21T14:06:19.632084+0000","last_fullsized":"2024-03-21T14:06:19.632084+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:03.733488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:03.733488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":198,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.628544+0000","last_change":"2024-03-21T14:05:25.911867+0000","last_active":"2024-03-21T14:06:19.628544+0000","last_peered":"2024-03-21T14:06:19.628544+0000","last_clean":"2024-03-21T14:06:19.628544+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:06:19.628544+0000","last_undegraded":"2024-03-21T14:06:19.628544+0000","last_fullsized":"2024-03-21T14:06:19.628544+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:05:25.911729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:25.911729+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.703513+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642263+0000","last_change":"2024-03-21T14:04:13.817866+0000","last_active":"2024-03-21T14:06:19.642263+0000","last_peered":"2024-03-21T14:06:19.642263+0000","last_clean":"2024-03-21T14:06:19.642263+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:06:19.642263+0000","last_undegraded":"2024-03-21T14:06:19.642263+0000","last_fullsized":"2024-03-21T14:06:19.642263+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:13.817745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:13.817745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"118'8103","reported_seq":8171,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.238345+0000","last_change":"2024-03-21T14:04:28.363760+0000","last_active":"2024-03-21T14:06:21.238345+0000","last_peered":"2024-03-21T14:06:21.238345+0000","last_clean":"2024-03-21T14:06:21.238345+0000","last_became_active":"2024-03-21T14:04:28.363404+0000","last_became_peered":"2024-03-21T14:04:28.363404+0000","last_unstale":"2024-03-21T14:06:21.238345+0000","last_undegraded":"2024-03-21T14:06:21.238345+0000","last_fullsized":"2024-03-21T14:06:21.238345+0000","mapping_epoch":89,"log_start":"97'2900","ondisk_log_start":"97'2900","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5203,"log_dups_size":0,"ondisk_log_size":5203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448200704,"num_objects":6839,"num_object_clones":0,"num_object_copies":20517,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6839,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8103,"num_write_kb":478144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.629838+0000","last_change":"2024-03-21T14:05:36.379278+0000","last_active":"2024-03-21T14:06:19.629838+0000","last_peered":"2024-03-21T14:06:19.629838+0000","last_clean":"2024-03-21T14:06:19.629838+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:06:19.629838+0000","last_undegraded":"2024-03-21T14:06:19.629838+0000","last_fullsized":"2024-03-21T14:06:19.629838+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.376091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.376091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.586944+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633353+0000","last_change":"2024-03-21T14:04:17.076807+0000","last_active":"2024-03-21T14:06:19.633353+0000","last_peered":"2024-03-21T14:06:19.633353+0000","last_clean":"2024-03-21T14:06:19.633353+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:06:19.633353+0000","last_undegraded":"2024-03-21T14:06:19.633353+0000","last_fullsized":"2024-03-21T14:06:19.633353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:17.076724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:17.076724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":367,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.742317+0000","last_change":"2024-03-21T14:06:13.952697+0000","last_active":"2024-03-21T14:06:19.742317+0000","last_peered":"2024-03-21T14:06:19.742317+0000","last_clean":"2024-03-21T14:06:19.742317+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:06:19.742317+0000","last_undegraded":"2024-03-21T14:06:19.742317+0000","last_fullsized":"2024-03-21T14:06:19.742317+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:06:13.952449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.952449+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.957684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":221,"num_read_kb":216,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":148,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634015+0000","last_change":"2024-03-21T14:06:18.867126+0000","last_active":"2024-03-21T14:06:19.634015+0000","last_peered":"2024-03-21T14:06:19.634015+0000","last_clean":"2024-03-21T14:06:19.634015+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:06:19.634015+0000","last_undegraded":"2024-03-21T14:06:19.634015+0000","last_fullsized":"2024-03-21T14:06:19.634015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:18.866994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.866994+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.123054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631811+0000","last_change":"2024-03-21T14:04:18.327801+0000","last_active":"2024-03-21T14:06:19.631811+0000","last_peered":"2024-03-21T14:06:19.631811+0000","last_clean":"2024-03-21T14:06:19.631811+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:06:19.631811+0000","last_undegraded":"2024-03-21T14:06:19.631811+0000","last_fullsized":"2024-03-21T14:06:19.631811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:18.327704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:18.327704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"118'8119","reported_seq":8187,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.241485+0000","last_change":"2024-03-21T14:04:28.368173+0000","last_active":"2024-03-21T14:06:21.241485+0000","last_peered":"2024-03-21T14:06:21.241485+0000","last_clean":"2024-03-21T14:06:21.241485+0000","last_became_active":"2024-03-21T14:04:28.367882+0000","last_became_peered":"2024-03-21T14:04:28.367882+0000","last_unstale":"2024-03-21T14:06:21.241485+0000","last_undegraded":"2024-03-21T14:06:21.241485+0000","last_fullsized":"2024-03-21T14:06:21.241485+0000","mapping_epoch":89,"log_start":"97'2300","ondisk_log_start":"97'2300","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5819,"log_dups_size":0,"ondisk_log_size":5819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455671808,"num_objects":6953,"num_object_clones":0,"num_object_copies":20859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8119,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632286+0000","last_change":"2024-03-21T14:05:34.327891+0000","last_active":"2024-03-21T14:06:19.632286+0000","last_peered":"2024-03-21T14:06:19.632286+0000","last_clean":"2024-03-21T14:06:19.632286+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:06:19.632286+0000","last_undegraded":"2024-03-21T14:06:19.632286+0000","last_fullsized":"2024-03-21T14:06:19.632286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.327068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.327068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.990999+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632679+0000","last_change":"2024-03-21T14:05:19.237932+0000","last_active":"2024-03-21T14:06:19.632679+0000","last_peered":"2024-03-21T14:06:19.632679+0000","last_clean":"2024-03-21T14:06:19.632679+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:06:19.632679+0000","last_undegraded":"2024-03-21T14:06:19.632679+0000","last_fullsized":"2024-03-21T14:06:19.632679+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.237452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.237452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:39.545828+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":166,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632175+0000","last_change":"2024-03-21T14:05:24.049566+0000","last_active":"2024-03-21T14:06:19.632175+0000","last_peered":"2024-03-21T14:06:19.632175+0000","last_clean":"2024-03-21T14:06:19.632175+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:06:19.632175+0000","last_undegraded":"2024-03-21T14:06:19.632175+0000","last_fullsized":"2024-03-21T14:06:19.632175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:05:24.049421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.049421+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.688262+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.625638+0000","last_change":"2024-03-21T14:06:18.996323+0000","last_active":"2024-03-21T14:06:19.625638+0000","last_peered":"2024-03-21T14:06:19.625638+0000","last_clean":"2024-03-21T14:06:19.625638+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:06:19.625638+0000","last_undegraded":"2024-03-21T14:06:19.625638+0000","last_fullsized":"2024-03-21T14:06:19.625638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.996252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.996252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.294909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"118'8164","reported_seq":8232,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.160796+0000","last_change":"2024-03-21T14:04:28.369265+0000","last_active":"2024-03-21T14:06:21.160796+0000","last_peered":"2024-03-21T14:06:21.160796+0000","last_clean":"2024-03-21T14:06:21.160796+0000","last_became_active":"2024-03-21T14:04:28.369002+0000","last_became_peered":"2024-03-21T14:04:28.369002+0000","last_unstale":"2024-03-21T14:06:21.160796+0000","last_undegraded":"2024-03-21T14:06:21.160796+0000","last_fullsized":"2024-03-21T14:06:21.160796+0000","mapping_epoch":89,"log_start":"99'3400","ondisk_log_start":"99'3400","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452198400,"num_objects":6900,"num_object_clones":0,"num_object_copies":20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8164,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630570+0000","last_change":"2024-03-21T14:05:11.153864+0000","last_active":"2024-03-21T14:06:19.630570+0000","last_peered":"2024-03-21T14:06:19.630570+0000","last_clean":"2024-03-21T14:06:19.630570+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:06:19.630570+0000","last_undegraded":"2024-03-21T14:06:19.630570+0000","last_fullsized":"2024-03-21T14:06:19.630570+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:11.153740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.153740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:29.073768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631597+0000","last_change":"2024-03-21T14:06:18.451847+0000","last_active":"2024-03-21T14:06:19.631597+0000","last_peered":"2024-03-21T14:06:19.631597+0000","last_clean":"2024-03-21T14:06:19.631597+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:06:19.631597+0000","last_undegraded":"2024-03-21T14:06:19.631597+0000","last_fullsized":"2024-03-21T14:06:19.631597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.451772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.451772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.610702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":145,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630244+0000","last_change":"2024-03-21T14:05:12.303597+0000","last_active":"2024-03-21T14:06:19.630244+0000","last_peered":"2024-03-21T14:06:19.630244+0000","last_clean":"2024-03-21T14:06:19.630244+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:06:19.630244+0000","last_undegraded":"2024-03-21T14:06:19.630244+0000","last_fullsized":"2024-03-21T14:06:19.630244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:12.303265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.303265+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:22.706188+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":180,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632960+0000","last_change":"2024-03-21T14:06:13.786619+0000","last_active":"2024-03-21T14:06:19.632960+0000","last_peered":"2024-03-21T14:06:19.632960+0000","last_clean":"2024-03-21T14:06:19.632960+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:06:19.632960+0000","last_undegraded":"2024-03-21T14:06:19.632960+0000","last_fullsized":"2024-03-21T14:06:19.632960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:13.786485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.786485+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.654794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"118'8246","reported_seq":8313,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.381914+0000","last_change":"2024-03-21T14:04:28.366919+0000","last_active":"2024-03-21T14:06:21.381914+0000","last_peered":"2024-03-21T14:06:21.381914+0000","last_clean":"2024-03-21T14:06:21.381914+0000","last_became_active":"2024-03-21T14:04:28.366634+0000","last_became_peered":"2024-03-21T14:04:28.366634+0000","last_unstale":"2024-03-21T14:06:21.381914+0000","last_undegraded":"2024-03-21T14:06:21.381914+0000","last_fullsized":"2024-03-21T14:06:21.381914+0000","mapping_epoch":89,"log_start":"99'3600","ondisk_log_start":"99'3600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4646,"log_dups_size":0,"ondisk_log_size":4646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454426624,"num_objects":6934,"num_object_clones":0,"num_object_copies":20802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8246,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.642227+0000","last_change":"2024-03-21T14:04:08.760984+0000","last_active":"2024-03-21T14:06:19.642227+0000","last_peered":"2024-03-21T14:06:19.642227+0000","last_clean":"2024-03-21T14:06:19.642227+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:06:19.642227+0000","last_undegraded":"2024-03-21T14:06:19.642227+0000","last_fullsized":"2024-03-21T14:06:19.642227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:08.760868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:08.760868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":135,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.633951+0000","last_change":"2024-03-21T14:05:20.137081+0000","last_active":"2024-03-21T14:06:19.633951+0000","last_peered":"2024-03-21T14:06:19.633951+0000","last_clean":"2024-03-21T14:06:19.633951+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:06:19.633951+0000","last_undegraded":"2024-03-21T14:06:19.633951+0000","last_fullsized":"2024-03-21T14:06:19.633951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:20.136955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.136955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.349906+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":151,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.622233+0000","last_change":"2024-03-21T14:06:18.141733+0000","last_active":"2024-03-21T14:06:19.622233+0000","last_peered":"2024-03-21T14:06:19.622233+0000","last_clean":"2024-03-21T14:06:19.622233+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:06:19.622233+0000","last_undegraded":"2024-03-21T14:06:19.622233+0000","last_fullsized":"2024-03-21T14:06:19.622233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:06:18.141619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.141619+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:29.717483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":172,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.637892+0000","last_change":"2024-03-21T14:04:09.593691+0000","last_active":"2024-03-21T14:06:19.637892+0000","last_peered":"2024-03-21T14:06:19.637892+0000","last_clean":"2024-03-21T14:06:19.637892+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:06:19.637892+0000","last_undegraded":"2024-03-21T14:06:19.637892+0000","last_fullsized":"2024-03-21T14:06:19.637892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:04:09.593571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:09.593571+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"118'8255","reported_seq":8323,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.405266+0000","last_change":"2024-03-21T14:04:28.364630+0000","last_active":"2024-03-21T14:06:21.405266+0000","last_peered":"2024-03-21T14:06:21.405266+0000","last_clean":"2024-03-21T14:06:21.405266+0000","last_became_active":"2024-03-21T14:04:28.364379+0000","last_became_peered":"2024-03-21T14:04:28.364379+0000","last_unstale":"2024-03-21T14:06:21.405266+0000","last_undegraded":"2024-03-21T14:06:21.405266+0000","last_fullsized":"2024-03-21T14:06:21.405266+0000","mapping_epoch":89,"log_start":"99'3600","ondisk_log_start":"99'3600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4655,"log_dups_size":0,"ondisk_log_size":4655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455671808,"num_objects":6953,"num_object_clones":0,"num_object_copies":20859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8255,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.620598+0000","last_change":"2024-03-21T14:06:17.164352+0000","last_active":"2024-03-21T14:06:19.620598+0000","last_peered":"2024-03-21T14:06:19.620598+0000","last_clean":"2024-03-21T14:06:19.620598+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:06:19.620598+0000","last_undegraded":"2024-03-21T14:06:19.620598+0000","last_fullsized":"2024-03-21T14:06:19.620598+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.164232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.164232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:43.626287+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632967+0000","last_change":"2024-03-21T14:05:29.550541+0000","last_active":"2024-03-21T14:06:19.632967+0000","last_peered":"2024-03-21T14:06:19.632967+0000","last_clean":"2024-03-21T14:06:19.632967+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:06:19.632967+0000","last_undegraded":"2024-03-21T14:06:19.632967+0000","last_fullsized":"2024-03-21T14:06:19.632967+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.548379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.548379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:55.684852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":162,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634922+0000","last_change":"2024-03-21T14:06:16.045314+0000","last_active":"2024-03-21T14:06:19.634922+0000","last_peered":"2024-03-21T14:06:19.634922+0000","last_clean":"2024-03-21T14:06:19.634922+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:06:19.634922+0000","last_undegraded":"2024-03-21T14:06:19.634922+0000","last_fullsized":"2024-03-21T14:06:19.634922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:06:16.045170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.045170+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:28.072141+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.622048+0000","last_change":"2024-03-21T14:05:27.174995+0000","last_active":"2024-03-21T14:06:19.622048+0000","last_peered":"2024-03-21T14:06:19.622048+0000","last_clean":"2024-03-21T14:06:19.622048+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:06:19.622048+0000","last_undegraded":"2024-03-21T14:06:19.622048+0000","last_fullsized":"2024-03-21T14:06:19.622048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.174851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.174851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.192729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"118'8034","reported_seq":8102,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.233146+0000","last_change":"2024-03-21T14:04:28.369284+0000","last_active":"2024-03-21T14:06:21.233146+0000","last_peered":"2024-03-21T14:06:21.233146+0000","last_clean":"2024-03-21T14:06:21.233146+0000","last_became_active":"2024-03-21T14:04:28.368956+0000","last_became_peered":"2024-03-21T14:04:28.368956+0000","last_unstale":"2024-03-21T14:06:21.233146+0000","last_undegraded":"2024-03-21T14:06:21.233146+0000","last_fullsized":"2024-03-21T14:06:21.233146+0000","mapping_epoch":89,"log_start":"96'1700","ondisk_log_start":"96'1700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":6334,"log_dups_size":0,"ondisk_log_size":6334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":20502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8034,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":122,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631424+0000","last_change":"2024-03-21T14:05:24.417631+0000","last_active":"2024-03-21T14:06:19.631424+0000","last_peered":"2024-03-21T14:06:19.631424+0000","last_clean":"2024-03-21T14:06:19.631424+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:06:19.631424+0000","last_undegraded":"2024-03-21T14:06:19.631424+0000","last_fullsized":"2024-03-21T14:06:19.631424+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.417495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.417495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:33.538419+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":180,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621688+0000","last_change":"2024-03-21T14:05:37.280589+0000","last_active":"2024-03-21T14:06:19.621688+0000","last_peered":"2024-03-21T14:06:19.621688+0000","last_clean":"2024-03-21T14:06:19.621688+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:06:19.621688+0000","last_undegraded":"2024-03-21T14:06:19.621688+0000","last_fullsized":"2024-03-21T14:06:19.621688+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:37.280472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.280472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:55.817892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.634674+0000","last_change":"2024-03-21T14:03:56.674692+0000","last_active":"2024-03-21T14:06:19.634674+0000","last_peered":"2024-03-21T14:06:19.634674+0000","last_clean":"2024-03-21T14:06:19.634674+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:06:19.634674+0000","last_undegraded":"2024-03-21T14:06:19.634674+0000","last_fullsized":"2024-03-21T14:06:19.634674+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:03:56.674610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:03:56.674610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":155,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.630382+0000","last_change":"2024-03-21T14:05:29.265158+0000","last_active":"2024-03-21T14:06:19.630382+0000","last_peered":"2024-03-21T14:06:19.630382+0000","last_clean":"2024-03-21T14:06:19.630382+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:06:19.630382+0000","last_undegraded":"2024-03-21T14:06:19.630382+0000","last_fullsized":"2024-03-21T14:06:19.630382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:29.265009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.265009+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.852216+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"118'8081","reported_seq":8199,"reported_epoch":118,"state":"active+clean+scrubbing","last_fresh":"2024-03-21T14:06:21.270531+0000","last_change":"2024-03-21T14:06:19.994193+0000","last_active":"2024-03-21T14:06:21.270531+0000","last_peered":"2024-03-21T14:06:21.270531+0000","last_clean":"2024-03-21T14:06:21.270531+0000","last_became_active":"2024-03-21T14:04:28.371337+0000","last_became_peered":"2024-03-21T14:04:28.371337+0000","last_unstale":"2024-03-21T14:06:21.270531+0000","last_undegraded":"2024-03-21T14:06:21.270531+0000","last_fullsized":"2024-03-21T14:06:21.270531+0000","mapping_epoch":89,"log_start":"96'1800","ondisk_log_start":"96'1800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":4599,"log_size":6281,"log_dups_size":0,"ondisk_log_size":6281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450625560,"num_objects":6877,"num_object_clones":0,"num_object_copies":20631,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6877,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8081,"num_write_kb":478593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632332+0000","last_change":"2024-03-21T14:05:10.657107+0000","last_active":"2024-03-21T14:06:19.632332+0000","last_peered":"2024-03-21T14:06:19.632332+0000","last_clean":"2024-03-21T14:06:19.632332+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:06:19.632332+0000","last_undegraded":"2024-03-21T14:06:19.632332+0000","last_fullsized":"2024-03-21T14:06:19.632332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:10.656780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:10.656780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.396348+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":118,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631760+0000","last_change":"2024-03-21T14:04:22.324361+0000","last_active":"2024-03-21T14:06:19.631760+0000","last_peered":"2024-03-21T14:06:19.631760+0000","last_clean":"2024-03-21T14:06:19.631760+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:06:19.631760+0000","last_undegraded":"2024-03-21T14:06:19.631760+0000","last_fullsized":"2024-03-21T14:06:19.631760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:22.324257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:22.324257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":134,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.622143+0000","last_change":"2024-03-21T14:05:33.267089+0000","last_active":"2024-03-21T14:06:19.622143+0000","last_peered":"2024-03-21T14:06:19.622143+0000","last_clean":"2024-03-21T14:06:19.622143+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:06:19.622143+0000","last_undegraded":"2024-03-21T14:06:19.622143+0000","last_fullsized":"2024-03-21T14:06:19.622143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:33.266985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.266985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:42.374468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":149,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.107973+0000","last_change":"2024-03-21T14:06:21.108018+0000","last_active":"2024-03-21T14:06:21.107973+0000","last_peered":"2024-03-21T14:06:21.107973+0000","last_clean":"2024-03-21T14:06:21.107973+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:06:21.107973+0000","last_undegraded":"2024-03-21T14:06:21.107973+0000","last_fullsized":"2024-03-21T14:06:21.107973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:06:21.107886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.107886+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.591227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"118'8250","reported_seq":8381,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.401436+0000","last_change":"2024-03-21T14:05:36.765363+0000","last_active":"2024-03-21T14:06:21.401436+0000","last_peered":"2024-03-21T14:06:21.401436+0000","last_clean":"2024-03-21T14:06:21.401436+0000","last_became_active":"2024-03-21T14:04:28.365174+0000","last_became_peered":"2024-03-21T14:04:28.365174+0000","last_unstale":"2024-03-21T14:06:21.401436+0000","last_undegraded":"2024-03-21T14:06:21.401436+0000","last_fullsized":"2024-03-21T14:06:21.401436+0000","mapping_epoch":89,"log_start":"94'1400","ondisk_log_start":"94'1400","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'5496","last_scrub_stamp":"2024-03-21T14:05:36.764945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.764945+0000","objects_scrubbed":5416,"log_size":6850,"log_dups_size":0,"ondisk_log_size":6850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:44.153675+0000","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456261632,"num_objects":6962,"num_object_clones":0,"num_object_copies":20886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8250,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":123,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632839+0000","last_change":"2024-03-21T14:05:33.294251+0000","last_active":"2024-03-21T14:06:19.632839+0000","last_peered":"2024-03-21T14:06:19.632839+0000","last_clean":"2024-03-21T14:06:19.632839+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:06:19.632839+0000","last_undegraded":"2024-03-21T14:06:19.632839+0000","last_fullsized":"2024-03-21T14:06:19.632839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:33.294119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.294119+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.044466+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.632830+0000","last_change":"2024-03-21T14:05:30.277978+0000","last_active":"2024-03-21T14:06:19.632830+0000","last_peered":"2024-03-21T14:06:19.632830+0000","last_clean":"2024-03-21T14:06:19.632830+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:06:19.632830+0000","last_undegraded":"2024-03-21T14:06:19.632830+0000","last_fullsized":"2024-03-21T14:06:19.632830+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:30.277645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:30.277645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.448928+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.625682+0000","last_change":"2024-03-21T14:04:16.928165+0000","last_active":"2024-03-21T14:06:19.625682+0000","last_peered":"2024-03-21T14:06:19.625682+0000","last_clean":"2024-03-21T14:06:19.625682+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:06:19.625682+0000","last_undegraded":"2024-03-21T14:06:19.625682+0000","last_fullsized":"2024-03-21T14:06:19.625682+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:16.928081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:16.928081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":158,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.622310+0000","last_change":"2024-03-21T14:06:16.191465+0000","last_active":"2024-03-21T14:06:19.622310+0000","last_peered":"2024-03-21T14:06:19.622310+0000","last_clean":"2024-03-21T14:06:19.622310+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:06:19.622310+0000","last_undegraded":"2024-03-21T14:06:19.622310+0000","last_fullsized":"2024-03-21T14:06:19.622310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:06:16.191249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.191249+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.656713+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"118'8370","reported_seq":8438,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:21.400751+0000","last_change":"2024-03-21T14:04:28.364839+0000","last_active":"2024-03-21T14:06:21.400751+0000","last_peered":"2024-03-21T14:06:21.400751+0000","last_clean":"2024-03-21T14:06:21.400751+0000","last_became_active":"2024-03-21T14:04:28.364570+0000","last_became_peered":"2024-03-21T14:04:28.364570+0000","last_unstale":"2024-03-21T14:06:21.400751+0000","last_undegraded":"2024-03-21T14:06:21.400751+0000","last_fullsized":"2024-03-21T14:06:21.400751+0000","mapping_epoch":89,"log_start":"97'2800","ondisk_log_start":"97'2800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5570,"log_dups_size":0,"ondisk_log_size":5570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463994880,"num_objects":7080,"num_object_clones":0,"num_object_copies":21240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7080,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8370,"num_write_kb":494400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":114,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621777+0000","last_change":"2024-03-21T14:05:11.171257+0000","last_active":"2024-03-21T14:06:19.621777+0000","last_peered":"2024-03-21T14:06:19.621777+0000","last_clean":"2024-03-21T14:06:19.621777+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:06:19.621777+0000","last_undegraded":"2024-03-21T14:06:19.621777+0000","last_fullsized":"2024-03-21T14:06:19.621777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:11.171149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.171149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:21.639804+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":181,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.626477+0000","last_change":"2024-03-21T14:05:32.346872+0000","last_active":"2024-03-21T14:06:19.626477+0000","last_peered":"2024-03-21T14:06:19.626477+0000","last_clean":"2024-03-21T14:06:19.626477+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:06:19.626477+0000","last_undegraded":"2024-03-21T14:06:19.626477+0000","last_fullsized":"2024-03-21T14:06:19.626477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.346553+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:43.290171+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":126,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.631236+0000","last_change":"2024-03-21T14:04:10.596303+0000","last_active":"2024-03-21T14:06:19.631236+0000","last_peered":"2024-03-21T14:06:19.631236+0000","last_clean":"2024-03-21T14:06:19.631236+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:06:19.631236+0000","last_undegraded":"2024-03-21T14:06:19.631236+0000","last_fullsized":"2024-03-21T14:06:19.631236+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:10.596224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:10.596224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":137,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-03-21T14:06:19.621233+0000","last_change":"2024-03-21T14:05:14.162485+0000","last_active":"2024-03-21T14:06:19.621233+0000","last_peered":"2024-03-21T14:06:19.621233+0000","last_clean":"2024-03-21T14:06:19.621233+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:06:19.621233+0000","last_undegraded":"2024-03-21T14:06:19.621233+0000","last_fullsized":"2024-03-21T14:06:19.621233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.162391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.162391+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.569569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7263682584,"num_objects":110836,"num_object_clones":0,"num_object_copies":332508,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110836,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131194,"num_write_kb":7744897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21789683712,"data_stored":21789671496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89794,"ondisk_log_size":89794,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":221,"num_read_kb":216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103218,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2286492,"kb_used_data":2219220,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91458660,"statfs":{"total":95995035648,"available":93653667840,"internally_reserved":0,"allocated":2272481280,"data_stored":2272071756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430263,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2283400,"kb_used_data":2207808,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91461752,"statfs":{"total":95995035648,"available":93656834048,"internally_reserved":0,"allocated":2260795392,"data_stored":2260404409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2775992,"kb_used_data":2682348,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":90969160,"statfs":{"total":95995035648,"available":93152419840,"internally_reserved":0,"allocated":2746724352,"data_stored":2746290323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3661092,"kb_used_data":3561880,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":90084060,"statfs":{"total":95995035648,"available":92246077440,"internally_reserved":0,"allocated":3647365120,"data_stored":3646950355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2738952,"kb_used_data":2667520,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91006200,"statfs":{"total":95995035648,"available":93190348800,"internally_reserved":0,"allocated":2731540480,"data_stored":2731149277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4112176,"kb_used_data":4000484,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":89632976,"statfs":{"total":95995035648,"available":91784167424,"internally_reserved":0,"allocated":4096495616,"data_stored":4096068541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018946,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2291368,"kb_used_data":2224092,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91453784,"statfs":{"total":95995035648,"available":93648674816,"internally_reserved":0,"allocated":2277470208,"data_stored":2277051039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691901,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1863836,"kb_used_data":1796560,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91881316,"statfs":{"total":95995035648,"available":94086467584,"internally_reserved":0,"allocated":1839677440,"data_stored":1839271100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3629518848,"data_stored":3629514776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2729381888,"data_stored":2729377816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2722758656,"data_stored":2722758656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4082958336,"data_stored":4082958336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2274230272,"data_stored":2274230272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1826619392,"data_stored":1826619392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2255163392,"data_stored":2255159320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2269052928,"data_stored":2269052928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:06:26.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:06:26.648 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:06:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:06:26.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: pgmap v139: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 504 op/s 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 7.b scrub ok 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:06:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:26 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:06:26.702 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:26.731 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:26.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:26.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:26.860 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:26.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:26.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: pgmap v139: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 504 op/s 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 7.b scrub ok 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:06:26.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: pgmap v139: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 586 B/s rd, 0 B/s wr, 504 op/s 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 7.b scrub ok 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:06:26.932 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:26 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:06:26.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:26.990 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:26.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:27.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:27.120 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:27.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:27.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:27.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:27.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T14:06:27.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:27.401 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:27.503 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:27.531 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:27.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:27.586 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:27.633 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:27.660 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:27.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:27.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:27.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:27.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:27.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:27.917 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:27 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:06:27.918 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:27 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:06:27.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:27.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:27 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:06:27.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:27 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:06:27.921 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:27.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:27.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:27.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:27 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:06:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:27 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:06:28.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:28.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:28.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:28.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:28.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:28.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:28.312 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:28.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:28.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:28.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:28.586 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:28.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:28.681 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:06:28.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:28.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:28.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:28.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:28.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:28.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:28.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:28.975 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.007 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:29.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:29.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: pgmap v140: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-03-21T14:06:29.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:06:29.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/782425570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: pgmap v140: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:06:29.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/782425570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:06:29.009 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:28 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:06:29.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:29.137 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: pgmap v140: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/782425570' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:06:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:28 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:06:29.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:29.267 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:29.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:29.397 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:29.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:29.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:29.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:29.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:29.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.796 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:29.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:29.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:29.909 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:29.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:29.958 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:29.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:29.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: osdmap e119: 8 total, 8 up, 8 in 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: 5.13 deep-scrub starts 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[29576]: 5.13 deep-scrub ok 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:06:29.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:29.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: osdmap e119: 8 total, 8 up, 8 in 2024-03-21T14:06:29.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: 5.13 deep-scrub starts 2024-03-21T14:06:29.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:29 smithi003 ceph-mon[40510]: 5.13 deep-scrub ok 2024-03-21T14:06:30.059 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:30.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:30.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: osdmap e119: 8 total, 8 up, 8 in 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: 5.13 deep-scrub starts 2024-03-21T14:06:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:29 smithi118 ceph-mon[33862]: 5.13 deep-scrub ok 2024-03-21T14:06:30.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:30.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:30.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:30.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:30.358 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:30.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:30.488 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:30.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:30.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:30.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:30.749 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: pgmap v142: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 982 op/s 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:06:30.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:06:30.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: pgmap v142: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 982 op/s 2024-03-21T14:06:30.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:06:30.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:06:30.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:06:30.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:30 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:30.851 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:30.878 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:30.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:30.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:30.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:31.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:31.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:31.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:31.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:31.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: pgmap v142: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 982 op/s 2024-03-21T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:06:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:30 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:31.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:31.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:31.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:31.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:31.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:31.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:31.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:31.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:31.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:31.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:31.699 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:31.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:31.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:31.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:06:31.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:31 smithi003 ceph-mon[29576]: 7.6 deep-scrub starts 2024-03-21T14:06:31.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:31 smithi003 ceph-mon[29576]: 5.19 deep-scrub starts 2024-03-21T14:06:31.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:31 smithi003 ceph-mon[29576]: 5.19 deep-scrub ok 2024-03-21T14:06:31.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:31 smithi003 ceph-mon[40510]: 7.6 deep-scrub starts 2024-03-21T14:06:31.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:31 smithi003 ceph-mon[40510]: 5.19 deep-scrub starts 2024-03-21T14:06:31.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:31 smithi003 ceph-mon[40510]: 5.19 deep-scrub ok 2024-03-21T14:06:31.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:31.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:31.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:32.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:32.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:31 smithi118 ceph-mon[33862]: 7.6 deep-scrub starts 2024-03-21T14:06:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:31 smithi118 ceph-mon[33862]: 5.19 deep-scrub starts 2024-03-21T14:06:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:31 smithi118 ceph-mon[33862]: 5.19 deep-scrub ok 2024-03-21T14:06:32.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:32.252 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:06:32.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:32.301 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:06:32.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:06:32.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:32.386 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:06:32.393 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:32.425 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:06:32.467 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:06:32.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:32.510 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:06:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:32.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:32.563 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:06:32.566 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:06:32.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:32.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: pgmap v143: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-03-21T14:06:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:06:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:06:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: 7.1 scrub starts 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/246895413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: pgmap v143: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:06:32.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:06:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:06:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: 7.1 scrub starts 2024-03-21T14:06:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/246895413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:32.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:32.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:32.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:32.949 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:32.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:32.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:33.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:33.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:33.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: pgmap v143: 148 pgs: 148 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: 7.1 scrub starts 2024-03-21T14:06:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/246895413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:33.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.253 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:33.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:33.354 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:33.381 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:33.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:33.512 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:33.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:33.642 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:33.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:33.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:33.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:33.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:33.853 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:33.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:33.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[40510]: 7.9 scrub starts 2024-03-21T14:06:33.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:06:33.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:06:33.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:06:33.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:06:33.952 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[40510]: 7.1 scrub ok 2024-03-21T14:06:33.955 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[29576]: 7.9 scrub starts 2024-03-21T14:06:33.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:06:33.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:06:33.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:06:33.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:06:33.956 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:33 smithi003 ceph-mon[29576]: 7.1 scrub ok 2024-03-21T14:06:33.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:33.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:34.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:34.103 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:33 smithi118 ceph-mon[33862]: 7.9 scrub starts 2024-03-21T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:33 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:33 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:33 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:33 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:06:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:33 smithi118 ceph-mon[33862]: 7.1 scrub ok 2024-03-21T14:06:34.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:34.228 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:34.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:34.358 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:34.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:34.483 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:34.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:34.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:34.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:34.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:34.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:34.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:34.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:34.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:34.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:35.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:34 smithi003 ceph-mon[29576]: pgmap v144: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:06:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:34 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:06:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:34 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:06:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:34 smithi003 ceph-mon[40510]: pgmap v144: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:06:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:34 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:06:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:34 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:06:35.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:35.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:35.166 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:34 smithi118 ceph-mon[33862]: pgmap v144: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:06:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:34 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:06:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:34 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:06:35.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:35.296 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:35.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:35.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:35.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:35.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:35.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:35.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 7.9 scrub ok 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 7.f scrub starts 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 7.7 deep-scrub starts 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:06:35.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 7.9 scrub ok 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 7.f scrub starts 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 7.7 deep-scrub starts 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:06:35.805 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:35 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:06:35.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:35.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:35.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:36.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:36.031 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:36.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 7.9 scrub ok 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 7.f scrub starts 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 7.7 deep-scrub starts 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:06:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:35 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:06:36.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:36.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:36.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:36.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:36.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:36.457 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:36.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:36.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:36.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:36.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:36.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:36.748 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:06:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:06:36.749 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:06:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:06:36.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:36.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:36.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:36.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[29576]: pgmap v145: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[29576]: 6.17 scrub starts 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[29576]: 6.17 scrub ok 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:06:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:06:37.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[40510]: pgmap v145: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:06:37.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[40510]: 6.17 scrub starts 2024-03-21T14:06:37.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:36 smithi003 ceph-mon[40510]: 6.17 scrub ok 2024-03-21T14:06:37.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:37.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:37.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:37.150 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:37.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:36 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:06:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:36 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:06:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:36 smithi118 ceph-mon[33862]: pgmap v145: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.1 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:06:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:36 smithi118 ceph-mon[33862]: 6.17 scrub starts 2024-03-21T14:06:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:36 smithi118 ceph-mon[33862]: 6.17 scrub ok 2024-03-21T14:06:37.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:37.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:37.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:37.388 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:37.418 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:37.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:37.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:37.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:37.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:06:37.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:37.666 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:06:37.666 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:06:37.687 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:37.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:37.815 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:37.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:37.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:37.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:37.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:37.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:06:38.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:06:38.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:37 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:06:38.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:38.072 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:06:38.080 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:06:38.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:38.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:06:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:37 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:06:38.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:38.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:38.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:38.303 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:06:38.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:38.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:38.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:38.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:38.502 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:38.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:38.612 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:06:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:38.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:38.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:38.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:38.892 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:38.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:38.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:38.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[29576]: pgmap v146: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[29576]: 7.a scrub starts 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[40510]: pgmap v146: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[40510]: 7.a scrub starts 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:06:38.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:38 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:06:39.007 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-03-21T14:06:37.309595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6342646208,"num_objects":96971,"num_object_clones":0,"num_object_copies":290913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96971,"num_whiteouts":0,"num_read":1171,"num_read_kb":1017,"num_write":145872,"num_write_kb":7747358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90542,"ondisk_log_size":90542,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19329384,"kb_used_data":18632872,"kb_used_omap":12,"kb_used_meta":696307,"kb_avail":730631832,"statfs":{"total":767960285184,"available":748166995968,"internally_reserved":0,"allocated":19080060928,"data_stored":19077139240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":713018928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,2,2,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-587202560,"num_objects":-8960,"num_object_clones":0,"num_object_copies":-26880,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8960,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8960,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002964"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670703+0000","last_change":"2024-03-21T14:05:24.217692+0000","last_active":"2024-03-21T14:06:28.670703+0000","last_peered":"2024-03-21T14:06:28.670703+0000","last_clean":"2024-03-21T14:06:28.670703+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:06:28.670703+0000","last_undegraded":"2024-03-21T14:06:28.670703+0000","last_fullsized":"2024-03-21T14:06:28.670703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.217564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.217564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:30.718694+0000","last_change":"2024-03-21T14:06:30.718732+0000","last_active":"2024-03-21T14:06:30.718694+0000","last_peered":"2024-03-21T14:06:30.718694+0000","last_clean":"2024-03-21T14:06:30.718694+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:06:30.718694+0000","last_undegraded":"2024-03-21T14:06:30.718694+0000","last_fullsized":"2024-03-21T14:06:30.718694+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:06:30.718605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:47.380214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694003+0000","last_change":"2024-03-21T14:05:37.038645+0000","last_active":"2024-03-21T14:06:28.694003+0000","last_peered":"2024-03-21T14:06:28.694003+0000","last_clean":"2024-03-21T14:06:28.694003+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:06:28.694003+0000","last_undegraded":"2024-03-21T14:06:28.694003+0000","last_fullsized":"2024-03-21T14:06:28.694003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:37.038531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.038531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.964335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":140,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670543+0000","last_change":"2024-03-21T14:04:25.952183+0000","last_active":"2024-03-21T14:06:28.670543+0000","last_peered":"2024-03-21T14:06:28.670543+0000","last_clean":"2024-03-21T14:06:28.670543+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:06:28.670543+0000","last_undegraded":"2024-03-21T14:06:28.670543+0000","last_fullsized":"2024-03-21T14:06:28.670543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:04:25.952054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:04:25.952054+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688272+0000","last_change":"2024-03-21T14:05:09.665071+0000","last_active":"2024-03-21T14:06:28.688272+0000","last_peered":"2024-03-21T14:06:28.688272+0000","last_clean":"2024-03-21T14:06:28.688272+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:06:28.688272+0000","last_undegraded":"2024-03-21T14:06:28.688272+0000","last_fullsized":"2024-03-21T14:06:28.688272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:09.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.664919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.367732+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670606+0000","last_change":"2024-03-21T14:05:32.256870+0000","last_active":"2024-03-21T14:06:28.670606+0000","last_peered":"2024-03-21T14:06:28.670606+0000","last_clean":"2024-03-21T14:06:28.670606+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:06:28.670606+0000","last_undegraded":"2024-03-21T14:06:28.670606+0000","last_fullsized":"2024-03-21T14:06:28.670606+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.256757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:51.002652+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687578+0000","last_change":"2024-03-21T14:05:32.643259+0000","last_active":"2024-03-21T14:06:28.687578+0000","last_peered":"2024-03-21T14:06:28.687578+0000","last_clean":"2024-03-21T14:06:28.687578+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:06:28.687578+0000","last_undegraded":"2024-03-21T14:06:28.687578+0000","last_fullsized":"2024-03-21T14:06:28.687578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.643133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.643133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:00.637058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685173+0000","last_change":"2024-03-21T14:05:24.664374+0000","last_active":"2024-03-21T14:06:28.685173+0000","last_peered":"2024-03-21T14:06:28.685173+0000","last_clean":"2024-03-21T14:06:28.685173+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:06:28.685173+0000","last_undegraded":"2024-03-21T14:06:28.685173+0000","last_fullsized":"2024-03-21T14:06:28.685173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:05:24.664099+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.664099+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.878115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.967481+0000","last_change":"2024-03-21T14:06:34.967518+0000","last_active":"2024-03-21T14:06:34.967481+0000","last_peered":"2024-03-21T14:06:34.967481+0000","last_clean":"2024-03-21T14:06:34.967481+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:06:34.967481+0000","last_undegraded":"2024-03-21T14:06:34.967481+0000","last_fullsized":"2024-03-21T14:06:34.967481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.967385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.967385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:55.339956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686053+0000","last_change":"2024-03-21T14:06:14.360406+0000","last_active":"2024-03-21T14:06:28.686053+0000","last_peered":"2024-03-21T14:06:28.686053+0000","last_clean":"2024-03-21T14:06:28.686053+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:06:28.686053+0000","last_undegraded":"2024-03-21T14:06:28.686053+0000","last_fullsized":"2024-03-21T14:06:28.686053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.360300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.360300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.401246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684202+0000","last_change":"2024-03-21T14:05:39.329035+0000","last_active":"2024-03-21T14:06:28.684202+0000","last_peered":"2024-03-21T14:06:28.684202+0000","last_clean":"2024-03-21T14:06:28.684202+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:06:28.684202+0000","last_undegraded":"2024-03-21T14:06:28.684202+0000","last_fullsized":"2024-03-21T14:06:28.684202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:39.328812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:39.328812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.855477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":140,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671242+0000","last_change":"2024-03-21T14:05:19.183943+0000","last_active":"2024-03-21T14:06:28.671242+0000","last_peered":"2024-03-21T14:06:28.671242+0000","last_clean":"2024-03-21T14:06:28.671242+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:06:28.671242+0000","last_undegraded":"2024-03-21T14:06:28.671242+0000","last_fullsized":"2024-03-21T14:06:28.671242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:05:19.183805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.183805+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.082634+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682986+0000","last_change":"2024-03-21T14:05:26.392444+0000","last_active":"2024-03-21T14:06:28.682986+0000","last_peered":"2024-03-21T14:06:28.682986+0000","last_clean":"2024-03-21T14:06:28.682986+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:06:28.682986+0000","last_undegraded":"2024-03-21T14:06:28.682986+0000","last_fullsized":"2024-03-21T14:06:28.682986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:26.392328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:26.392328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:46.561081+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686492+0000","last_change":"2024-03-21T14:05:12.880436+0000","last_active":"2024-03-21T14:06:28.686492+0000","last_peered":"2024-03-21T14:06:28.686492+0000","last_clean":"2024-03-21T14:06:28.686492+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:06:28.686492+0000","last_undegraded":"2024-03-21T14:06:28.686492+0000","last_fullsized":"2024-03-21T14:06:28.686492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:12.880320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.880320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683701+0000","last_change":"2024-03-21T14:05:14.427489+0000","last_active":"2024-03-21T14:06:28.683701+0000","last_peered":"2024-03-21T14:06:28.683701+0000","last_clean":"2024-03-21T14:06:28.683701+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:06:28.683701+0000","last_undegraded":"2024-03-21T14:06:28.683701+0000","last_fullsized":"2024-03-21T14:06:28.683701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:14.427346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.427346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.303573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671368+0000","last_change":"2024-03-21T14:05:13.203259+0000","last_active":"2024-03-21T14:06:28.671368+0000","last_peered":"2024-03-21T14:06:28.671368+0000","last_clean":"2024-03-21T14:06:28.671368+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:06:28.671368+0000","last_undegraded":"2024-03-21T14:06:28.671368+0000","last_fullsized":"2024-03-21T14:06:28.671368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:05:13.203132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.203132+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:31.940513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671129+0000","last_change":"2024-03-21T14:05:21.139383+0000","last_active":"2024-03-21T14:06:28.671129+0000","last_peered":"2024-03-21T14:06:28.671129+0000","last_clean":"2024-03-21T14:06:28.671129+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:06:28.671129+0000","last_undegraded":"2024-03-21T14:06:28.671129+0000","last_fullsized":"2024-03-21T14:06:28.671129+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:21.139263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:21.139263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.485769+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.681749+0000","last_change":"2024-03-21T14:06:17.620381+0000","last_active":"2024-03-21T14:06:28.681749+0000","last_peered":"2024-03-21T14:06:28.681749+0000","last_clean":"2024-03-21T14:06:28.681749+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:06:28.681749+0000","last_undegraded":"2024-03-21T14:06:28.681749+0000","last_fullsized":"2024-03-21T14:06:28.681749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.620252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.620252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:21.968626+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":144,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.002405+0000","last_change":"2024-03-21T14:06:36.002443+0000","last_active":"2024-03-21T14:06:36.002405+0000","last_peered":"2024-03-21T14:06:36.002405+0000","last_clean":"2024-03-21T14:06:36.002405+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:06:36.002405+0000","last_undegraded":"2024-03-21T14:06:36.002405+0000","last_fullsized":"2024-03-21T14:06:36.002405+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:36.002333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.002333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:59.288742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687080+0000","last_change":"2024-03-21T14:06:16.097848+0000","last_active":"2024-03-21T14:06:28.687080+0000","last_peered":"2024-03-21T14:06:28.687080+0000","last_clean":"2024-03-21T14:06:28.687080+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:06:28.687080+0000","last_undegraded":"2024-03-21T14:06:28.687080+0000","last_fullsized":"2024-03-21T14:06:28.687080+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:06:16.097602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.097602+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.783390+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688220+0000","last_change":"2024-03-21T14:04:14.595422+0000","last_active":"2024-03-21T14:06:28.688220+0000","last_peered":"2024-03-21T14:06:28.688220+0000","last_clean":"2024-03-21T14:06:28.688220+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:06:28.688220+0000","last_undegraded":"2024-03-21T14:06:28.688220+0000","last_fullsized":"2024-03-21T14:06:28.688220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:14.595340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:14.595340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671700+0000","last_change":"2024-03-21T14:05:22.170101+0000","last_active":"2024-03-21T14:06:28.671700+0000","last_peered":"2024-03-21T14:06:28.671700+0000","last_clean":"2024-03-21T14:06:28.671700+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:06:28.671700+0000","last_undegraded":"2024-03-21T14:06:28.671700+0000","last_fullsized":"2024-03-21T14:06:28.671700+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.169947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.169947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:46.686385+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670859+0000","last_change":"2024-03-21T14:05:31.213796+0000","last_active":"2024-03-21T14:06:28.670859+0000","last_peered":"2024-03-21T14:06:28.670859+0000","last_clean":"2024-03-21T14:06:28.670859+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:06:28.670859+0000","last_undegraded":"2024-03-21T14:06:28.670859+0000","last_fullsized":"2024-03-21T14:06:28.670859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.213713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.213713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.560680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.285898+0000","last_change":"2024-03-21T14:06:33.285958+0000","last_active":"2024-03-21T14:06:33.285898+0000","last_peered":"2024-03-21T14:06:33.285898+0000","last_clean":"2024-03-21T14:06:33.285898+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:06:33.285898+0000","last_undegraded":"2024-03-21T14:06:33.285898+0000","last_fullsized":"2024-03-21T14:06:33.285898+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:33.285816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.285816+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:02.251092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":125,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688273+0000","last_change":"2024-03-21T14:05:22.883709+0000","last_active":"2024-03-21T14:06:28.688273+0000","last_peered":"2024-03-21T14:06:28.688273+0000","last_clean":"2024-03-21T14:06:28.688273+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:06:28.688273+0000","last_undegraded":"2024-03-21T14:06:28.688273+0000","last_fullsized":"2024-03-21T14:06:28.688273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:22.883594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.883594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.619002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":191,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682392+0000","last_change":"2024-03-21T14:05:09.894660+0000","last_active":"2024-03-21T14:06:28.682392+0000","last_peered":"2024-03-21T14:06:28.682392+0000","last_clean":"2024-03-21T14:06:28.682392+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:06:28.682392+0000","last_undegraded":"2024-03-21T14:06:28.682392+0000","last_fullsized":"2024-03-21T14:06:28.682392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:09.894577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.894577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.107501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":153,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682199+0000","last_change":"2024-03-21T14:06:25.667933+0000","last_active":"2024-03-21T14:06:28.682199+0000","last_peered":"2024-03-21T14:06:28.682199+0000","last_clean":"2024-03-21T14:06:28.682199+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:06:28.682199+0000","last_undegraded":"2024-03-21T14:06:28.682199+0000","last_fullsized":"2024-03-21T14:06:28.682199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:06:25.667803+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.667803+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:51.379688+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.706546+0000","last_change":"2024-03-21T14:05:18.144792+0000","last_active":"2024-03-21T14:06:28.706546+0000","last_peered":"2024-03-21T14:06:28.706546+0000","last_clean":"2024-03-21T14:06:28.706546+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:06:28.706546+0000","last_undegraded":"2024-03-21T14:06:28.706546+0000","last_fullsized":"2024-03-21T14:06:28.706546+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:18.144307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.144307+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.897107+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.681470+0000","last_change":"2024-03-21T14:05:35.356518+0000","last_active":"2024-03-21T14:06:28.681470+0000","last_peered":"2024-03-21T14:06:28.681470+0000","last_clean":"2024-03-21T14:06:28.681470+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:06:28.681470+0000","last_undegraded":"2024-03-21T14:06:28.681470+0000","last_fullsized":"2024-03-21T14:06:28.681470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.356273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.356273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:00.889025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.707598+0000","last_change":"2024-03-21T14:05:16.155310+0000","last_active":"2024-03-21T14:06:28.707598+0000","last_peered":"2024-03-21T14:06:28.707598+0000","last_clean":"2024-03-21T14:06:28.707598+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:06:28.707598+0000","last_undegraded":"2024-03-21T14:06:28.707598+0000","last_fullsized":"2024-03-21T14:06:28.707598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.155028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.155028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.502395+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684474+0000","last_change":"2024-03-21T14:05:28.539817+0000","last_active":"2024-03-21T14:06:28.684474+0000","last_peered":"2024-03-21T14:06:28.684474+0000","last_clean":"2024-03-21T14:06:28.684474+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:06:28.684474+0000","last_undegraded":"2024-03-21T14:06:28.684474+0000","last_fullsized":"2024-03-21T14:06:28.684474+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.539674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.539674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.618118+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":151,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682248+0000","last_change":"2024-03-21T14:05:12.953038+0000","last_active":"2024-03-21T14:06:28.682248+0000","last_peered":"2024-03-21T14:06:28.682248+0000","last_clean":"2024-03-21T14:06:28.682248+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:06:28.682248+0000","last_undegraded":"2024-03-21T14:06:28.682248+0000","last_fullsized":"2024-03-21T14:06:28.682248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:12.952946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.952946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.647609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686396+0000","last_change":"2024-03-21T14:05:27.971209+0000","last_active":"2024-03-21T14:06:28.686396+0000","last_peered":"2024-03-21T14:06:28.686396+0000","last_clean":"2024-03-21T14:06:28.686396+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:06:28.686396+0000","last_undegraded":"2024-03-21T14:06:28.686396+0000","last_fullsized":"2024-03-21T14:06:28.686396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.971081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.971081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:52.888530+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686369+0000","last_change":"2024-03-21T14:05:35.015627+0000","last_active":"2024-03-21T14:06:28.686369+0000","last_peered":"2024-03-21T14:06:28.686369+0000","last_clean":"2024-03-21T14:06:28.686369+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:06:28.686369+0000","last_undegraded":"2024-03-21T14:06:28.686369+0000","last_fullsized":"2024-03-21T14:06:28.686369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.015517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.015517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.197434+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694214+0000","last_change":"2024-03-21T14:05:31.027870+0000","last_active":"2024-03-21T14:06:28.694214+0000","last_peered":"2024-03-21T14:06:28.694214+0000","last_clean":"2024-03-21T14:06:28.694214+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:06:28.694214+0000","last_undegraded":"2024-03-21T14:06:28.694214+0000","last_fullsized":"2024-03-21T14:06:28.694214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.027754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.027754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:33.511018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":138,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684755+0000","last_change":"2024-03-21T14:05:36.765565+0000","last_active":"2024-03-21T14:06:28.684755+0000","last_peered":"2024-03-21T14:06:28.684755+0000","last_clean":"2024-03-21T14:06:28.684755+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:06:28.684755+0000","last_undegraded":"2024-03-21T14:06:28.684755+0000","last_fullsized":"2024-03-21T14:06:28.684755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:05:36.765228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.765228+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.122075+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688943+0000","last_change":"2024-03-21T14:06:27.194477+0000","last_active":"2024-03-21T14:06:28.688943+0000","last_peered":"2024-03-21T14:06:28.688943+0000","last_clean":"2024-03-21T14:06:28.688943+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:06:28.688943+0000","last_undegraded":"2024-03-21T14:06:28.688943+0000","last_fullsized":"2024-03-21T14:06:28.688943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.194189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.194189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:57.100324+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687257+0000","last_change":"2024-03-21T14:05:07.887048+0000","last_active":"2024-03-21T14:06:28.687257+0000","last_peered":"2024-03-21T14:06:28.687257+0000","last_clean":"2024-03-21T14:06:28.687257+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:06:28.687257+0000","last_undegraded":"2024-03-21T14:06:28.687257+0000","last_fullsized":"2024-03-21T14:06:28.687257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:07.886924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:07.886924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":142,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683659+0000","last_change":"2024-03-21T14:05:17.362797+0000","last_active":"2024-03-21T14:06:28.683659+0000","last_peered":"2024-03-21T14:06:28.683659+0000","last_clean":"2024-03-21T14:06:28.683659+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:06:28.683659+0000","last_undegraded":"2024-03-21T14:06:28.683659+0000","last_fullsized":"2024-03-21T14:06:28.683659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:17.362665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:17.362665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":151,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685062+0000","last_change":"2024-03-21T14:05:38.851695+0000","last_active":"2024-03-21T14:06:28.685062+0000","last_peered":"2024-03-21T14:06:28.685062+0000","last_clean":"2024-03-21T14:06:28.685062+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:06:28.685062+0000","last_undegraded":"2024-03-21T14:06:28.685062+0000","last_fullsized":"2024-03-21T14:06:28.685062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.849289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.849289+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:59.036012+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.693694+0000","last_change":"2024-03-21T14:06:27.935782+0000","last_active":"2024-03-21T14:06:28.693694+0000","last_peered":"2024-03-21T14:06:28.693694+0000","last_clean":"2024-03-21T14:06:28.693694+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:06:28.693694+0000","last_undegraded":"2024-03-21T14:06:28.693694+0000","last_fullsized":"2024-03-21T14:06:28.693694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.935658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.935658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.122257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:29.146781+0000","last_change":"2024-03-21T14:06:29.146817+0000","last_active":"2024-03-21T14:06:29.146781+0000","last_peered":"2024-03-21T14:06:29.146781+0000","last_clean":"2024-03-21T14:06:29.146781+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:06:29.146781+0000","last_undegraded":"2024-03-21T14:06:29.146781+0000","last_fullsized":"2024-03-21T14:06:29.146781+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:06:29.146695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.757841+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686814+0000","last_change":"2024-03-21T14:06:28.067288+0000","last_active":"2024-03-21T14:06:28.686814+0000","last_peered":"2024-03-21T14:06:28.686814+0000","last_clean":"2024-03-21T14:06:28.686814+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:06:28.686814+0000","last_undegraded":"2024-03-21T14:06:28.686814+0000","last_fullsized":"2024-03-21T14:06:28.686814+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.067196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.067196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.480869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":156,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683783+0000","last_change":"2024-03-21T14:06:24.661616+0000","last_active":"2024-03-21T14:06:28.683783+0000","last_peered":"2024-03-21T14:06:28.683783+0000","last_clean":"2024-03-21T14:06:28.683783+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:06:28.683783+0000","last_undegraded":"2024-03-21T14:06:28.683783+0000","last_fullsized":"2024-03-21T14:06:28.683783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:24.661482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:24.661482+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.597062+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684689+0000","last_change":"2024-03-21T14:04:27.572389+0000","last_active":"2024-03-21T14:06:28.684689+0000","last_peered":"2024-03-21T14:06:28.684689+0000","last_clean":"2024-03-21T14:06:28.684689+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:06:28.684689+0000","last_undegraded":"2024-03-21T14:06:28.684689+0000","last_fullsized":"2024-03-21T14:06:28.684689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.572309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.572309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683565+0000","last_change":"2024-03-21T14:05:38.341843+0000","last_active":"2024-03-21T14:06:28.683565+0000","last_peered":"2024-03-21T14:06:28.683565+0000","last_clean":"2024-03-21T14:06:28.683565+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:06:28.683565+0000","last_undegraded":"2024-03-21T14:06:28.683565+0000","last_fullsized":"2024-03-21T14:06:28.683565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.341705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.341705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.933155+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671057+0000","last_change":"2024-03-21T14:05:36.278697+0000","last_active":"2024-03-21T14:06:28.671057+0000","last_peered":"2024-03-21T14:06:28.671057+0000","last_clean":"2024-03-21T14:06:28.671057+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:06:28.671057+0000","last_undegraded":"2024-03-21T14:06:28.671057+0000","last_fullsized":"2024-03-21T14:06:28.671057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.278592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.856682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":169,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671527+0000","last_change":"2024-03-21T14:05:38.294588+0000","last_active":"2024-03-21T14:06:28.671527+0000","last_peered":"2024-03-21T14:06:28.671527+0000","last_clean":"2024-03-21T14:06:28.671527+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:06:28.671527+0000","last_undegraded":"2024-03-21T14:06:28.671527+0000","last_fullsized":"2024-03-21T14:06:28.671527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.294441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.294441+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:01.276451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.705491+0000","last_change":"2024-03-21T14:05:38.046118+0000","last_active":"2024-03-21T14:06:28.705491+0000","last_peered":"2024-03-21T14:06:28.705491+0000","last_clean":"2024-03-21T14:06:28.705491+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:06:28.705491+0000","last_undegraded":"2024-03-21T14:06:28.705491+0000","last_fullsized":"2024-03-21T14:06:28.705491+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.045998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.045998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:52.031731+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":185,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684319+0000","last_change":"2024-03-21T14:05:13.668961+0000","last_active":"2024-03-21T14:06:28.684319+0000","last_peered":"2024-03-21T14:06:28.684319+0000","last_clean":"2024-03-21T14:06:28.684319+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:06:28.684319+0000","last_undegraded":"2024-03-21T14:06:28.684319+0000","last_fullsized":"2024-03-21T14:06:28.684319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:13.668830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.668830+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:32.074618+0000","last_change":"2024-03-21T14:06:32.074638+0000","last_active":"2024-03-21T14:06:32.074618+0000","last_peered":"2024-03-21T14:06:32.074618+0000","last_clean":"2024-03-21T14:06:32.074618+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:06:32.074618+0000","last_undegraded":"2024-03-21T14:06:32.074618+0000","last_fullsized":"2024-03-21T14:06:32.074618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:32.074556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:06:32.074556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:32.791813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":139,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.697122+0000","last_change":"2024-03-21T14:05:14.713009+0000","last_active":"2024-03-21T14:06:28.697122+0000","last_peered":"2024-03-21T14:06:28.697122+0000","last_clean":"2024-03-21T14:06:28.697122+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:06:28.697122+0000","last_undegraded":"2024-03-21T14:06:28.697122+0000","last_fullsized":"2024-03-21T14:06:28.697122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.712091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.712091+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:44.503021+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.093279+0000","last_change":"2024-03-21T14:06:36.093312+0000","last_active":"2024-03-21T14:06:36.093279+0000","last_peered":"2024-03-21T14:06:36.093279+0000","last_clean":"2024-03-21T14:06:36.093279+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:06:36.093279+0000","last_undegraded":"2024-03-21T14:06:36.093279+0000","last_fullsized":"2024-03-21T14:06:36.093279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:36.093213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.093213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:56.600582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":185,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.697324+0000","last_change":"2024-03-21T14:06:20.079291+0000","last_active":"2024-03-21T14:06:28.697324+0000","last_peered":"2024-03-21T14:06:28.697324+0000","last_clean":"2024-03-21T14:06:28.697324+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:06:28.697324+0000","last_undegraded":"2024-03-21T14:06:28.697324+0000","last_fullsized":"2024-03-21T14:06:28.697324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:20.079142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.079142+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.032560+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694546+0000","last_change":"2024-03-21T14:06:20.884332+0000","last_active":"2024-03-21T14:06:28.694546+0000","last_peered":"2024-03-21T14:06:28.694546+0000","last_clean":"2024-03-21T14:06:28.694546+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:06:28.694546+0000","last_undegraded":"2024-03-21T14:06:28.694546+0000","last_fullsized":"2024-03-21T14:06:28.694546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:20.884210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.884210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.916333+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684485+0000","last_change":"2024-03-21T14:06:20.431407+0000","last_active":"2024-03-21T14:06:28.684485+0000","last_peered":"2024-03-21T14:06:28.684485+0000","last_clean":"2024-03-21T14:06:28.684485+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:06:28.684485+0000","last_undegraded":"2024-03-21T14:06:28.684485+0000","last_fullsized":"2024-03-21T14:06:28.684485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:20.431279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.431279+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.061041+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687367+0000","last_change":"2024-03-21T14:05:29.391237+0000","last_active":"2024-03-21T14:06:28.687367+0000","last_peered":"2024-03-21T14:06:28.687367+0000","last_clean":"2024-03-21T14:06:28.687367+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:06:28.687367+0000","last_undegraded":"2024-03-21T14:06:28.687367+0000","last_fullsized":"2024-03-21T14:06:28.687367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.391091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.391091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.808745+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688620+0000","last_change":"2024-03-21T14:04:22.564083+0000","last_active":"2024-03-21T14:06:28.688620+0000","last_peered":"2024-03-21T14:06:28.688620+0000","last_clean":"2024-03-21T14:06:28.688620+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:06:28.688620+0000","last_undegraded":"2024-03-21T14:06:28.688620+0000","last_fullsized":"2024-03-21T14:06:28.688620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:22.563986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:22.563986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686241+0000","last_change":"2024-03-21T14:05:16.811057+0000","last_active":"2024-03-21T14:06:28.686241+0000","last_peered":"2024-03-21T14:06:28.686241+0000","last_clean":"2024-03-21T14:06:28.686241+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:06:28.686241+0000","last_undegraded":"2024-03-21T14:06:28.686241+0000","last_fullsized":"2024-03-21T14:06:28.686241+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.810928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.810928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.707364+0000","last_change":"2024-03-21T14:06:16.830628+0000","last_active":"2024-03-21T14:06:28.707364+0000","last_peered":"2024-03-21T14:06:28.707364+0000","last_clean":"2024-03-21T14:06:28.707364+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:06:28.707364+0000","last_undegraded":"2024-03-21T14:06:28.707364+0000","last_fullsized":"2024-03-21T14:06:28.707364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:16.830503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.830503+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:17.447722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686706+0000","last_change":"2024-03-21T14:05:15.857954+0000","last_active":"2024-03-21T14:06:28.686706+0000","last_peered":"2024-03-21T14:06:28.686706+0000","last_clean":"2024-03-21T14:06:28.686706+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:06:28.686706+0000","last_undegraded":"2024-03-21T14:06:28.686706+0000","last_fullsized":"2024-03-21T14:06:28.686706+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:15.857799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.857799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:42.408109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683687+0000","last_change":"2024-03-21T14:05:28.362497+0000","last_active":"2024-03-21T14:06:28.683687+0000","last_peered":"2024-03-21T14:06:28.683687+0000","last_clean":"2024-03-21T14:06:28.683687+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:06:28.683687+0000","last_undegraded":"2024-03-21T14:06:28.683687+0000","last_fullsized":"2024-03-21T14:06:28.683687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.362424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.362424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.774731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682166+0000","last_change":"2024-03-21T14:05:08.883023+0000","last_active":"2024-03-21T14:06:28.682166+0000","last_peered":"2024-03-21T14:06:28.682166+0000","last_clean":"2024-03-21T14:06:28.682166+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:06:28.682166+0000","last_undegraded":"2024-03-21T14:06:28.682166+0000","last_fullsized":"2024-03-21T14:06:28.682166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:08.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:08.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":133,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688515+0000","last_change":"2024-03-21T14:05:15.725635+0000","last_active":"2024-03-21T14:06:28.688515+0000","last_peered":"2024-03-21T14:06:28.688515+0000","last_clean":"2024-03-21T14:06:28.688515+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:06:28.688515+0000","last_undegraded":"2024-03-21T14:06:28.688515+0000","last_fullsized":"2024-03-21T14:06:28.688515+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:05:15.725275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.725275+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"119'9152","reported_seq":9300,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.255790+0000","last_change":"2024-03-21T14:06:25.814602+0000","last_active":"2024-03-21T14:06:36.255790+0000","last_peered":"2024-03-21T14:06:36.255790+0000","last_clean":"2024-03-21T14:06:36.255790+0000","last_became_active":"2024-03-21T14:04:28.368403+0000","last_became_peered":"2024-03-21T14:04:28.368403+0000","last_unstale":"2024-03-21T14:06:36.255790+0000","last_undegraded":"2024-03-21T14:06:36.255790+0000","last_fullsized":"2024-03-21T14:06:36.255790+0000","mapping_epoch":89,"log_start":"99'4100","ondisk_log_start":"99'4100","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8546","last_scrub_stamp":"2024-03-21T14:06:25.814421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.814421+0000","objects_scrubbed":6773,"log_size":5052,"log_dups_size":0,"ondisk_log_size":5052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.221261+0000","scrub_duration":2159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":399507456,"num_objects":6096,"num_object_clones":0,"num_object_copies":18288,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6096,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":487936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683471+0000","last_change":"2024-03-21T14:06:16.410140+0000","last_active":"2024-03-21T14:06:28.683471+0000","last_peered":"2024-03-21T14:06:28.683471+0000","last_clean":"2024-03-21T14:06:28.683471+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:06:28.683471+0000","last_undegraded":"2024-03-21T14:06:28.683471+0000","last_fullsized":"2024-03-21T14:06:28.683471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:16.409991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.409991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.498428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687120+0000","last_change":"2024-03-21T14:06:26.071562+0000","last_active":"2024-03-21T14:06:28.687120+0000","last_peered":"2024-03-21T14:06:28.687120+0000","last_clean":"2024-03-21T14:06:28.687120+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:06:28.687120+0000","last_undegraded":"2024-03-21T14:06:28.687120+0000","last_fullsized":"2024-03-21T14:06:28.687120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:06:26.071314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:26.071314+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.081185+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":157,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683006+0000","last_change":"2024-03-21T14:05:40.337551+0000","last_active":"2024-03-21T14:06:28.683006+0000","last_peered":"2024-03-21T14:06:28.683006+0000","last_clean":"2024-03-21T14:06:28.683006+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:06:28.683006+0000","last_undegraded":"2024-03-21T14:06:28.683006+0000","last_fullsized":"2024-03-21T14:06:28.683006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:40.337278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:40.337278+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.381891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":185,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:35.127079+0000","last_change":"2024-03-21T14:06:35.127130+0000","last_active":"2024-03-21T14:06:35.127079+0000","last_peered":"2024-03-21T14:06:35.127079+0000","last_clean":"2024-03-21T14:06:35.127079+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:06:35.127079+0000","last_undegraded":"2024-03-21T14:06:35.127079+0000","last_fullsized":"2024-03-21T14:06:35.127079+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:35.126990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:35.126990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:00.060225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"119'9108","reported_seq":9178,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.377726+0000","last_change":"2024-03-21T14:04:28.366763+0000","last_active":"2024-03-21T14:06:36.377726+0000","last_peered":"2024-03-21T14:06:36.377726+0000","last_clean":"2024-03-21T14:06:36.377726+0000","last_became_active":"2024-03-21T14:04:28.366208+0000","last_became_peered":"2024-03-21T14:04:28.366208+0000","last_unstale":"2024-03-21T14:06:36.377726+0000","last_undegraded":"2024-03-21T14:06:36.377726+0000","last_fullsized":"2024-03-21T14:06:36.377726+0000","mapping_epoch":89,"log_start":"101'4400","ondisk_log_start":"101'4400","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398065664,"num_objects":6074,"num_object_clones":0,"num_object_copies":18222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":485824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688366+0000","last_change":"2024-03-21T14:05:23.704653+0000","last_active":"2024-03-21T14:06:28.688366+0000","last_peered":"2024-03-21T14:06:28.688366+0000","last_clean":"2024-03-21T14:06:28.688366+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:06:28.688366+0000","last_undegraded":"2024-03-21T14:06:28.688366+0000","last_fullsized":"2024-03-21T14:06:28.688366+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.703365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.703365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:31.525757+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686388+0000","last_change":"2024-03-21T14:05:19.908957+0000","last_active":"2024-03-21T14:06:28.686388+0000","last_peered":"2024-03-21T14:06:28.686388+0000","last_clean":"2024-03-21T14:06:28.686388+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:06:28.686388+0000","last_undegraded":"2024-03-21T14:06:28.686388+0000","last_fullsized":"2024-03-21T14:06:28.686388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.908814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.908814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:40.177906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671354+0000","last_change":"2024-03-21T14:05:12.169866+0000","last_active":"2024-03-21T14:06:28.671354+0000","last_peered":"2024-03-21T14:06:28.671354+0000","last_clean":"2024-03-21T14:06:28.671354+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:06:28.671354+0000","last_undegraded":"2024-03-21T14:06:28.671354+0000","last_fullsized":"2024-03-21T14:06:28.671354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:05:12.169735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.169735+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.683299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.915330+0000","last_change":"2024-03-21T14:06:34.915366+0000","last_active":"2024-03-21T14:06:34.915330+0000","last_peered":"2024-03-21T14:06:34.915330+0000","last_clean":"2024-03-21T14:06:34.915330+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:06:34.915330+0000","last_undegraded":"2024-03-21T14:06:34.915330+0000","last_fullsized":"2024-03-21T14:06:34.915330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.915163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.915163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.070587+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"119'9107","reported_seq":9247,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.446673+0000","last_change":"2024-03-21T14:06:33.954281+0000","last_active":"2024-03-21T14:06:36.446673+0000","last_peered":"2024-03-21T14:06:36.446673+0000","last_clean":"2024-03-21T14:06:36.446673+0000","last_became_active":"2024-03-21T14:04:28.367553+0000","last_became_peered":"2024-03-21T14:04:28.367553+0000","last_unstale":"2024-03-21T14:06:36.446673+0000","last_undegraded":"2024-03-21T14:06:36.446673+0000","last_fullsized":"2024-03-21T14:06:36.446673+0000","mapping_epoch":89,"log_start":"99'3500","ondisk_log_start":"99'3500","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'8984","last_scrub_stamp":"2024-03-21T14:06:33.954062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.954062+0000","objects_scrubbed":6105,"log_size":5607,"log_dups_size":0,"ondisk_log_size":5607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.888906+0000","scrub_duration":1976,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388431872,"num_objects":5927,"num_object_clones":0,"num_object_copies":17781,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5927,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9107,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686477+0000","last_change":"2024-03-21T14:06:25.080381+0000","last_active":"2024-03-21T14:06:28.686477+0000","last_peered":"2024-03-21T14:06:28.686477+0000","last_clean":"2024-03-21T14:06:28.686477+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:06:28.686477+0000","last_undegraded":"2024-03-21T14:06:28.686477+0000","last_fullsized":"2024-03-21T14:06:28.686477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:25.080145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.080145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.576368+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.689879+0000","last_change":"2024-03-21T14:05:20.225369+0000","last_active":"2024-03-21T14:06:28.689879+0000","last_peered":"2024-03-21T14:06:28.689879+0000","last_clean":"2024-03-21T14:06:28.689879+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:06:28.689879+0000","last_undegraded":"2024-03-21T14:06:28.689879+0000","last_fullsized":"2024-03-21T14:06:28.689879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:20.225234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.225234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.070460+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687030+0000","last_change":"2024-03-21T14:06:19.057440+0000","last_active":"2024-03-21T14:06:28.687030+0000","last_peered":"2024-03-21T14:06:28.687030+0000","last_clean":"2024-03-21T14:06:28.687030+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:06:28.687030+0000","last_undegraded":"2024-03-21T14:06:28.687030+0000","last_fullsized":"2024-03-21T14:06:28.687030+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:19.057319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:19.057319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.818134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":163,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682050+0000","last_change":"2024-03-21T14:06:26.007667+0000","last_active":"2024-03-21T14:06:28.682050+0000","last_peered":"2024-03-21T14:06:28.682050+0000","last_clean":"2024-03-21T14:06:28.682050+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:06:28.682050+0000","last_undegraded":"2024-03-21T14:06:28.682050+0000","last_fullsized":"2024-03-21T14:06:28.682050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:06:26.007559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:26.007559+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.730733+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"119'9134","reported_seq":9284,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.410423+0000","last_change":"2024-03-21T14:06:21.462241+0000","last_active":"2024-03-21T14:06:36.410423+0000","last_peered":"2024-03-21T14:06:36.410423+0000","last_clean":"2024-03-21T14:06:36.410423+0000","last_became_active":"2024-03-21T14:04:28.360990+0000","last_became_peered":"2024-03-21T14:04:28.360990+0000","last_unstale":"2024-03-21T14:06:36.410423+0000","last_undegraded":"2024-03-21T14:06:36.410423+0000","last_fullsized":"2024-03-21T14:06:36.410423+0000","mapping_epoch":89,"log_start":"97'2300","ondisk_log_start":"97'2300","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8224","last_scrub_stamp":"2024-03-21T14:06:21.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.462077+0000","objects_scrubbed":6973,"log_size":6834,"log_dups_size":0,"ondisk_log_size":6834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.847162+0000","scrub_duration":2328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393478144,"num_objects":6004,"num_object_clones":0,"num_object_copies":18012,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6004,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9134,"num_write_kb":484416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.705271+0000","last_change":"2024-03-21T14:05:34.000829+0000","last_active":"2024-03-21T14:06:28.705271+0000","last_peered":"2024-03-21T14:06:28.705271+0000","last_clean":"2024-03-21T14:06:28.705271+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:06:28.705271+0000","last_undegraded":"2024-03-21T14:06:28.705271+0000","last_fullsized":"2024-03-21T14:06:28.705271+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.000689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.026990+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:30.047695+0000","last_change":"2024-03-21T14:06:30.047729+0000","last_active":"2024-03-21T14:06:30.047695+0000","last_peered":"2024-03-21T14:06:30.047695+0000","last_clean":"2024-03-21T14:06:30.047695+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:06:30.047695+0000","last_undegraded":"2024-03-21T14:06:30.047695+0000","last_fullsized":"2024-03-21T14:06:30.047695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:30.047613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:06:30.047613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.635296+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683790+0000","last_change":"2024-03-21T14:05:23.381853+0000","last_active":"2024-03-21T14:06:28.683790+0000","last_peered":"2024-03-21T14:06:28.683790+0000","last_clean":"2024-03-21T14:06:28.683790+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:06:28.683790+0000","last_undegraded":"2024-03-21T14:06:28.683790+0000","last_fullsized":"2024-03-21T14:06:28.683790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.381726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.381726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.801210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.700098+0000","last_change":"2024-03-21T14:06:15.846648+0000","last_active":"2024-03-21T14:06:28.700098+0000","last_peered":"2024-03-21T14:06:28.700098+0000","last_clean":"2024-03-21T14:06:28.700098+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:06:28.700098+0000","last_undegraded":"2024-03-21T14:06:28.700098+0000","last_fullsized":"2024-03-21T14:06:28.700098+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:15.846504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.846504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.520598+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"119'9095","reported_seq":9152,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-21T14:06:36.252605+0000","last_change":"2024-03-21T14:06:35.026655+0000","last_active":"2024-03-21T14:06:36.252605+0000","last_peered":"2024-03-21T14:06:36.252605+0000","last_clean":"2024-03-21T14:06:36.252605+0000","last_became_active":"2024-03-21T14:04:28.369020+0000","last_became_peered":"2024-03-21T14:04:28.369020+0000","last_unstale":"2024-03-21T14:06:36.252605+0000","last_undegraded":"2024-03-21T14:06:36.252605+0000","last_fullsized":"2024-03-21T14:06:36.252605+0000","mapping_epoch":89,"log_start":"97'2800","ondisk_log_start":"97'2800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":199,"log_size":6295,"log_dups_size":0,"ondisk_log_size":6295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389808128,"num_objects":5948,"num_object_clones":0,"num_object_copies":17844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8904,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.689484+0000","last_change":"2024-03-21T14:05:16.720441+0000","last_active":"2024-03-21T14:06:28.689484+0000","last_peered":"2024-03-21T14:06:28.689484+0000","last_clean":"2024-03-21T14:06:28.689484+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:06:28.689484+0000","last_undegraded":"2024-03-21T14:06:28.689484+0000","last_fullsized":"2024-03-21T14:06:28.689484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.720331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.720331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.562402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683228+0000","last_change":"2024-03-21T14:05:11.492743+0000","last_active":"2024-03-21T14:06:28.683228+0000","last_peered":"2024-03-21T14:06:28.683228+0000","last_clean":"2024-03-21T14:06:28.683228+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:06:28.683228+0000","last_undegraded":"2024-03-21T14:06:28.683228+0000","last_fullsized":"2024-03-21T14:06:28.683228+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:05:11.492609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.492609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670877+0000","last_change":"2024-03-21T14:05:18.141165+0000","last_active":"2024-03-21T14:06:28.670877+0000","last_peered":"2024-03-21T14:06:28.670877+0000","last_clean":"2024-03-21T14:06:28.670877+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:06:28.670877+0000","last_undegraded":"2024-03-21T14:06:28.670877+0000","last_fullsized":"2024-03-21T14:06:28.670877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:18.140955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.140955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.081222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":183,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683510+0000","last_change":"2024-03-21T14:05:12.482027+0000","last_active":"2024-03-21T14:06:28.683510+0000","last_peered":"2024-03-21T14:06:28.683510+0000","last_clean":"2024-03-21T14:06:28.683510+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:06:28.683510+0000","last_undegraded":"2024-03-21T14:06:28.683510+0000","last_fullsized":"2024-03-21T14:06:28.683510+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:05:12.481864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.481864+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.106831+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.705095+0000","last_change":"2024-03-21T14:06:14.840624+0000","last_active":"2024-03-21T14:06:28.705095+0000","last_peered":"2024-03-21T14:06:28.705095+0000","last_clean":"2024-03-21T14:06:28.705095+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:06:28.705095+0000","last_undegraded":"2024-03-21T14:06:28.705095+0000","last_fullsized":"2024-03-21T14:06:28.705095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.840509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.840509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.244776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"119'9255","reported_seq":9406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.442160+0000","last_change":"2024-03-21T14:06:21.025026+0000","last_active":"2024-03-21T14:06:36.442160+0000","last_peered":"2024-03-21T14:06:36.442160+0000","last_clean":"2024-03-21T14:06:36.442160+0000","last_became_active":"2024-03-21T14:04:28.367387+0000","last_became_peered":"2024-03-21T14:04:28.367387+0000","last_unstale":"2024-03-21T14:06:36.442160+0000","last_undegraded":"2024-03-21T14:06:36.442160+0000","last_fullsized":"2024-03-21T14:06:36.442160+0000","mapping_epoch":89,"log_start":"99'3600","ondisk_log_start":"99'3600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8368","last_scrub_stamp":"2024-03-21T14:06:21.024886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.024886+0000","objects_scrubbed":7201,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.840880+0000","scrub_duration":3116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":407044096,"num_objects":6211,"num_object_clones":0,"num_object_copies":18633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9255,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687132+0000","last_change":"2024-03-21T14:06:11.955196+0000","last_active":"2024-03-21T14:06:28.687132+0000","last_peered":"2024-03-21T14:06:28.687132+0000","last_clean":"2024-03-21T14:06:28.687132+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:06:28.687132+0000","last_undegraded":"2024-03-21T14:06:28.687132+0000","last_fullsized":"2024-03-21T14:06:28.687132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:11.955082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:11.955082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.410006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683358+0000","last_change":"2024-03-21T14:05:22.226038+0000","last_active":"2024-03-21T14:06:28.683358+0000","last_peered":"2024-03-21T14:06:28.683358+0000","last_clean":"2024-03-21T14:06:28.683358+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:06:28.683358+0000","last_undegraded":"2024-03-21T14:06:28.683358+0000","last_fullsized":"2024-03-21T14:06:28.683358+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.225815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.225815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:48.284109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687150+0000","last_change":"2024-03-21T14:06:12.171064+0000","last_active":"2024-03-21T14:06:28.687150+0000","last_peered":"2024-03-21T14:06:28.687150+0000","last_clean":"2024-03-21T14:06:28.687150+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:06:28.687150+0000","last_undegraded":"2024-03-21T14:06:28.687150+0000","last_fullsized":"2024-03-21T14:06:28.687150+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:12.170919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:12.170919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.011481+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":172,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:31.724257+0000","last_change":"2024-03-21T14:06:31.724307+0000","last_active":"2024-03-21T14:06:31.724257+0000","last_peered":"2024-03-21T14:06:31.724257+0000","last_clean":"2024-03-21T14:06:31.724257+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:06:31.724257+0000","last_undegraded":"2024-03-21T14:06:31.724257+0000","last_fullsized":"2024-03-21T14:06:31.724257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:06:31.724171+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:06:31.724171+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.792472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:32.979934+0000","last_change":"2024-03-21T14:06:32.979972+0000","last_active":"2024-03-21T14:06:32.979934+0000","last_peered":"2024-03-21T14:06:32.979934+0000","last_clean":"2024-03-21T14:06:32.979934+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:06:32.979934+0000","last_undegraded":"2024-03-21T14:06:32.979934+0000","last_fullsized":"2024-03-21T14:06:32.979934+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:32.979871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:32.979871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:52.805427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"119'9867","reported_seq":9962,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-21T14:06:36.471051+0000","last_change":"2024-03-21T14:06:29.986006+0000","last_active":"2024-03-21T14:06:36.471051+0000","last_peered":"2024-03-21T14:06:36.471051+0000","last_clean":"2024-03-21T14:06:36.471051+0000","last_became_active":"2024-03-21T14:04:28.363923+0000","last_became_peered":"2024-03-21T14:04:28.363923+0000","last_unstale":"2024-03-21T14:06:36.471051+0000","last_undegraded":"2024-03-21T14:06:36.471051+0000","last_fullsized":"2024-03-21T14:06:36.471051+0000","mapping_epoch":89,"log_start":"101'4300","ondisk_log_start":"101'4300","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":836,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397606912,"num_objects":6067,"num_object_clones":0,"num_object_copies":18201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9089,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684727+0000","last_change":"2024-03-21T14:06:15.375779+0000","last_active":"2024-03-21T14:06:28.684727+0000","last_peered":"2024-03-21T14:06:28.684727+0000","last_clean":"2024-03-21T14:06:28.684727+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:06:28.684727+0000","last_undegraded":"2024-03-21T14:06:28.684727+0000","last_fullsized":"2024-03-21T14:06:28.684727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:15.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.375667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.848357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":390,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686680+0000","last_change":"2024-03-21T14:05:31.964155+0000","last_active":"2024-03-21T14:06:28.686680+0000","last_peered":"2024-03-21T14:06:28.686680+0000","last_clean":"2024-03-21T14:06:28.686680+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:06:28.686680+0000","last_undegraded":"2024-03-21T14:06:28.686680+0000","last_fullsized":"2024-03-21T14:06:28.686680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-03-21T14:05:31.964052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.964052+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:51.267636+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687740+0000","last_change":"2024-03-21T14:06:28.186432+0000","last_active":"2024-03-21T14:06:28.687740+0000","last_peered":"2024-03-21T14:06:28.687740+0000","last_clean":"2024-03-21T14:06:28.687740+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:06:28.687740+0000","last_undegraded":"2024-03-21T14:06:28.687740+0000","last_fullsized":"2024-03-21T14:06:28.687740+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.186306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.186306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:32.719009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":203,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682417+0000","last_change":"2024-03-21T14:05:25.911867+0000","last_active":"2024-03-21T14:06:28.682417+0000","last_peered":"2024-03-21T14:06:28.682417+0000","last_clean":"2024-03-21T14:06:28.682417+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:06:28.682417+0000","last_undegraded":"2024-03-21T14:06:28.682417+0000","last_fullsized":"2024-03-21T14:06:28.682417+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:05:25.911729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:25.911729+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.703513+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688337+0000","last_change":"2024-03-21T14:04:13.817866+0000","last_active":"2024-03-21T14:06:28.688337+0000","last_peered":"2024-03-21T14:06:28.688337+0000","last_clean":"2024-03-21T14:06:28.688337+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:06:28.688337+0000","last_undegraded":"2024-03-21T14:06:28.688337+0000","last_fullsized":"2024-03-21T14:06:28.688337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:13.817745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:04:13.817745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"119'8961","reported_seq":9031,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.234046+0000","last_change":"2024-03-21T14:04:28.363760+0000","last_active":"2024-03-21T14:06:36.234046+0000","last_peered":"2024-03-21T14:06:36.234046+0000","last_clean":"2024-03-21T14:06:36.234046+0000","last_became_active":"2024-03-21T14:04:28.363404+0000","last_became_peered":"2024-03-21T14:04:28.363404+0000","last_unstale":"2024-03-21T14:06:36.234046+0000","last_undegraded":"2024-03-21T14:06:36.234046+0000","last_fullsized":"2024-03-21T14:06:36.234046+0000","mapping_epoch":89,"log_start":"99'3700","ondisk_log_start":"99'3700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5261,"log_dups_size":0,"ondisk_log_size":5261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391970816,"num_objects":5981,"num_object_clones":0,"num_object_copies":17943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8961,"num_write_kb":478144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682817+0000","last_change":"2024-03-21T14:05:36.379278+0000","last_active":"2024-03-21T14:06:28.682817+0000","last_peered":"2024-03-21T14:06:28.682817+0000","last_clean":"2024-03-21T14:06:28.682817+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:06:28.682817+0000","last_undegraded":"2024-03-21T14:06:28.682817+0000","last_fullsized":"2024-03-21T14:06:28.682817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.376091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.376091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.586944+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.961351+0000","last_change":"2024-03-21T14:06:28.961386+0000","last_active":"2024-03-21T14:06:28.961351+0000","last_peered":"2024-03-21T14:06:28.961351+0000","last_clean":"2024-03-21T14:06:28.961351+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:06:28.961351+0000","last_undegraded":"2024-03-21T14:06:28.961351+0000","last_fullsized":"2024-03-21T14:06:28.961351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.961272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.961272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.502960+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":385,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:35.758201+0000","last_change":"2024-03-21T14:06:13.952697+0000","last_active":"2024-03-21T14:06:35.758201+0000","last_peered":"2024-03-21T14:06:35.758201+0000","last_clean":"2024-03-21T14:06:35.758201+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:06:35.758201+0000","last_undegraded":"2024-03-21T14:06:35.758201+0000","last_fullsized":"2024-03-21T14:06:35.758201+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:06:13.952449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.952449+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.957684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":236,"num_read_kb":231,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":150,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.706373+0000","last_change":"2024-03-21T14:06:18.867126+0000","last_active":"2024-03-21T14:06:28.706373+0000","last_peered":"2024-03-21T14:06:28.706373+0000","last_clean":"2024-03-21T14:06:28.706373+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:06:28.706373+0000","last_undegraded":"2024-03-21T14:06:28.706373+0000","last_fullsized":"2024-03-21T14:06:28.706373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:18.866994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.866994+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.123054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685095+0000","last_change":"2024-03-21T14:06:27.461385+0000","last_active":"2024-03-21T14:06:28.685095+0000","last_peered":"2024-03-21T14:06:28.685095+0000","last_clean":"2024-03-21T14:06:28.685095+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:06:28.685095+0000","last_undegraded":"2024-03-21T14:06:28.685095+0000","last_fullsized":"2024-03-21T14:06:28.685095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.461298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.461298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.789908+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"119'9044","reported_seq":9191,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.237820+0000","last_change":"2024-03-21T14:06:25.324945+0000","last_active":"2024-03-21T14:06:36.237820+0000","last_peered":"2024-03-21T14:06:36.237820+0000","last_clean":"2024-03-21T14:06:36.237820+0000","last_became_active":"2024-03-21T14:04:28.367882+0000","last_became_peered":"2024-03-21T14:04:28.367882+0000","last_unstale":"2024-03-21T14:06:36.237820+0000","last_undegraded":"2024-03-21T14:06:36.237820+0000","last_fullsized":"2024-03-21T14:06:36.237820+0000","mapping_epoch":89,"log_start":"98'3200","ondisk_log_start":"98'3200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8391","last_scrub_stamp":"2024-03-21T14:06:25.321852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.321852+0000","objects_scrubbed":6732,"log_size":5844,"log_dups_size":0,"ondisk_log_size":5844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.120366+0000","scrub_duration":1849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395051008,"num_objects":6028,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682031+0000","last_change":"2024-03-21T14:05:34.327891+0000","last_active":"2024-03-21T14:06:28.682031+0000","last_peered":"2024-03-21T14:06:28.682031+0000","last_clean":"2024-03-21T14:06:28.682031+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:06:28.682031+0000","last_undegraded":"2024-03-21T14:06:28.682031+0000","last_fullsized":"2024-03-21T14:06:28.682031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.327068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.327068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.990999+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683042+0000","last_change":"2024-03-21T14:05:19.237932+0000","last_active":"2024-03-21T14:06:28.683042+0000","last_peered":"2024-03-21T14:06:28.683042+0000","last_clean":"2024-03-21T14:06:28.683042+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:06:28.683042+0000","last_undegraded":"2024-03-21T14:06:28.683042+0000","last_fullsized":"2024-03-21T14:06:28.683042+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.237452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.237452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:39.545828+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":168,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694337+0000","last_change":"2024-03-21T14:05:24.049566+0000","last_active":"2024-03-21T14:06:28.694337+0000","last_peered":"2024-03-21T14:06:28.694337+0000","last_clean":"2024-03-21T14:06:28.694337+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:06:28.694337+0000","last_undegraded":"2024-03-21T14:06:28.694337+0000","last_fullsized":"2024-03-21T14:06:28.694337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:05:24.049421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.049421+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.688262+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682143+0000","last_change":"2024-03-21T14:06:18.996323+0000","last_active":"2024-03-21T14:06:28.682143+0000","last_peered":"2024-03-21T14:06:28.682143+0000","last_clean":"2024-03-21T14:06:28.682143+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:06:28.682143+0000","last_undegraded":"2024-03-21T14:06:28.682143+0000","last_fullsized":"2024-03-21T14:06:28.682143+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.996252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.996252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.294909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"119'9042","reported_seq":9183,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.231872+0000","last_change":"2024-03-21T14:06:33.292754+0000","last_active":"2024-03-21T14:06:36.231872+0000","last_peered":"2024-03-21T14:06:36.231872+0000","last_clean":"2024-03-21T14:06:36.231872+0000","last_became_active":"2024-03-21T14:04:28.369002+0000","last_became_peered":"2024-03-21T14:04:28.369002+0000","last_unstale":"2024-03-21T14:06:36.231872+0000","last_undegraded":"2024-03-21T14:06:36.231872+0000","last_fullsized":"2024-03-21T14:06:36.231872+0000","mapping_epoch":89,"log_start":"100'4200","ondisk_log_start":"100'4200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'8884","last_scrub_stamp":"2024-03-21T14:06:33.292611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.292611+0000","objects_scrubbed":6207,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:58.862461+0000","scrub_duration":1119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394657792,"num_objects":6022,"num_object_clones":0,"num_object_copies":18066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6022,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9042,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.700211+0000","last_change":"2024-03-21T14:05:11.153864+0000","last_active":"2024-03-21T14:06:28.700211+0000","last_peered":"2024-03-21T14:06:28.700211+0000","last_clean":"2024-03-21T14:06:28.700211+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:06:28.700211+0000","last_undegraded":"2024-03-21T14:06:28.700211+0000","last_fullsized":"2024-03-21T14:06:28.700211+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:11.153740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.153740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:29.073768+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684363+0000","last_change":"2024-03-21T14:06:18.451847+0000","last_active":"2024-03-21T14:06:28.684363+0000","last_peered":"2024-03-21T14:06:28.684363+0000","last_clean":"2024-03-21T14:06:28.684363+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:06:28.684363+0000","last_undegraded":"2024-03-21T14:06:28.684363+0000","last_fullsized":"2024-03-21T14:06:28.684363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.451772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.451772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.610702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683646+0000","last_change":"2024-03-21T14:06:27.736989+0000","last_active":"2024-03-21T14:06:28.683646+0000","last_peered":"2024-03-21T14:06:28.683646+0000","last_clean":"2024-03-21T14:06:28.683646+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:06:28.683646+0000","last_undegraded":"2024-03-21T14:06:28.683646+0000","last_fullsized":"2024-03-21T14:06:28.683646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:06:27.736695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.736695+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:30.734373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":185,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.700029+0000","last_change":"2024-03-21T14:06:13.786619+0000","last_active":"2024-03-21T14:06:28.700029+0000","last_peered":"2024-03-21T14:06:28.700029+0000","last_clean":"2024-03-21T14:06:28.700029+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:06:28.700029+0000","last_undegraded":"2024-03-21T14:06:28.700029+0000","last_fullsized":"2024-03-21T14:06:28.700029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:13.786485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.786485+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.654794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"119'9119","reported_seq":9190,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.430202+0000","last_change":"2024-03-21T14:04:28.366919+0000","last_active":"2024-03-21T14:06:36.430202+0000","last_peered":"2024-03-21T14:06:36.430202+0000","last_clean":"2024-03-21T14:06:36.430202+0000","last_became_active":"2024-03-21T14:04:28.366634+0000","last_became_peered":"2024-03-21T14:04:28.366634+0000","last_unstale":"2024-03-21T14:06:36.430202+0000","last_undegraded":"2024-03-21T14:06:36.430202+0000","last_fullsized":"2024-03-21T14:06:36.430202+0000","mapping_epoch":89,"log_start":"101'4500","ondisk_log_start":"101'4500","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397213696,"num_objects":6061,"num_object_clones":0,"num_object_copies":18183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9119,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.155926+0000","last_change":"2024-03-21T14:06:34.155961+0000","last_active":"2024-03-21T14:06:34.155926+0000","last_peered":"2024-03-21T14:06:34.155926+0000","last_clean":"2024-03-21T14:06:34.155926+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:06:34.155926+0000","last_undegraded":"2024-03-21T14:06:34.155926+0000","last_fullsized":"2024-03-21T14:06:34.155926+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.155840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.155840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:49.184912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.706276+0000","last_change":"2024-03-21T14:05:20.137081+0000","last_active":"2024-03-21T14:06:28.706276+0000","last_peered":"2024-03-21T14:06:28.706276+0000","last_clean":"2024-03-21T14:06:28.706276+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:06:28.706276+0000","last_undegraded":"2024-03-21T14:06:28.706276+0000","last_fullsized":"2024-03-21T14:06:28.706276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:20.136955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.136955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.349906+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":153,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671090+0000","last_change":"2024-03-21T14:06:18.141733+0000","last_active":"2024-03-21T14:06:28.671090+0000","last_peered":"2024-03-21T14:06:28.671090+0000","last_clean":"2024-03-21T14:06:28.671090+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:06:28.671090+0000","last_undegraded":"2024-03-21T14:06:28.671090+0000","last_fullsized":"2024-03-21T14:06:28.671090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:06:18.141619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.141619+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:29.717483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":185,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.344400+0000","last_change":"2024-03-21T14:06:36.344441+0000","last_active":"2024-03-21T14:06:36.344400+0000","last_peered":"2024-03-21T14:06:36.344400+0000","last_clean":"2024-03-21T14:06:36.344400+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:06:36.344400+0000","last_undegraded":"2024-03-21T14:06:36.344400+0000","last_fullsized":"2024-03-21T14:06:36.344400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:36.344322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.344322+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:49.912913+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"119'9131","reported_seq":9202,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.428035+0000","last_change":"2024-03-21T14:04:28.364630+0000","last_active":"2024-03-21T14:06:36.428035+0000","last_peered":"2024-03-21T14:06:36.428035+0000","last_clean":"2024-03-21T14:06:36.428035+0000","last_became_active":"2024-03-21T14:04:28.364379+0000","last_became_peered":"2024-03-21T14:04:28.364379+0000","last_unstale":"2024-03-21T14:06:36.428035+0000","last_undegraded":"2024-03-21T14:06:36.428035+0000","last_fullsized":"2024-03-21T14:06:36.428035+0000","mapping_epoch":89,"log_start":"101'4500","ondisk_log_start":"101'4500","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398262272,"num_objects":6077,"num_object_clones":0,"num_object_copies":18231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6077,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9131,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670716+0000","last_change":"2024-03-21T14:06:17.164352+0000","last_active":"2024-03-21T14:06:28.670716+0000","last_peered":"2024-03-21T14:06:28.670716+0000","last_clean":"2024-03-21T14:06:28.670716+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:06:28.670716+0000","last_undegraded":"2024-03-21T14:06:28.670716+0000","last_fullsized":"2024-03-21T14:06:28.670716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.164232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.164232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:43.626287+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688356+0000","last_change":"2024-03-21T14:05:29.550541+0000","last_active":"2024-03-21T14:06:28.688356+0000","last_peered":"2024-03-21T14:06:28.688356+0000","last_clean":"2024-03-21T14:06:28.688356+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:06:28.688356+0000","last_undegraded":"2024-03-21T14:06:28.688356+0000","last_fullsized":"2024-03-21T14:06:28.688356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.548379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.548379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:55.684852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685020+0000","last_change":"2024-03-21T14:06:16.045314+0000","last_active":"2024-03-21T14:06:28.685020+0000","last_peered":"2024-03-21T14:06:28.685020+0000","last_clean":"2024-03-21T14:06:28.685020+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:06:28.685020+0000","last_undegraded":"2024-03-21T14:06:28.685020+0000","last_fullsized":"2024-03-21T14:06:28.685020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:06:16.045170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.045170+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:28.072141+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670824+0000","last_change":"2024-03-21T14:05:27.174995+0000","last_active":"2024-03-21T14:06:28.670824+0000","last_peered":"2024-03-21T14:06:28.670824+0000","last_clean":"2024-03-21T14:06:28.670824+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:06:28.670824+0000","last_undegraded":"2024-03-21T14:06:28.670824+0000","last_fullsized":"2024-03-21T14:06:28.670824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.174851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.174851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.192729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"119'8870","reported_seq":8940,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.236754+0000","last_change":"2024-03-21T14:04:28.369284+0000","last_active":"2024-03-21T14:06:36.236754+0000","last_peered":"2024-03-21T14:06:36.236754+0000","last_clean":"2024-03-21T14:06:36.236754+0000","last_became_active":"2024-03-21T14:04:28.368956+0000","last_became_peered":"2024-03-21T14:04:28.368956+0000","last_unstale":"2024-03-21T14:06:36.236754+0000","last_undegraded":"2024-03-21T14:06:36.236754+0000","last_fullsized":"2024-03-21T14:06:36.236754+0000","mapping_epoch":89,"log_start":"97'2600","ondisk_log_start":"97'2600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":6270,"log_dups_size":0,"ondisk_log_size":6270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393084928,"num_objects":5998,"num_object_clones":0,"num_object_copies":17994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5998,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8870,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684105+0000","last_change":"2024-03-21T14:05:24.417631+0000","last_active":"2024-03-21T14:06:28.684105+0000","last_peered":"2024-03-21T14:06:28.684105+0000","last_clean":"2024-03-21T14:06:28.684105+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:06:28.684105+0000","last_undegraded":"2024-03-21T14:06:28.684105+0000","last_fullsized":"2024-03-21T14:06:28.684105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.417495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.417495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:33.538419+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":185,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671461+0000","last_change":"2024-03-21T14:05:37.280589+0000","last_active":"2024-03-21T14:06:28.671461+0000","last_peered":"2024-03-21T14:06:28.671461+0000","last_clean":"2024-03-21T14:06:28.671461+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:06:28.671461+0000","last_undegraded":"2024-03-21T14:06:28.671461+0000","last_fullsized":"2024-03-21T14:06:28.671461+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:37.280472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.280472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:55.817892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.326248+0000","last_change":"2024-03-21T14:06:34.326281+0000","last_active":"2024-03-21T14:06:34.326248+0000","last_peered":"2024-03-21T14:06:34.326248+0000","last_clean":"2024-03-21T14:06:34.326248+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:06:34.326248+0000","last_undegraded":"2024-03-21T14:06:34.326248+0000","last_fullsized":"2024-03-21T14:06:34.326248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.326168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.326168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.255180+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":157,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682539+0000","last_change":"2024-03-21T14:05:29.265158+0000","last_active":"2024-03-21T14:06:28.682539+0000","last_peered":"2024-03-21T14:06:28.682539+0000","last_clean":"2024-03-21T14:06:28.682539+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:06:28.682539+0000","last_undegraded":"2024-03-21T14:06:28.682539+0000","last_fullsized":"2024-03-21T14:06:28.682539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:29.265009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.265009+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.852216+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"119'8930","reported_seq":9079,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.174846+0000","last_change":"2024-03-21T14:06:21.944486+0000","last_active":"2024-03-21T14:06:36.174846+0000","last_peered":"2024-03-21T14:06:36.174846+0000","last_clean":"2024-03-21T14:06:36.174846+0000","last_became_active":"2024-03-21T14:04:28.371337+0000","last_became_peered":"2024-03-21T14:04:28.371337+0000","last_unstale":"2024-03-21T14:06:36.174846+0000","last_undegraded":"2024-03-21T14:06:36.174846+0000","last_fullsized":"2024-03-21T14:06:36.174846+0000","mapping_epoch":89,"log_start":"97'2600","ondisk_log_start":"97'2600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8126","last_scrub_stamp":"2024-03-21T14:06:21.944346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.944346+0000","objects_scrubbed":6902,"log_size":6330,"log_dups_size":0,"ondisk_log_size":6330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.534040+0000","scrub_duration":1949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":394985496,"num_objects":6028,"num_object_clones":0,"num_object_copies":18084,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6028,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8930,"num_write_kb":478593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.696750+0000","last_change":"2024-03-21T14:05:10.657107+0000","last_active":"2024-03-21T14:06:28.696750+0000","last_peered":"2024-03-21T14:06:28.696750+0000","last_clean":"2024-03-21T14:06:28.696750+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:06:28.696750+0000","last_undegraded":"2024-03-21T14:06:28.696750+0000","last_fullsized":"2024-03-21T14:06:28.696750+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:10.656780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:10.656780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.396348+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.396207+0000","last_change":"2024-03-21T14:06:34.396244+0000","last_active":"2024-03-21T14:06:34.396207+0000","last_peered":"2024-03-21T14:06:34.396207+0000","last_clean":"2024-03-21T14:06:34.396207+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:06:34.396207+0000","last_undegraded":"2024-03-21T14:06:34.396207+0000","last_fullsized":"2024-03-21T14:06:34.396207+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.396131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.396131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:00.169873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670921+0000","last_change":"2024-03-21T14:05:33.267089+0000","last_active":"2024-03-21T14:06:28.670921+0000","last_peered":"2024-03-21T14:06:28.670921+0000","last_clean":"2024-03-21T14:06:28.670921+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:06:28.670921+0000","last_undegraded":"2024-03-21T14:06:28.670921+0000","last_fullsized":"2024-03-21T14:06:28.670921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:33.266985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.266985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:42.374468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":151,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671061+0000","last_change":"2024-03-21T14:06:21.108018+0000","last_active":"2024-03-21T14:06:28.671061+0000","last_peered":"2024-03-21T14:06:28.671061+0000","last_clean":"2024-03-21T14:06:28.671061+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:06:28.671061+0000","last_undegraded":"2024-03-21T14:06:28.671061+0000","last_fullsized":"2024-03-21T14:06:28.671061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:06:21.107886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.107886+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.591227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"119'9152","reported_seq":9283,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.335805+0000","last_change":"2024-03-21T14:05:36.765363+0000","last_active":"2024-03-21T14:06:36.335805+0000","last_peered":"2024-03-21T14:06:36.335805+0000","last_clean":"2024-03-21T14:06:36.335805+0000","last_became_active":"2024-03-21T14:04:28.365174+0000","last_became_peered":"2024-03-21T14:04:28.365174+0000","last_unstale":"2024-03-21T14:06:36.335805+0000","last_undegraded":"2024-03-21T14:06:36.335805+0000","last_fullsized":"2024-03-21T14:06:36.335805+0000","mapping_epoch":89,"log_start":"97'2300","ondisk_log_start":"97'2300","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'5496","last_scrub_stamp":"2024-03-21T14:05:36.764945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.764945+0000","objects_scrubbed":5416,"log_size":6852,"log_dups_size":0,"ondisk_log_size":6852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:44.153675+0000","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397148160,"num_objects":6060,"num_object_clones":0,"num_object_copies":18180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":125,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687207+0000","last_change":"2024-03-21T14:05:33.294251+0000","last_active":"2024-03-21T14:06:28.687207+0000","last_peered":"2024-03-21T14:06:28.687207+0000","last_clean":"2024-03-21T14:06:28.687207+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:06:28.687207+0000","last_undegraded":"2024-03-21T14:06:28.687207+0000","last_fullsized":"2024-03-21T14:06:28.687207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:33.294119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.294119+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.044466+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683130+0000","last_change":"2024-03-21T14:05:30.277978+0000","last_active":"2024-03-21T14:06:28.683130+0000","last_peered":"2024-03-21T14:06:28.683130+0000","last_clean":"2024-03-21T14:06:28.683130+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:06:28.683130+0000","last_undegraded":"2024-03-21T14:06:28.683130+0000","last_fullsized":"2024-03-21T14:06:28.683130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:30.277645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:30.277645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.448928+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:31.951691+0000","last_change":"2024-03-21T14:06:31.951712+0000","last_active":"2024-03-21T14:06:31.951691+0000","last_peered":"2024-03-21T14:06:31.951691+0000","last_clean":"2024-03-21T14:06:31.951691+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:06:31.951691+0000","last_undegraded":"2024-03-21T14:06:31.951691+0000","last_fullsized":"2024-03-21T14:06:31.951691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:31.951638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:31.951638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:34.454617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":160,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671341+0000","last_change":"2024-03-21T14:06:16.191465+0000","last_active":"2024-03-21T14:06:28.671341+0000","last_peered":"2024-03-21T14:06:28.671341+0000","last_clean":"2024-03-21T14:06:28.671341+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:06:28.671341+0000","last_undegraded":"2024-03-21T14:06:28.671341+0000","last_fullsized":"2024-03-21T14:06:28.671341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:06:16.191249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.191249+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.656713+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"119'9259","reported_seq":9364,"reported_epoch":119,"state":"active+clean+scrubbing","last_fresh":"2024-03-21T14:06:36.466189+0000","last_change":"2024-03-21T14:06:33.994650+0000","last_active":"2024-03-21T14:06:36.466189+0000","last_peered":"2024-03-21T14:06:36.466189+0000","last_clean":"2024-03-21T14:06:36.466189+0000","last_became_active":"2024-03-21T14:04:28.364570+0000","last_became_peered":"2024-03-21T14:04:28.364570+0000","last_unstale":"2024-03-21T14:06:36.466189+0000","last_undegraded":"2024-03-21T14:06:36.466189+0000","last_fullsized":"2024-03-21T14:06:36.466189+0000","mapping_epoch":89,"log_start":"99'3700","ondisk_log_start":"99'3700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":3349,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":405733376,"num_objects":6191,"num_object_clones":0,"num_object_copies":18573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9259,"num_write_kb":494400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671326+0000","last_change":"2024-03-21T14:05:11.171257+0000","last_active":"2024-03-21T14:06:28.671326+0000","last_peered":"2024-03-21T14:06:28.671326+0000","last_clean":"2024-03-21T14:06:28.671326+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:06:28.671326+0000","last_undegraded":"2024-03-21T14:06:28.671326+0000","last_fullsized":"2024-03-21T14:06:28.671326+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:11.171149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.171149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":186,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687144+0000","last_change":"2024-03-21T14:05:32.346872+0000","last_active":"2024-03-21T14:06:28.687144+0000","last_peered":"2024-03-21T14:06:28.687144+0000","last_clean":"2024-03-21T14:06:28.687144+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:06:28.687144+0000","last_undegraded":"2024-03-21T14:06:28.687144+0000","last_fullsized":"2024-03-21T14:06:28.687144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.346553+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:43.290171+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688026+0000","last_change":"2024-03-21T14:04:10.596303+0000","last_active":"2024-03-21T14:06:28.688026+0000","last_peered":"2024-03-21T14:06:28.688026+0000","last_clean":"2024-03-21T14:06:28.688026+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:06:28.688026+0000","last_undegraded":"2024-03-21T14:06:28.688026+0000","last_fullsized":"2024-03-21T14:06:28.688026+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:10.596224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:04:10.596224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":139,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671601+0000","last_change":"2024-03-21T14:05:14.162485+0000","last_active":"2024-03-21T14:06:28.671601+0000","last_peered":"2024-03-21T14:06:28.671601+0000","last_clean":"2024-03-21T14:06:28.671601+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:06:28.671601+0000","last_undegraded":"2024-03-21T14:06:28.671601+0000","last_fullsized":"2024-03-21T14:06:28.671601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.162391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.162391+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.569569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6342049816,"num_objects":96773,"num_object_clones":0,"num_object_copies":290319,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96773,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":145257,"num_write_kb":7744897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19025899520,"data_stored":19025887304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89926,"ondisk_log_size":89926,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":236,"num_read_kb":231,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103221,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2011396,"kb_used_data":1939964,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91733756,"statfs":{"total":95995035648,"available":93935366144,"internally_reserved":0,"allocated":1986523136,"data_stored":1986160037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430266,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2005616,"kb_used_data":1930024,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91739536,"statfs":{"total":95995035648,"available":93941284864,"internally_reserved":0,"allocated":1976344576,"data_stored":1976000018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2433948,"kb_used_data":2336148,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":91311204,"statfs":{"total":95995035648,"available":93502672896,"internally_reserved":0,"allocated":2392215552,"data_stored":2391827948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3218632,"kb_used_data":3106944,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90526520,"statfs":{"total":95995035648,"available":92699156480,"internally_reserved":0,"allocated":3181510656,"data_stored":3181142316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2422384,"kb_used_data":2328744,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":91322768,"statfs":{"total":95995035648,"available":93514514432,"internally_reserved":0,"allocated":2384633856,"data_stored":2384289078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182473,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3600020,"kb_used_data":3488204,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":90145132,"statfs":{"total":95995035648,"available":92308615168,"internally_reserved":0,"allocated":3571920896,"data_stored":3571540246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":114358727},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018949,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2007884,"kb_used_data":1940612,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91737268,"statfs":{"total":95995035648,"available":93938962432,"internally_reserved":0,"allocated":1987186688,"data_stored":1986813944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691904,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1629504,"kb_used_data":1562232,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92115648,"statfs":{"total":95995035648,"available":94326423552,"internally_reserved":0,"allocated":1599725568,"data_stored":1599365653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3170570240,"data_stored":3170566168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2382827520,"data_stored":2382823448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2375221248,"data_stored":2375221248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3567386624,"data_stored":3567386624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1984364544,"data_stored":1984364544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1590624256,"data_stored":1590624256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1971916800,"data_stored":1971912728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1982988288,"data_stored":1982988288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:06:39.013 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T14:06:39.013 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:06:39.014 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-21T14:06:39.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:06:39.014 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:38 smithi118 ceph-mon[33862]: pgmap v146: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 896 op/s 2024-03-21T14:06:39.014 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:38 smithi118 ceph-mon[33862]: 7.a scrub starts 2024-03-21T14:06:39.015 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:38 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:06:39.015 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:38 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:06:39.015 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:38 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:06:39.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:39.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:39.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:39.201 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:39.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:39.333 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:39.362 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:39.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:39.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:39.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:39.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:39.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:39.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:39.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:39.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:39.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[29576]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:06:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[29576]: 7.a scrub ok 2024-03-21T14:06:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:06:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:06:40.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[40510]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:06:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[40510]: 7.a scrub ok 2024-03-21T14:06:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:06:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:39 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:06:40.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:40.041 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:40.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:40.171 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:40.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:39 smithi118 ceph-mon[33862]: from='client.24800 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:40.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:39 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:06:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:39 smithi118 ceph-mon[33862]: 7.a scrub ok 2024-03-21T14:06:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:39 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:06:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:39 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:06:40.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:40.300 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:40.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:40.530 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:40.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:40.658 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:40.686 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:40.769 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:40.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:40.842 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:40.896 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:40.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:40.972 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:40.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:40.994 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:40 smithi003 ceph-mon[29576]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 843 op/s 2024-03-21T14:06:40.995 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:40 smithi003 ceph-mon[40510]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 843 op/s 2024-03-21T14:06:41.008 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:40 smithi118 ceph-mon[33862]: pgmap v147: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 962 B/s rd, 0 B/s wr, 843 op/s 2024-03-21T14:06:41.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:41.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:41.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:41.332 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:06:41.332 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:06:41.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:41.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:41.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:41.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:41.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:41.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:41.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:41.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:41.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:41.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:41.927 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:41.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:06:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:06:42.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:06:42.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:06:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:06:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:06:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:06:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:41 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:06:42.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:42.054 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:42.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:42.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:41 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:06:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:41 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:06:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:41 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:06:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:41 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:06:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:42.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:42.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:06:42.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:42.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:42.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:42.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:42.541 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:06:42.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:42.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:42.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:42.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:42.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:42.817 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:06:42.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:42.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:42.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:42.925 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:42 smithi118 ceph-mon[33862]: pgmap v148: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 846 op/s 2024-03-21T14:06:42.925 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:42 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2036203037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:06:42.925 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:06:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:42 smithi003 ceph-mon[29576]: pgmap v148: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 846 op/s 2024-03-21T14:06:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:42 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2036203037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:06:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:06:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:42 smithi003 ceph-mon[40510]: pgmap v148: 148 pgs: 1 active+clean+scrubbing, 145 active+clean, 2 active+clean+scrubbing+deep; 5.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 846 op/s 2024-03-21T14:06:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:42 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2036203037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:06:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:06:43.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:43.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:43.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:43.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:43.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:43.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:43.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:43.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:43.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:43.428 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:43.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:43.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:43.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:43.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:43.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:43.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:43.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:43.959 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:43.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: 7.3 scrub starts 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: 7.f scrub ok 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:06:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[29576]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: 7.3 scrub starts 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: 7.f scrub ok 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:06:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:43 smithi003 ceph-mon[40510]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T14:06:44.028 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: 7.3 scrub starts 2024-03-21T14:06:44.028 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: 7.f scrub ok 2024-03-21T14:06:44.028 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:06:44.028 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:06:44.028 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:44.028 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:06:44.029 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:43 smithi118 ceph-mon[33862]: osdmap e120: 8 total, 8 up, 8 in 2024-03-21T14:06:44.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:44.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:44.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:44.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:44.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:44.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:44.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:44.487 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:44.517 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:06:44.517 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:06:44.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:44.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:44.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:44.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:44.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:44.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:44.921 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:44.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:44.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:44 smithi003 ceph-mon[29576]: pgmap v150: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 683 op/s 2024-03-21T14:06:44.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:44 smithi003 ceph-mon[40510]: pgmap v150: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 683 op/s 2024-03-21T14:06:45.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:45.051 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:45.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:45.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:45.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:44 smithi118 ceph-mon[33862]: pgmap v150: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 683 op/s 2024-03-21T14:06:45.217 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:06:45.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:45.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.329 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:45.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:45.459 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:45.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:45.562 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:06:45.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:06:45.598 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:45.602 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:06:45.644 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:06:45.687 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:06:45.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:45.727 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:06:45.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:45.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:06:45.779 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:06:45.820 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:06:45.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:45.860 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:45.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:45.875 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:06:45.884 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:06:45.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:45.982 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2673199373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:45.982 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:06:45.982 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:46.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:46.076 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:06:46.077 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2673199373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:46.077 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:06:46.077 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:46.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2673199373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:46.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:06:46.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:06:46.083 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:06:46.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:46.147 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:46.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:46.273 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:46.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:46.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:46.404 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:46.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:46.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:46.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:46.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:46.679 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:46.740 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:06:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:06:46.749 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:06:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:06:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:06:46.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.810 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:46.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:46.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:46.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:46.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:47.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:46 smithi003 ceph-mon[29576]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 684 op/s 2024-03-21T14:06:47.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:06:47.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:46 smithi003 ceph-mon[40510]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 684 op/s 2024-03-21T14:06:47.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:06:47.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:47.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.078 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:47.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:47.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:47.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:46 smithi118 ceph-mon[33862]: pgmap v151: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 684 op/s 2024-03-21T14:06:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:06:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:47.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:47.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:47.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:47.471 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:47.500 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:47.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:47.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:47.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:47.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:47.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:47.819 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2024-03-21T14:06:45.313413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6142564800,"num_objects":93918,"num_object_clones":0,"num_object_copies":281754,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93918,"num_whiteouts":0,"num_read":1176,"num_read_kb":1022,"num_write":148925,"num_write_kb":7747358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90734,"ondisk_log_size":90734,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18719848,"kb_used_data":18046824,"kb_used_omap":12,"kb_used_meta":672755,"kb_avail":731241368,"statfs":{"total":767960285184,"available":748791160832,"internally_reserved":0,"allocated":18479947776,"data_stored":18477026088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":688901680},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-447807488,"num_objects":-6833,"num_object_clones":0,"num_object_copies":-20499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6833,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6833,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004214"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670703+0000","last_change":"2024-03-21T14:05:24.217692+0000","last_active":"2024-03-21T14:06:28.670703+0000","last_peered":"2024-03-21T14:06:28.670703+0000","last_clean":"2024-03-21T14:06:28.670703+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:06:28.670703+0000","last_undegraded":"2024-03-21T14:06:28.670703+0000","last_fullsized":"2024-03-21T14:06:28.670703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.217564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.217564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:30.718694+0000","last_change":"2024-03-21T14:06:30.718732+0000","last_active":"2024-03-21T14:06:30.718694+0000","last_peered":"2024-03-21T14:06:30.718694+0000","last_clean":"2024-03-21T14:06:30.718694+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:06:30.718694+0000","last_undegraded":"2024-03-21T14:06:30.718694+0000","last_fullsized":"2024-03-21T14:06:30.718694+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:06:30.718605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:47.380214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694003+0000","last_change":"2024-03-21T14:05:37.038645+0000","last_active":"2024-03-21T14:06:28.694003+0000","last_peered":"2024-03-21T14:06:28.694003+0000","last_clean":"2024-03-21T14:06:28.694003+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:06:28.694003+0000","last_undegraded":"2024-03-21T14:06:28.694003+0000","last_fullsized":"2024-03-21T14:06:28.694003+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:37.038531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.038531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.964335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:38.152471+0000","last_change":"2024-03-21T14:06:38.152523+0000","last_active":"2024-03-21T14:06:38.152471+0000","last_peered":"2024-03-21T14:06:38.152471+0000","last_clean":"2024-03-21T14:06:38.152471+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:06:38.152471+0000","last_undegraded":"2024-03-21T14:06:38.152471+0000","last_fullsized":"2024-03-21T14:06:38.152471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:38.152382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:38.152382+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:53.181079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688272+0000","last_change":"2024-03-21T14:05:09.665071+0000","last_active":"2024-03-21T14:06:28.688272+0000","last_peered":"2024-03-21T14:06:28.688272+0000","last_clean":"2024-03-21T14:06:28.688272+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:06:28.688272+0000","last_undegraded":"2024-03-21T14:06:28.688272+0000","last_fullsized":"2024-03-21T14:06:28.688272+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:09.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.664919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.367732+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670606+0000","last_change":"2024-03-21T14:05:32.256870+0000","last_active":"2024-03-21T14:06:28.670606+0000","last_peered":"2024-03-21T14:06:28.670606+0000","last_clean":"2024-03-21T14:06:28.670606+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:06:28.670606+0000","last_undegraded":"2024-03-21T14:06:28.670606+0000","last_fullsized":"2024-03-21T14:06:28.670606+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.256757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:51.002652+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687578+0000","last_change":"2024-03-21T14:05:32.643259+0000","last_active":"2024-03-21T14:06:28.687578+0000","last_peered":"2024-03-21T14:06:28.687578+0000","last_clean":"2024-03-21T14:06:28.687578+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:06:28.687578+0000","last_undegraded":"2024-03-21T14:06:28.687578+0000","last_fullsized":"2024-03-21T14:06:28.687578+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.643133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.643133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:00.637058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685173+0000","last_change":"2024-03-21T14:05:24.664374+0000","last_active":"2024-03-21T14:06:28.685173+0000","last_peered":"2024-03-21T14:06:28.685173+0000","last_clean":"2024-03-21T14:06:28.685173+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:06:28.685173+0000","last_undegraded":"2024-03-21T14:06:28.685173+0000","last_fullsized":"2024-03-21T14:06:28.685173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:05:24.664099+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.664099+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.878115+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.967481+0000","last_change":"2024-03-21T14:06:34.967518+0000","last_active":"2024-03-21T14:06:34.967481+0000","last_peered":"2024-03-21T14:06:34.967481+0000","last_clean":"2024-03-21T14:06:34.967481+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:06:34.967481+0000","last_undegraded":"2024-03-21T14:06:34.967481+0000","last_fullsized":"2024-03-21T14:06:34.967481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.967385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.967385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:55.339956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686053+0000","last_change":"2024-03-21T14:06:14.360406+0000","last_active":"2024-03-21T14:06:28.686053+0000","last_peered":"2024-03-21T14:06:28.686053+0000","last_clean":"2024-03-21T14:06:28.686053+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:06:28.686053+0000","last_undegraded":"2024-03-21T14:06:28.686053+0000","last_fullsized":"2024-03-21T14:06:28.686053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.360300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.360300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.401246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684202+0000","last_change":"2024-03-21T14:05:39.329035+0000","last_active":"2024-03-21T14:06:28.684202+0000","last_peered":"2024-03-21T14:06:28.684202+0000","last_clean":"2024-03-21T14:06:28.684202+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:06:28.684202+0000","last_undegraded":"2024-03-21T14:06:28.684202+0000","last_fullsized":"2024-03-21T14:06:28.684202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:39.328812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:39.328812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.855477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":140,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671242+0000","last_change":"2024-03-21T14:05:19.183943+0000","last_active":"2024-03-21T14:06:28.671242+0000","last_peered":"2024-03-21T14:06:28.671242+0000","last_clean":"2024-03-21T14:06:28.671242+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:06:28.671242+0000","last_undegraded":"2024-03-21T14:06:28.671242+0000","last_fullsized":"2024-03-21T14:06:28.671242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:05:19.183805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.183805+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.082634+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682986+0000","last_change":"2024-03-21T14:05:26.392444+0000","last_active":"2024-03-21T14:06:28.682986+0000","last_peered":"2024-03-21T14:06:28.682986+0000","last_clean":"2024-03-21T14:06:28.682986+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:06:28.682986+0000","last_undegraded":"2024-03-21T14:06:28.682986+0000","last_fullsized":"2024-03-21T14:06:28.682986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:26.392328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:26.392328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:46.561081+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686492+0000","last_change":"2024-03-21T14:05:12.880436+0000","last_active":"2024-03-21T14:06:28.686492+0000","last_peered":"2024-03-21T14:06:28.686492+0000","last_clean":"2024-03-21T14:06:28.686492+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:06:28.686492+0000","last_undegraded":"2024-03-21T14:06:28.686492+0000","last_fullsized":"2024-03-21T14:06:28.686492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:12.880320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.880320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683701+0000","last_change":"2024-03-21T14:05:14.427489+0000","last_active":"2024-03-21T14:06:28.683701+0000","last_peered":"2024-03-21T14:06:28.683701+0000","last_clean":"2024-03-21T14:06:28.683701+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:06:28.683701+0000","last_undegraded":"2024-03-21T14:06:28.683701+0000","last_fullsized":"2024-03-21T14:06:28.683701+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:14.427346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.427346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.303573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671368+0000","last_change":"2024-03-21T14:05:13.203259+0000","last_active":"2024-03-21T14:06:28.671368+0000","last_peered":"2024-03-21T14:06:28.671368+0000","last_clean":"2024-03-21T14:06:28.671368+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:06:28.671368+0000","last_undegraded":"2024-03-21T14:06:28.671368+0000","last_fullsized":"2024-03-21T14:06:28.671368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:05:13.203132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.203132+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:31.940513+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671173+0000","last_change":"2024-03-21T14:05:21.139383+0000","last_active":"2024-03-21T14:06:28.671173+0000","last_peered":"2024-03-21T14:06:28.671173+0000","last_clean":"2024-03-21T14:06:28.671173+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:06:28.671173+0000","last_undegraded":"2024-03-21T14:06:28.671173+0000","last_fullsized":"2024-03-21T14:06:28.671173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:21.139263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:21.139263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.485769+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.681749+0000","last_change":"2024-03-21T14:06:17.620381+0000","last_active":"2024-03-21T14:06:28.681749+0000","last_peered":"2024-03-21T14:06:28.681749+0000","last_clean":"2024-03-21T14:06:28.681749+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:06:28.681749+0000","last_undegraded":"2024-03-21T14:06:28.681749+0000","last_fullsized":"2024-03-21T14:06:28.681749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.620252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.620252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:21.968626+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":144,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.002405+0000","last_change":"2024-03-21T14:06:36.002443+0000","last_active":"2024-03-21T14:06:36.002405+0000","last_peered":"2024-03-21T14:06:36.002405+0000","last_clean":"2024-03-21T14:06:36.002405+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:06:36.002405+0000","last_undegraded":"2024-03-21T14:06:36.002405+0000","last_fullsized":"2024-03-21T14:06:36.002405+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:36.002333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.002333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:59.288742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687080+0000","last_change":"2024-03-21T14:06:16.097848+0000","last_active":"2024-03-21T14:06:28.687080+0000","last_peered":"2024-03-21T14:06:28.687080+0000","last_clean":"2024-03-21T14:06:28.687080+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:06:28.687080+0000","last_undegraded":"2024-03-21T14:06:28.687080+0000","last_fullsized":"2024-03-21T14:06:28.687080+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:06:16.097602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.097602+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.783390+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:38.311465+0000","last_change":"2024-03-21T14:06:38.311505+0000","last_active":"2024-03-21T14:06:38.311465+0000","last_peered":"2024-03-21T14:06:38.311465+0000","last_clean":"2024-03-21T14:06:38.311465+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:06:38.311465+0000","last_undegraded":"2024-03-21T14:06:38.311465+0000","last_fullsized":"2024-03-21T14:06:38.311465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:38.311384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:38.311384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.066697+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671700+0000","last_change":"2024-03-21T14:05:22.170101+0000","last_active":"2024-03-21T14:06:28.671700+0000","last_peered":"2024-03-21T14:06:28.671700+0000","last_clean":"2024-03-21T14:06:28.671700+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:06:28.671700+0000","last_undegraded":"2024-03-21T14:06:28.671700+0000","last_fullsized":"2024-03-21T14:06:28.671700+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.169947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.169947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:46.686385+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670859+0000","last_change":"2024-03-21T14:05:31.213796+0000","last_active":"2024-03-21T14:06:28.670859+0000","last_peered":"2024-03-21T14:06:28.670859+0000","last_clean":"2024-03-21T14:06:28.670859+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:06:28.670859+0000","last_undegraded":"2024-03-21T14:06:28.670859+0000","last_fullsized":"2024-03-21T14:06:28.670859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.213713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.213713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.560680+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.285898+0000","last_change":"2024-03-21T14:06:33.285958+0000","last_active":"2024-03-21T14:06:33.285898+0000","last_peered":"2024-03-21T14:06:33.285898+0000","last_clean":"2024-03-21T14:06:33.285898+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:06:33.285898+0000","last_undegraded":"2024-03-21T14:06:33.285898+0000","last_fullsized":"2024-03-21T14:06:33.285898+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:33.285816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.285816+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:02.251092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":125,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688273+0000","last_change":"2024-03-21T14:05:22.883709+0000","last_active":"2024-03-21T14:06:28.688273+0000","last_peered":"2024-03-21T14:06:28.688273+0000","last_clean":"2024-03-21T14:06:28.688273+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:06:28.688273+0000","last_undegraded":"2024-03-21T14:06:28.688273+0000","last_fullsized":"2024-03-21T14:06:28.688273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:22.883594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.883594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.619002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":192,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.537756+0000","last_change":"2024-03-21T14:05:09.894660+0000","last_active":"2024-03-21T14:06:33.537756+0000","last_peered":"2024-03-21T14:06:33.537756+0000","last_clean":"2024-03-21T14:06:33.537756+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:06:33.537756+0000","last_undegraded":"2024-03-21T14:06:33.537756+0000","last_fullsized":"2024-03-21T14:06:33.537756+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:09.894577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.894577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":153,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682199+0000","last_change":"2024-03-21T14:06:25.667933+0000","last_active":"2024-03-21T14:06:28.682199+0000","last_peered":"2024-03-21T14:06:28.682199+0000","last_clean":"2024-03-21T14:06:28.682199+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:06:28.682199+0000","last_undegraded":"2024-03-21T14:06:28.682199+0000","last_fullsized":"2024-03-21T14:06:28.682199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:06:25.667803+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.667803+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:51.379688+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.707348+0000","last_change":"2024-03-21T14:05:18.144792+0000","last_active":"2024-03-21T14:06:28.707348+0000","last_peered":"2024-03-21T14:06:28.707348+0000","last_clean":"2024-03-21T14:06:28.707348+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:06:28.707348+0000","last_undegraded":"2024-03-21T14:06:28.707348+0000","last_fullsized":"2024-03-21T14:06:28.707348+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:18.144307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.144307+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.681470+0000","last_change":"2024-03-21T14:05:35.356518+0000","last_active":"2024-03-21T14:06:28.681470+0000","last_peered":"2024-03-21T14:06:28.681470+0000","last_clean":"2024-03-21T14:06:28.681470+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:06:28.681470+0000","last_undegraded":"2024-03-21T14:06:28.681470+0000","last_fullsized":"2024-03-21T14:06:28.681470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.356273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.356273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:00.889025+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":121,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.707636+0000","last_change":"2024-03-21T14:05:16.155310+0000","last_active":"2024-03-21T14:06:28.707636+0000","last_peered":"2024-03-21T14:06:28.707636+0000","last_clean":"2024-03-21T14:06:28.707636+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:06:28.707636+0000","last_undegraded":"2024-03-21T14:06:28.707636+0000","last_fullsized":"2024-03-21T14:06:28.707636+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.155028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.155028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684474+0000","last_change":"2024-03-21T14:05:28.539817+0000","last_active":"2024-03-21T14:06:28.684474+0000","last_peered":"2024-03-21T14:06:28.684474+0000","last_clean":"2024-03-21T14:06:28.684474+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:06:28.684474+0000","last_undegraded":"2024-03-21T14:06:28.684474+0000","last_fullsized":"2024-03-21T14:06:28.684474+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.539674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.539674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.618118+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":151,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682248+0000","last_change":"2024-03-21T14:05:12.953038+0000","last_active":"2024-03-21T14:06:28.682248+0000","last_peered":"2024-03-21T14:06:28.682248+0000","last_clean":"2024-03-21T14:06:28.682248+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:06:28.682248+0000","last_undegraded":"2024-03-21T14:06:28.682248+0000","last_fullsized":"2024-03-21T14:06:28.682248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:12.952946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.952946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:34.647609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686396+0000","last_change":"2024-03-21T14:05:27.971209+0000","last_active":"2024-03-21T14:06:28.686396+0000","last_peered":"2024-03-21T14:06:28.686396+0000","last_clean":"2024-03-21T14:06:28.686396+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:06:28.686396+0000","last_undegraded":"2024-03-21T14:06:28.686396+0000","last_fullsized":"2024-03-21T14:06:28.686396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.971081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.971081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:52.888530+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686369+0000","last_change":"2024-03-21T14:05:35.015627+0000","last_active":"2024-03-21T14:06:28.686369+0000","last_peered":"2024-03-21T14:06:28.686369+0000","last_clean":"2024-03-21T14:06:28.686369+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:06:28.686369+0000","last_undegraded":"2024-03-21T14:06:28.686369+0000","last_fullsized":"2024-03-21T14:06:28.686369+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.015517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.015517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:36.197434+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694214+0000","last_change":"2024-03-21T14:05:31.027870+0000","last_active":"2024-03-21T14:06:28.694214+0000","last_peered":"2024-03-21T14:06:28.694214+0000","last_clean":"2024-03-21T14:06:28.694214+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:06:28.694214+0000","last_undegraded":"2024-03-21T14:06:28.694214+0000","last_fullsized":"2024-03-21T14:06:28.694214+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.027754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.027754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:33.511018+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":138,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684755+0000","last_change":"2024-03-21T14:05:36.765565+0000","last_active":"2024-03-21T14:06:28.684755+0000","last_peered":"2024-03-21T14:06:28.684755+0000","last_clean":"2024-03-21T14:06:28.684755+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:06:28.684755+0000","last_undegraded":"2024-03-21T14:06:28.684755+0000","last_fullsized":"2024-03-21T14:06:28.684755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:05:36.765228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.765228+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.122075+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688943+0000","last_change":"2024-03-21T14:06:27.194477+0000","last_active":"2024-03-21T14:06:28.688943+0000","last_peered":"2024-03-21T14:06:28.688943+0000","last_clean":"2024-03-21T14:06:28.688943+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:06:28.688943+0000","last_undegraded":"2024-03-21T14:06:28.688943+0000","last_fullsized":"2024-03-21T14:06:28.688943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.194189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.194189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:57.100324+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687257+0000","last_change":"2024-03-21T14:05:07.887048+0000","last_active":"2024-03-21T14:06:28.687257+0000","last_peered":"2024-03-21T14:06:28.687257+0000","last_clean":"2024-03-21T14:06:28.687257+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:06:28.687257+0000","last_undegraded":"2024-03-21T14:06:28.687257+0000","last_fullsized":"2024-03-21T14:06:28.687257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:07.886924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:07.886924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":142,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683659+0000","last_change":"2024-03-21T14:05:17.362797+0000","last_active":"2024-03-21T14:06:28.683659+0000","last_peered":"2024-03-21T14:06:28.683659+0000","last_clean":"2024-03-21T14:06:28.683659+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:06:28.683659+0000","last_undegraded":"2024-03-21T14:06:28.683659+0000","last_fullsized":"2024-03-21T14:06:28.683659+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:17.362665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:17.362665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":151,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685062+0000","last_change":"2024-03-21T14:05:38.851695+0000","last_active":"2024-03-21T14:06:28.685062+0000","last_peered":"2024-03-21T14:06:28.685062+0000","last_clean":"2024-03-21T14:06:28.685062+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:06:28.685062+0000","last_undegraded":"2024-03-21T14:06:28.685062+0000","last_fullsized":"2024-03-21T14:06:28.685062+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.849289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.849289+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:59.036012+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.693694+0000","last_change":"2024-03-21T14:06:27.935782+0000","last_active":"2024-03-21T14:06:28.693694+0000","last_peered":"2024-03-21T14:06:28.693694+0000","last_clean":"2024-03-21T14:06:28.693694+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:06:28.693694+0000","last_undegraded":"2024-03-21T14:06:28.693694+0000","last_fullsized":"2024-03-21T14:06:28.693694+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.935658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.935658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.122257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:29.146781+0000","last_change":"2024-03-21T14:06:29.146817+0000","last_active":"2024-03-21T14:06:29.146781+0000","last_peered":"2024-03-21T14:06:29.146781+0000","last_clean":"2024-03-21T14:06:29.146781+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:06:29.146781+0000","last_undegraded":"2024-03-21T14:06:29.146781+0000","last_fullsized":"2024-03-21T14:06:29.146781+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:06:29.146695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.757841+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686814+0000","last_change":"2024-03-21T14:06:28.067288+0000","last_active":"2024-03-21T14:06:28.686814+0000","last_peered":"2024-03-21T14:06:28.686814+0000","last_clean":"2024-03-21T14:06:28.686814+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:06:28.686814+0000","last_undegraded":"2024-03-21T14:06:28.686814+0000","last_fullsized":"2024-03-21T14:06:28.686814+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.067196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.067196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.480869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":156,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683783+0000","last_change":"2024-03-21T14:06:24.661616+0000","last_active":"2024-03-21T14:06:28.683783+0000","last_peered":"2024-03-21T14:06:28.683783+0000","last_clean":"2024-03-21T14:06:28.683783+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:06:28.683783+0000","last_undegraded":"2024-03-21T14:06:28.683783+0000","last_fullsized":"2024-03-21T14:06:28.683783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:24.661482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:24.661482+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.597062+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684689+0000","last_change":"2024-03-21T14:04:27.572389+0000","last_active":"2024-03-21T14:06:28.684689+0000","last_peered":"2024-03-21T14:06:28.684689+0000","last_clean":"2024-03-21T14:06:28.684689+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:06:28.684689+0000","last_undegraded":"2024-03-21T14:06:28.684689+0000","last_fullsized":"2024-03-21T14:06:28.684689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.572309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.572309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683565+0000","last_change":"2024-03-21T14:05:38.341843+0000","last_active":"2024-03-21T14:06:28.683565+0000","last_peered":"2024-03-21T14:06:28.683565+0000","last_clean":"2024-03-21T14:06:28.683565+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:06:28.683565+0000","last_undegraded":"2024-03-21T14:06:28.683565+0000","last_fullsized":"2024-03-21T14:06:28.683565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.341705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.341705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.933155+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671057+0000","last_change":"2024-03-21T14:05:36.278697+0000","last_active":"2024-03-21T14:06:28.671057+0000","last_peered":"2024-03-21T14:06:28.671057+0000","last_clean":"2024-03-21T14:06:28.671057+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:06:28.671057+0000","last_undegraded":"2024-03-21T14:06:28.671057+0000","last_fullsized":"2024-03-21T14:06:28.671057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.278592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.856682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":169,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671527+0000","last_change":"2024-03-21T14:05:38.294588+0000","last_active":"2024-03-21T14:06:28.671527+0000","last_peered":"2024-03-21T14:06:28.671527+0000","last_clean":"2024-03-21T14:06:28.671527+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:06:28.671527+0000","last_undegraded":"2024-03-21T14:06:28.671527+0000","last_fullsized":"2024-03-21T14:06:28.671527+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.294441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.294441+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:01.276451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.705491+0000","last_change":"2024-03-21T14:05:38.046118+0000","last_active":"2024-03-21T14:06:28.705491+0000","last_peered":"2024-03-21T14:06:28.705491+0000","last_clean":"2024-03-21T14:06:28.705491+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:06:28.705491+0000","last_undegraded":"2024-03-21T14:06:28.705491+0000","last_fullsized":"2024-03-21T14:06:28.705491+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.045998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.045998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:52.031731+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":194,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.361002+0000","last_change":"2024-03-21T14:06:36.361002+0000","last_active":"2024-03-21T14:06:36.361002+0000","last_peered":"2024-03-21T14:06:36.361002+0000","last_clean":"2024-03-21T14:06:36.361002+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:06:36.361002+0000","last_undegraded":"2024-03-21T14:06:36.361002+0000","last_fullsized":"2024-03-21T14:06:36.361002+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:36.360920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.360920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:53.901675+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:32.074618+0000","last_change":"2024-03-21T14:06:32.074638+0000","last_active":"2024-03-21T14:06:32.074618+0000","last_peered":"2024-03-21T14:06:32.074618+0000","last_clean":"2024-03-21T14:06:32.074618+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:06:32.074618+0000","last_undegraded":"2024-03-21T14:06:32.074618+0000","last_fullsized":"2024-03-21T14:06:32.074618+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:32.074556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:06:32.074556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:32.791813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":139,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.697122+0000","last_change":"2024-03-21T14:05:14.713009+0000","last_active":"2024-03-21T14:06:28.697122+0000","last_peered":"2024-03-21T14:06:28.697122+0000","last_clean":"2024-03-21T14:06:28.697122+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:06:28.697122+0000","last_undegraded":"2024-03-21T14:06:28.697122+0000","last_fullsized":"2024-03-21T14:06:28.697122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.712091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.712091+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:44.503021+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:36.093279+0000","last_change":"2024-03-21T14:06:36.093312+0000","last_active":"2024-03-21T14:06:36.093279+0000","last_peered":"2024-03-21T14:06:36.093279+0000","last_clean":"2024-03-21T14:06:36.093279+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:06:36.093279+0000","last_undegraded":"2024-03-21T14:06:36.093279+0000","last_fullsized":"2024-03-21T14:06:36.093279+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:36.093213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.093213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:56.600582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":186,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.539763+0000","last_change":"2024-03-21T14:06:20.079291+0000","last_active":"2024-03-21T14:06:33.539763+0000","last_peered":"2024-03-21T14:06:33.539763+0000","last_clean":"2024-03-21T14:06:33.539763+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:06:33.539763+0000","last_undegraded":"2024-03-21T14:06:33.539763+0000","last_fullsized":"2024-03-21T14:06:33.539763+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:20.079142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.079142+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.032560+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694546+0000","last_change":"2024-03-21T14:06:20.884332+0000","last_active":"2024-03-21T14:06:28.694546+0000","last_peered":"2024-03-21T14:06:28.694546+0000","last_clean":"2024-03-21T14:06:28.694546+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:06:28.694546+0000","last_undegraded":"2024-03-21T14:06:28.694546+0000","last_fullsized":"2024-03-21T14:06:28.694546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:20.884210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.884210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.916333+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684485+0000","last_change":"2024-03-21T14:06:20.431407+0000","last_active":"2024-03-21T14:06:28.684485+0000","last_peered":"2024-03-21T14:06:28.684485+0000","last_clean":"2024-03-21T14:06:28.684485+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:06:28.684485+0000","last_undegraded":"2024-03-21T14:06:28.684485+0000","last_fullsized":"2024-03-21T14:06:28.684485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:20.431279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.431279+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.061041+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687367+0000","last_change":"2024-03-21T14:05:29.391237+0000","last_active":"2024-03-21T14:06:28.687367+0000","last_peered":"2024-03-21T14:06:28.687367+0000","last_clean":"2024-03-21T14:06:28.687367+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:06:28.687367+0000","last_undegraded":"2024-03-21T14:06:28.687367+0000","last_fullsized":"2024-03-21T14:06:28.687367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.391091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.391091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.808745+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:40.314619+0000","last_change":"2024-03-21T14:06:40.314652+0000","last_active":"2024-03-21T14:06:40.314619+0000","last_peered":"2024-03-21T14:06:40.314619+0000","last_clean":"2024-03-21T14:06:40.314619+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:06:40.314619+0000","last_undegraded":"2024-03-21T14:06:40.314619+0000","last_fullsized":"2024-03-21T14:06:40.314619+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:40.314528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:40.314528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:48.007862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686241+0000","last_change":"2024-03-21T14:05:16.811057+0000","last_active":"2024-03-21T14:06:28.686241+0000","last_peered":"2024-03-21T14:06:28.686241+0000","last_clean":"2024-03-21T14:06:28.686241+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:06:28.686241+0000","last_undegraded":"2024-03-21T14:06:28.686241+0000","last_fullsized":"2024-03-21T14:06:28.686241+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.810928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.810928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.707364+0000","last_change":"2024-03-21T14:06:16.830628+0000","last_active":"2024-03-21T14:06:28.707364+0000","last_peered":"2024-03-21T14:06:28.707364+0000","last_clean":"2024-03-21T14:06:28.707364+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:06:28.707364+0000","last_undegraded":"2024-03-21T14:06:28.707364+0000","last_fullsized":"2024-03-21T14:06:28.707364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:16.830503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.830503+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:17.447722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686706+0000","last_change":"2024-03-21T14:05:15.857954+0000","last_active":"2024-03-21T14:06:28.686706+0000","last_peered":"2024-03-21T14:06:28.686706+0000","last_clean":"2024-03-21T14:06:28.686706+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:06:28.686706+0000","last_undegraded":"2024-03-21T14:06:28.686706+0000","last_fullsized":"2024-03-21T14:06:28.686706+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:15.857799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.857799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:42.408109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683687+0000","last_change":"2024-03-21T14:05:28.362497+0000","last_active":"2024-03-21T14:06:28.683687+0000","last_peered":"2024-03-21T14:06:28.683687+0000","last_clean":"2024-03-21T14:06:28.683687+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:06:28.683687+0000","last_undegraded":"2024-03-21T14:06:28.683687+0000","last_fullsized":"2024-03-21T14:06:28.683687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.362424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.362424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.774731+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682166+0000","last_change":"2024-03-21T14:05:08.883023+0000","last_active":"2024-03-21T14:06:28.682166+0000","last_peered":"2024-03-21T14:06:28.682166+0000","last_clean":"2024-03-21T14:06:28.682166+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:06:28.682166+0000","last_undegraded":"2024-03-21T14:06:28.682166+0000","last_fullsized":"2024-03-21T14:06:28.682166+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:08.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:08.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":133,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688515+0000","last_change":"2024-03-21T14:05:15.725635+0000","last_active":"2024-03-21T14:06:28.688515+0000","last_peered":"2024-03-21T14:06:28.688515+0000","last_clean":"2024-03-21T14:06:28.688515+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:06:28.688515+0000","last_undegraded":"2024-03-21T14:06:28.688515+0000","last_fullsized":"2024-03-21T14:06:28.688515+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:05:15.725275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.725275+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"119'9310","reported_seq":9458,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.340156+0000","last_change":"2024-03-21T14:06:25.814602+0000","last_active":"2024-03-21T14:06:41.340156+0000","last_peered":"2024-03-21T14:06:41.340156+0000","last_clean":"2024-03-21T14:06:41.340156+0000","last_became_active":"2024-03-21T14:04:28.368403+0000","last_became_peered":"2024-03-21T14:04:28.368403+0000","last_unstale":"2024-03-21T14:06:41.340156+0000","last_undegraded":"2024-03-21T14:06:41.340156+0000","last_fullsized":"2024-03-21T14:06:41.340156+0000","mapping_epoch":89,"log_start":"100'4300","ondisk_log_start":"100'4300","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8546","last_scrub_stamp":"2024-03-21T14:06:25.814421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.814421+0000","objects_scrubbed":6773,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.221261+0000","scrub_duration":2159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389152768,"num_objects":5938,"num_object_clones":0,"num_object_copies":17814,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5938,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9310,"num_write_kb":487936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683471+0000","last_change":"2024-03-21T14:06:16.410140+0000","last_active":"2024-03-21T14:06:28.683471+0000","last_peered":"2024-03-21T14:06:28.683471+0000","last_clean":"2024-03-21T14:06:28.683471+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:06:28.683471+0000","last_undegraded":"2024-03-21T14:06:28.683471+0000","last_fullsized":"2024-03-21T14:06:28.683471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:16.409991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.409991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.498428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687120+0000","last_change":"2024-03-21T14:06:26.071562+0000","last_active":"2024-03-21T14:06:28.687120+0000","last_peered":"2024-03-21T14:06:28.687120+0000","last_clean":"2024-03-21T14:06:28.687120+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:06:28.687120+0000","last_undegraded":"2024-03-21T14:06:28.687120+0000","last_fullsized":"2024-03-21T14:06:28.687120+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:06:26.071314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:26.071314+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.081185+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":157,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683006+0000","last_change":"2024-03-21T14:05:40.337551+0000","last_active":"2024-03-21T14:06:28.683006+0000","last_peered":"2024-03-21T14:06:28.683006+0000","last_clean":"2024-03-21T14:06:28.683006+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:06:28.683006+0000","last_undegraded":"2024-03-21T14:06:28.683006+0000","last_fullsized":"2024-03-21T14:06:28.683006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:40.337278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:40.337278+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.381891+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":186,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:35.127130+0000","last_change":"2024-03-21T14:06:35.127130+0000","last_active":"2024-03-21T14:06:35.127130+0000","last_peered":"2024-03-21T14:06:35.127130+0000","last_clean":"2024-03-21T14:06:35.127130+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:06:35.127130+0000","last_undegraded":"2024-03-21T14:06:35.127130+0000","last_fullsized":"2024-03-21T14:06:35.127130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:35.126990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:35.126990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:00.060225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"119'9292","reported_seq":9431,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.461823+0000","last_change":"2024-03-21T14:06:38.868454+0000","last_active":"2024-03-21T14:06:41.461823+0000","last_peered":"2024-03-21T14:06:41.461823+0000","last_clean":"2024-03-21T14:06:41.461823+0000","last_became_active":"2024-03-21T14:04:28.366208+0000","last_became_peered":"2024-03-21T14:04:28.366208+0000","last_unstale":"2024-03-21T14:06:41.461823+0000","last_undegraded":"2024-03-21T14:06:41.461823+0000","last_fullsized":"2024-03-21T14:06:41.461823+0000","mapping_epoch":89,"log_start":"101'4600","ondisk_log_start":"101'4600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'9182","last_scrub_stamp":"2024-03-21T14:06:38.868297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:38.868297+0000","objects_scrubbed":6021,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:57.781353+0000","scrub_duration":1537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386007040,"num_objects":5890,"num_object_clones":0,"num_object_copies":17670,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5890,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":485824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688366+0000","last_change":"2024-03-21T14:05:23.704653+0000","last_active":"2024-03-21T14:06:28.688366+0000","last_peered":"2024-03-21T14:06:28.688366+0000","last_clean":"2024-03-21T14:06:28.688366+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:06:28.688366+0000","last_undegraded":"2024-03-21T14:06:28.688366+0000","last_fullsized":"2024-03-21T14:06:28.688366+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.703365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.703365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:31.525757+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686388+0000","last_change":"2024-03-21T14:05:19.908957+0000","last_active":"2024-03-21T14:06:28.686388+0000","last_peered":"2024-03-21T14:06:28.686388+0000","last_clean":"2024-03-21T14:06:28.686388+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:06:28.686388+0000","last_undegraded":"2024-03-21T14:06:28.686388+0000","last_fullsized":"2024-03-21T14:06:28.686388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.908814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.908814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:40.177906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671354+0000","last_change":"2024-03-21T14:05:12.169866+0000","last_active":"2024-03-21T14:06:28.671354+0000","last_peered":"2024-03-21T14:06:28.671354+0000","last_clean":"2024-03-21T14:06:28.671354+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:06:28.671354+0000","last_undegraded":"2024-03-21T14:06:28.671354+0000","last_fullsized":"2024-03-21T14:06:28.671354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:05:12.169735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.169735+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.683299+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.915330+0000","last_change":"2024-03-21T14:06:34.915366+0000","last_active":"2024-03-21T14:06:34.915330+0000","last_peered":"2024-03-21T14:06:34.915330+0000","last_clean":"2024-03-21T14:06:34.915330+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:06:34.915330+0000","last_undegraded":"2024-03-21T14:06:34.915330+0000","last_fullsized":"2024-03-21T14:06:34.915330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.915163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.915163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.070587+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"119'9278","reported_seq":9419,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.488448+0000","last_change":"2024-03-21T14:06:33.954281+0000","last_active":"2024-03-21T14:06:41.488448+0000","last_peered":"2024-03-21T14:06:41.488448+0000","last_clean":"2024-03-21T14:06:41.488448+0000","last_became_active":"2024-03-21T14:04:28.367553+0000","last_became_peered":"2024-03-21T14:04:28.367553+0000","last_unstale":"2024-03-21T14:06:41.488448+0000","last_undegraded":"2024-03-21T14:06:41.488448+0000","last_fullsized":"2024-03-21T14:06:41.488448+0000","mapping_epoch":89,"log_start":"99'3700","ondisk_log_start":"99'3700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'8984","last_scrub_stamp":"2024-03-21T14:06:33.954062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.954062+0000","objects_scrubbed":6105,"log_size":5578,"log_dups_size":0,"ondisk_log_size":5578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.888906+0000","scrub_duration":1976,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377225216,"num_objects":5756,"num_object_clones":0,"num_object_copies":17268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9278,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686477+0000","last_change":"2024-03-21T14:06:25.080381+0000","last_active":"2024-03-21T14:06:28.686477+0000","last_peered":"2024-03-21T14:06:28.686477+0000","last_clean":"2024-03-21T14:06:28.686477+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:06:28.686477+0000","last_undegraded":"2024-03-21T14:06:28.686477+0000","last_fullsized":"2024-03-21T14:06:28.686477+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:25.080145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.080145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.576368+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.689879+0000","last_change":"2024-03-21T14:05:20.225369+0000","last_active":"2024-03-21T14:06:28.689879+0000","last_peered":"2024-03-21T14:06:28.689879+0000","last_clean":"2024-03-21T14:06:28.689879+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:06:28.689879+0000","last_undegraded":"2024-03-21T14:06:28.689879+0000","last_fullsized":"2024-03-21T14:06:28.689879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:20.225234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.225234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:45.070460+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687030+0000","last_change":"2024-03-21T14:06:19.057440+0000","last_active":"2024-03-21T14:06:28.687030+0000","last_peered":"2024-03-21T14:06:28.687030+0000","last_clean":"2024-03-21T14:06:28.687030+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:06:28.687030+0000","last_undegraded":"2024-03-21T14:06:28.687030+0000","last_fullsized":"2024-03-21T14:06:28.687030+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:19.057319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:19.057319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.818134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":163,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682050+0000","last_change":"2024-03-21T14:06:26.007667+0000","last_active":"2024-03-21T14:06:28.682050+0000","last_peered":"2024-03-21T14:06:28.682050+0000","last_clean":"2024-03-21T14:06:28.682050+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:06:28.682050+0000","last_undegraded":"2024-03-21T14:06:28.682050+0000","last_fullsized":"2024-03-21T14:06:28.682050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:06:26.007559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:26.007559+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.730733+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"119'9346","reported_seq":9496,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.288608+0000","last_change":"2024-03-21T14:06:21.462241+0000","last_active":"2024-03-21T14:06:41.288608+0000","last_peered":"2024-03-21T14:06:41.288608+0000","last_clean":"2024-03-21T14:06:41.288608+0000","last_became_active":"2024-03-21T14:04:28.360990+0000","last_became_peered":"2024-03-21T14:04:28.360990+0000","last_unstale":"2024-03-21T14:06:41.288608+0000","last_undegraded":"2024-03-21T14:06:41.288608+0000","last_fullsized":"2024-03-21T14:06:41.288608+0000","mapping_epoch":89,"log_start":"97'2500","ondisk_log_start":"97'2500","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8224","last_scrub_stamp":"2024-03-21T14:06:21.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.462077+0000","objects_scrubbed":6973,"log_size":6846,"log_dups_size":0,"ondisk_log_size":6846,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.847162+0000","scrub_duration":2328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379584512,"num_objects":5792,"num_object_clones":0,"num_object_copies":17376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5792,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9346,"num_write_kb":484416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.705271+0000","last_change":"2024-03-21T14:05:34.000829+0000","last_active":"2024-03-21T14:06:28.705271+0000","last_peered":"2024-03-21T14:06:28.705271+0000","last_clean":"2024-03-21T14:06:28.705271+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:06:28.705271+0000","last_undegraded":"2024-03-21T14:06:28.705271+0000","last_fullsized":"2024-03-21T14:06:28.705271+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.000689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.026990+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:30.047695+0000","last_change":"2024-03-21T14:06:30.047729+0000","last_active":"2024-03-21T14:06:30.047695+0000","last_peered":"2024-03-21T14:06:30.047695+0000","last_clean":"2024-03-21T14:06:30.047695+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:06:30.047695+0000","last_undegraded":"2024-03-21T14:06:30.047695+0000","last_fullsized":"2024-03-21T14:06:30.047695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:30.047613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:06:30.047613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.635296+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683790+0000","last_change":"2024-03-21T14:05:23.381853+0000","last_active":"2024-03-21T14:06:28.683790+0000","last_peered":"2024-03-21T14:06:28.683790+0000","last_clean":"2024-03-21T14:06:28.683790+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:06:28.683790+0000","last_undegraded":"2024-03-21T14:06:28.683790+0000","last_fullsized":"2024-03-21T14:06:28.683790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.381726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.381726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.801210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.700098+0000","last_change":"2024-03-21T14:06:15.846648+0000","last_active":"2024-03-21T14:06:28.700098+0000","last_peered":"2024-03-21T14:06:28.700098+0000","last_clean":"2024-03-21T14:06:28.700098+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:06:28.700098+0000","last_undegraded":"2024-03-21T14:06:28.700098+0000","last_fullsized":"2024-03-21T14:06:28.700098+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:15.846504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.846504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.520598+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"119'10014","reported_seq":10107,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-21T14:06:41.243575+0000","last_change":"2024-03-21T14:06:35.026655+0000","last_active":"2024-03-21T14:06:41.243575+0000","last_peered":"2024-03-21T14:06:41.243575+0000","last_clean":"2024-03-21T14:06:41.243575+0000","last_became_active":"2024-03-21T14:04:28.369020+0000","last_became_peered":"2024-03-21T14:04:28.369020+0000","last_unstale":"2024-03-21T14:06:41.243575+0000","last_undegraded":"2024-03-21T14:06:41.243575+0000","last_fullsized":"2024-03-21T14:06:41.243575+0000","mapping_epoch":89,"log_start":"99'3700","ondisk_log_start":"99'3700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":924,"log_size":6314,"log_dups_size":0,"ondisk_log_size":6314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 6s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376438784,"num_objects":5744,"num_object_clones":0,"num_object_copies":17232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9108,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.689484+0000","last_change":"2024-03-21T14:05:16.720441+0000","last_active":"2024-03-21T14:06:28.689484+0000","last_peered":"2024-03-21T14:06:28.689484+0000","last_clean":"2024-03-21T14:06:28.689484+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:06:28.689484+0000","last_undegraded":"2024-03-21T14:06:28.689484+0000","last_fullsized":"2024-03-21T14:06:28.689484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.720331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.720331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.562402+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683272+0000","last_change":"2024-03-21T14:05:11.492743+0000","last_active":"2024-03-21T14:06:28.683272+0000","last_peered":"2024-03-21T14:06:28.683272+0000","last_clean":"2024-03-21T14:06:28.683272+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:06:28.683272+0000","last_undegraded":"2024-03-21T14:06:28.683272+0000","last_fullsized":"2024-03-21T14:06:28.683272+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:05:11.492609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.492609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670877+0000","last_change":"2024-03-21T14:05:18.141165+0000","last_active":"2024-03-21T14:06:28.670877+0000","last_peered":"2024-03-21T14:06:28.670877+0000","last_clean":"2024-03-21T14:06:28.670877+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:06:28.670877+0000","last_undegraded":"2024-03-21T14:06:28.670877+0000","last_fullsized":"2024-03-21T14:06:28.670877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:18.140955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.140955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.081222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":184,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.537617+0000","last_change":"2024-03-21T14:05:12.482027+0000","last_active":"2024-03-21T14:06:33.537617+0000","last_peered":"2024-03-21T14:06:33.537617+0000","last_clean":"2024-03-21T14:06:33.537617+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:06:33.537617+0000","last_undegraded":"2024-03-21T14:06:33.537617+0000","last_fullsized":"2024-03-21T14:06:33.537617+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:05:12.481864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.481864+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.705095+0000","last_change":"2024-03-21T14:06:14.840624+0000","last_active":"2024-03-21T14:06:28.705095+0000","last_peered":"2024-03-21T14:06:28.705095+0000","last_clean":"2024-03-21T14:06:28.705095+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:06:28.705095+0000","last_undegraded":"2024-03-21T14:06:28.705095+0000","last_fullsized":"2024-03-21T14:06:28.705095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.840509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.840509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.244776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"119'9434","reported_seq":9585,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.358872+0000","last_change":"2024-03-21T14:06:21.025026+0000","last_active":"2024-03-21T14:06:41.358872+0000","last_peered":"2024-03-21T14:06:41.358872+0000","last_clean":"2024-03-21T14:06:41.358872+0000","last_became_active":"2024-03-21T14:04:28.367387+0000","last_became_peered":"2024-03-21T14:04:28.367387+0000","last_unstale":"2024-03-21T14:06:41.358872+0000","last_undegraded":"2024-03-21T14:06:41.358872+0000","last_fullsized":"2024-03-21T14:06:41.358872+0000","mapping_epoch":89,"log_start":"99'3800","ondisk_log_start":"99'3800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8368","last_scrub_stamp":"2024-03-21T14:06:21.024886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.024886+0000","objects_scrubbed":7201,"log_size":5634,"log_dups_size":0,"ondisk_log_size":5634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.840880+0000","scrub_duration":3116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":395313152,"num_objects":6032,"num_object_clones":0,"num_object_copies":18096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6032,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687132+0000","last_change":"2024-03-21T14:06:11.955196+0000","last_active":"2024-03-21T14:06:28.687132+0000","last_peered":"2024-03-21T14:06:28.687132+0000","last_clean":"2024-03-21T14:06:28.687132+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:06:28.687132+0000","last_undegraded":"2024-03-21T14:06:28.687132+0000","last_fullsized":"2024-03-21T14:06:28.687132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:11.955082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:11.955082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.410006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683358+0000","last_change":"2024-03-21T14:05:22.226038+0000","last_active":"2024-03-21T14:06:28.683358+0000","last_peered":"2024-03-21T14:06:28.683358+0000","last_clean":"2024-03-21T14:06:28.683358+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:06:28.683358+0000","last_undegraded":"2024-03-21T14:06:28.683358+0000","last_fullsized":"2024-03-21T14:06:28.683358+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.225815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.225815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:48.284109+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":154,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687193+0000","last_change":"2024-03-21T14:06:12.171064+0000","last_active":"2024-03-21T14:06:28.687193+0000","last_peered":"2024-03-21T14:06:28.687193+0000","last_clean":"2024-03-21T14:06:28.687193+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:06:28.687193+0000","last_undegraded":"2024-03-21T14:06:28.687193+0000","last_fullsized":"2024-03-21T14:06:28.687193+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:12.170919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:12.170919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.011481+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":172,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:31.724257+0000","last_change":"2024-03-21T14:06:31.724307+0000","last_active":"2024-03-21T14:06:31.724257+0000","last_peered":"2024-03-21T14:06:31.724257+0000","last_clean":"2024-03-21T14:06:31.724257+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:06:31.724257+0000","last_undegraded":"2024-03-21T14:06:31.724257+0000","last_fullsized":"2024-03-21T14:06:31.724257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:06:31.724171+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:06:31.724171+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.792472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:32.979934+0000","last_change":"2024-03-21T14:06:32.979972+0000","last_active":"2024-03-21T14:06:32.979934+0000","last_peered":"2024-03-21T14:06:32.979934+0000","last_clean":"2024-03-21T14:06:32.979934+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:06:32.979934+0000","last_undegraded":"2024-03-21T14:06:32.979934+0000","last_fullsized":"2024-03-21T14:06:32.979934+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:32.979871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:32.979871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:52.805427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"119'10784","reported_seq":10920,"reported_epoch":119,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-21T14:06:41.525858+0000","last_change":"2024-03-21T14:06:29.986006+0000","last_active":"2024-03-21T14:06:41.525858+0000","last_peered":"2024-03-21T14:06:41.525858+0000","last_clean":"2024-03-21T14:06:41.525858+0000","last_became_active":"2024-03-21T14:04:28.363923+0000","last_became_peered":"2024-03-21T14:04:28.363923+0000","last_unstale":"2024-03-21T14:06:41.525858+0000","last_undegraded":"2024-03-21T14:06:41.525858+0000","last_fullsized":"2024-03-21T14:06:41.525858+0000","mapping_epoch":89,"log_start":"104'5200","ondisk_log_start":"104'5200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":1573,"log_size":5584,"log_dups_size":0,"ondisk_log_size":5584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 11s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384958464,"num_objects":5874,"num_object_clones":0,"num_object_copies":17622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9282,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684727+0000","last_change":"2024-03-21T14:06:15.375779+0000","last_active":"2024-03-21T14:06:28.684727+0000","last_peered":"2024-03-21T14:06:28.684727+0000","last_clean":"2024-03-21T14:06:28.684727+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:06:28.684727+0000","last_undegraded":"2024-03-21T14:06:28.684727+0000","last_fullsized":"2024-03-21T14:06:28.684727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:15.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.375667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.848357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":390,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.686680+0000","last_change":"2024-03-21T14:05:31.964155+0000","last_active":"2024-03-21T14:06:28.686680+0000","last_peered":"2024-03-21T14:06:28.686680+0000","last_clean":"2024-03-21T14:06:28.686680+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:06:28.686680+0000","last_undegraded":"2024-03-21T14:06:28.686680+0000","last_fullsized":"2024-03-21T14:06:28.686680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-03-21T14:05:31.964052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.964052+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:51.267636+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687780+0000","last_change":"2024-03-21T14:06:28.186432+0000","last_active":"2024-03-21T14:06:28.687780+0000","last_peered":"2024-03-21T14:06:28.687780+0000","last_clean":"2024-03-21T14:06:28.687780+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:06:28.687780+0000","last_undegraded":"2024-03-21T14:06:28.687780+0000","last_fullsized":"2024-03-21T14:06:28.687780+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.186306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.186306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:32.719009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":204,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.537798+0000","last_change":"2024-03-21T14:05:25.911867+0000","last_active":"2024-03-21T14:06:33.537798+0000","last_peered":"2024-03-21T14:06:33.537798+0000","last_clean":"2024-03-21T14:06:33.537798+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:06:33.537798+0000","last_undegraded":"2024-03-21T14:06:33.537798+0000","last_fullsized":"2024-03-21T14:06:33.537798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:05:25.911729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:25.911729+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:41.703513+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:37.087672+0000","last_change":"2024-03-21T14:06:37.087710+0000","last_active":"2024-03-21T14:06:37.087672+0000","last_peered":"2024-03-21T14:06:37.087672+0000","last_clean":"2024-03-21T14:06:37.087672+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:06:37.087672+0000","last_undegraded":"2024-03-21T14:06:37.087672+0000","last_fullsized":"2024-03-21T14:06:37.087672+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:37.087593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:37.087593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:43.851819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"119'9170","reported_seq":9241,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.251563+0000","last_change":"2024-03-21T14:04:28.363760+0000","last_active":"2024-03-21T14:06:41.251563+0000","last_peered":"2024-03-21T14:06:41.251563+0000","last_clean":"2024-03-21T14:06:41.251563+0000","last_became_active":"2024-03-21T14:04:28.363404+0000","last_became_peered":"2024-03-21T14:04:28.363404+0000","last_unstale":"2024-03-21T14:06:41.251563+0000","last_undegraded":"2024-03-21T14:06:41.251563+0000","last_fullsized":"2024-03-21T14:06:41.251563+0000","mapping_epoch":89,"log_start":"99'3900","ondisk_log_start":"99'3900","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":378273792,"num_objects":5772,"num_object_clones":0,"num_object_copies":17316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9170,"num_write_kb":478144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682817+0000","last_change":"2024-03-21T14:05:36.379278+0000","last_active":"2024-03-21T14:06:28.682817+0000","last_peered":"2024-03-21T14:06:28.682817+0000","last_clean":"2024-03-21T14:06:28.682817+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:06:28.682817+0000","last_undegraded":"2024-03-21T14:06:28.682817+0000","last_fullsized":"2024-03-21T14:06:28.682817+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.376091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.376091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:04.586944+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.961351+0000","last_change":"2024-03-21T14:06:28.961386+0000","last_active":"2024-03-21T14:06:28.961351+0000","last_peered":"2024-03-21T14:06:28.961351+0000","last_clean":"2024-03-21T14:06:28.961351+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:06:28.961351+0000","last_undegraded":"2024-03-21T14:06:28.961351+0000","last_fullsized":"2024-03-21T14:06:28.961351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.961272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.961272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.502960+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":390,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:39.762324+0000","last_change":"2024-03-21T14:06:13.952697+0000","last_active":"2024-03-21T14:06:39.762324+0000","last_peered":"2024-03-21T14:06:39.762324+0000","last_clean":"2024-03-21T14:06:39.762324+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:06:39.762324+0000","last_undegraded":"2024-03-21T14:06:39.762324+0000","last_fullsized":"2024-03-21T14:06:39.762324+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:06:13.952449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.952449+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.957684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":241,"num_read_kb":236,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":150,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.706373+0000","last_change":"2024-03-21T14:06:18.867126+0000","last_active":"2024-03-21T14:06:28.706373+0000","last_peered":"2024-03-21T14:06:28.706373+0000","last_clean":"2024-03-21T14:06:28.706373+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:06:28.706373+0000","last_undegraded":"2024-03-21T14:06:28.706373+0000","last_fullsized":"2024-03-21T14:06:28.706373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:18.866994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.866994+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.123054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685095+0000","last_change":"2024-03-21T14:06:27.461385+0000","last_active":"2024-03-21T14:06:28.685095+0000","last_peered":"2024-03-21T14:06:28.685095+0000","last_clean":"2024-03-21T14:06:28.685095+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:06:28.685095+0000","last_undegraded":"2024-03-21T14:06:28.685095+0000","last_fullsized":"2024-03-21T14:06:28.685095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.461298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.461298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.789908+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"119'9252","reported_seq":9399,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.179552+0000","last_change":"2024-03-21T14:06:25.324945+0000","last_active":"2024-03-21T14:06:41.179552+0000","last_peered":"2024-03-21T14:06:41.179552+0000","last_clean":"2024-03-21T14:06:41.179552+0000","last_became_active":"2024-03-21T14:04:28.367882+0000","last_became_peered":"2024-03-21T14:04:28.367882+0000","last_unstale":"2024-03-21T14:06:41.179552+0000","last_undegraded":"2024-03-21T14:06:41.179552+0000","last_fullsized":"2024-03-21T14:06:41.179552+0000","mapping_epoch":89,"log_start":"99'3400","ondisk_log_start":"99'3400","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8391","last_scrub_stamp":"2024-03-21T14:06:25.321852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.321852+0000","objects_scrubbed":6732,"log_size":5852,"log_dups_size":0,"ondisk_log_size":5852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.120366+0000","scrub_duration":1849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381419520,"num_objects":5820,"num_object_clones":0,"num_object_copies":17460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9252,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682031+0000","last_change":"2024-03-21T14:05:34.327891+0000","last_active":"2024-03-21T14:06:28.682031+0000","last_peered":"2024-03-21T14:06:28.682031+0000","last_clean":"2024-03-21T14:06:28.682031+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:06:28.682031+0000","last_undegraded":"2024-03-21T14:06:28.682031+0000","last_fullsized":"2024-03-21T14:06:28.682031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.327068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.327068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:47.990999+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683042+0000","last_change":"2024-03-21T14:05:19.237932+0000","last_active":"2024-03-21T14:06:28.683042+0000","last_peered":"2024-03-21T14:06:28.683042+0000","last_clean":"2024-03-21T14:06:28.683042+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:06:28.683042+0000","last_undegraded":"2024-03-21T14:06:28.683042+0000","last_fullsized":"2024-03-21T14:06:28.683042+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.237452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.237452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:39.545828+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":168,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.694337+0000","last_change":"2024-03-21T14:05:24.049566+0000","last_active":"2024-03-21T14:06:28.694337+0000","last_peered":"2024-03-21T14:06:28.694337+0000","last_clean":"2024-03-21T14:06:28.694337+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:06:28.694337+0000","last_undegraded":"2024-03-21T14:06:28.694337+0000","last_fullsized":"2024-03-21T14:06:28.694337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:05:24.049421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.049421+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:35.688262+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682143+0000","last_change":"2024-03-21T14:06:18.996323+0000","last_active":"2024-03-21T14:06:28.682143+0000","last_peered":"2024-03-21T14:06:28.682143+0000","last_clean":"2024-03-21T14:06:28.682143+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:06:28.682143+0000","last_undegraded":"2024-03-21T14:06:28.682143+0000","last_fullsized":"2024-03-21T14:06:28.682143+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.996252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.996252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.294909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"119'9235","reported_seq":9376,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.256000+0000","last_change":"2024-03-21T14:06:33.292754+0000","last_active":"2024-03-21T14:06:41.256000+0000","last_peered":"2024-03-21T14:06:41.256000+0000","last_clean":"2024-03-21T14:06:41.256000+0000","last_became_active":"2024-03-21T14:04:28.369002+0000","last_became_peered":"2024-03-21T14:04:28.369002+0000","last_unstale":"2024-03-21T14:06:41.256000+0000","last_undegraded":"2024-03-21T14:06:41.256000+0000","last_fullsized":"2024-03-21T14:06:41.256000+0000","mapping_epoch":89,"log_start":"101'4400","ondisk_log_start":"101'4400","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'8884","last_scrub_stamp":"2024-03-21T14:06:33.292611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.292611+0000","objects_scrubbed":6207,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:58.862461+0000","scrub_duration":1119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382009344,"num_objects":5829,"num_object_clones":0,"num_object_copies":17487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9235,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":117,"reported_epoch":119,"state":"active+clean+scrubbing","last_fresh":"2024-03-21T14:06:28.700261+0000","last_change":"2024-03-21T14:06:37.051018+0000","last_active":"2024-03-21T14:06:28.700261+0000","last_peered":"2024-03-21T14:06:28.700261+0000","last_clean":"2024-03-21T14:06:28.700261+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:06:28.700261+0000","last_undegraded":"2024-03-21T14:06:28.700261+0000","last_fullsized":"2024-03-21T14:06:28.700261+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:11.153740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.153740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684363+0000","last_change":"2024-03-21T14:06:18.451847+0000","last_active":"2024-03-21T14:06:28.684363+0000","last_peered":"2024-03-21T14:06:28.684363+0000","last_clean":"2024-03-21T14:06:28.684363+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:06:28.684363+0000","last_undegraded":"2024-03-21T14:06:28.684363+0000","last_fullsized":"2024-03-21T14:06:28.684363+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.451772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.451772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.610702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":155,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683646+0000","last_change":"2024-03-21T14:06:27.736989+0000","last_active":"2024-03-21T14:06:28.683646+0000","last_peered":"2024-03-21T14:06:28.683646+0000","last_clean":"2024-03-21T14:06:28.683646+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:06:28.683646+0000","last_undegraded":"2024-03-21T14:06:28.683646+0000","last_fullsized":"2024-03-21T14:06:28.683646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:06:27.736695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.736695+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:30.734373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":186,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.538083+0000","last_change":"2024-03-21T14:06:13.786619+0000","last_active":"2024-03-21T14:06:33.538083+0000","last_peered":"2024-03-21T14:06:33.538083+0000","last_clean":"2024-03-21T14:06:33.538083+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:06:33.538083+0000","last_undegraded":"2024-03-21T14:06:33.538083+0000","last_fullsized":"2024-03-21T14:06:33.538083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:13.786485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.786485+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.654794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"119'9321","reported_seq":9392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.335899+0000","last_change":"2024-03-21T14:04:28.366919+0000","last_active":"2024-03-21T14:06:41.335899+0000","last_peered":"2024-03-21T14:06:41.335899+0000","last_clean":"2024-03-21T14:06:41.335899+0000","last_became_active":"2024-03-21T14:04:28.366634+0000","last_became_peered":"2024-03-21T14:04:28.366634+0000","last_unstale":"2024-03-21T14:06:41.335899+0000","last_undegraded":"2024-03-21T14:06:41.335899+0000","last_fullsized":"2024-03-21T14:06:41.335899+0000","mapping_epoch":89,"log_start":"101'4700","ondisk_log_start":"101'4700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383975424,"num_objects":5859,"num_object_clones":0,"num_object_copies":17577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9321,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.155926+0000","last_change":"2024-03-21T14:06:34.155961+0000","last_active":"2024-03-21T14:06:34.155926+0000","last_peered":"2024-03-21T14:06:34.155926+0000","last_clean":"2024-03-21T14:06:34.155926+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:06:34.155926+0000","last_undegraded":"2024-03-21T14:06:34.155926+0000","last_fullsized":"2024-03-21T14:06:34.155926+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.155840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.155840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:49.184912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.706276+0000","last_change":"2024-03-21T14:05:20.137081+0000","last_active":"2024-03-21T14:06:28.706276+0000","last_peered":"2024-03-21T14:06:28.706276+0000","last_clean":"2024-03-21T14:06:28.706276+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:06:28.706276+0000","last_undegraded":"2024-03-21T14:06:28.706276+0000","last_fullsized":"2024-03-21T14:06:28.706276+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:20.136955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.136955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:49.349906+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":153,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671090+0000","last_change":"2024-03-21T14:06:18.141733+0000","last_active":"2024-03-21T14:06:28.671090+0000","last_peered":"2024-03-21T14:06:28.671090+0000","last_clean":"2024-03-21T14:06:28.671090+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:06:28.671090+0000","last_undegraded":"2024-03-21T14:06:28.671090+0000","last_fullsized":"2024-03-21T14:06:28.671090+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:06:18.141619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.141619+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:29.717483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":187,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:38.538438+0000","last_change":"2024-03-21T14:06:36.344441+0000","last_active":"2024-03-21T14:06:38.538438+0000","last_peered":"2024-03-21T14:06:38.538438+0000","last_clean":"2024-03-21T14:06:38.538438+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:06:38.538438+0000","last_undegraded":"2024-03-21T14:06:38.538438+0000","last_fullsized":"2024-03-21T14:06:38.538438+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:36.344322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.344322+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:49.912913+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"119'9308","reported_seq":9379,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.442691+0000","last_change":"2024-03-21T14:04:28.364630+0000","last_active":"2024-03-21T14:06:41.442691+0000","last_peered":"2024-03-21T14:06:41.442691+0000","last_clean":"2024-03-21T14:06:41.442691+0000","last_became_active":"2024-03-21T14:04:28.364379+0000","last_became_peered":"2024-03-21T14:04:28.364379+0000","last_unstale":"2024-03-21T14:06:41.442691+0000","last_undegraded":"2024-03-21T14:06:41.442691+0000","last_fullsized":"2024-03-21T14:06:41.442691+0000","mapping_epoch":89,"log_start":"101'4600","ondisk_log_start":"101'4600","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4708,"log_dups_size":0,"ondisk_log_size":4708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386662400,"num_objects":5900,"num_object_clones":0,"num_object_copies":17700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9308,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670716+0000","last_change":"2024-03-21T14:06:17.164352+0000","last_active":"2024-03-21T14:06:28.670716+0000","last_peered":"2024-03-21T14:06:28.670716+0000","last_clean":"2024-03-21T14:06:28.670716+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:06:28.670716+0000","last_undegraded":"2024-03-21T14:06:28.670716+0000","last_fullsized":"2024-03-21T14:06:28.670716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.164232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.164232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:43.626287+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.688356+0000","last_change":"2024-03-21T14:05:29.550541+0000","last_active":"2024-03-21T14:06:28.688356+0000","last_peered":"2024-03-21T14:06:28.688356+0000","last_clean":"2024-03-21T14:06:28.688356+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:06:28.688356+0000","last_undegraded":"2024-03-21T14:06:28.688356+0000","last_fullsized":"2024-03-21T14:06:28.688356+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.548379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.548379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:55.684852+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.685063+0000","last_change":"2024-03-21T14:06:16.045314+0000","last_active":"2024-03-21T14:06:28.685063+0000","last_peered":"2024-03-21T14:06:28.685063+0000","last_clean":"2024-03-21T14:06:28.685063+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:06:28.685063+0000","last_undegraded":"2024-03-21T14:06:28.685063+0000","last_fullsized":"2024-03-21T14:06:28.685063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:06:16.045170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.045170+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:28.072141+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670824+0000","last_change":"2024-03-21T14:05:27.174995+0000","last_active":"2024-03-21T14:06:28.670824+0000","last_peered":"2024-03-21T14:06:28.670824+0000","last_clean":"2024-03-21T14:06:28.670824+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:06:28.670824+0000","last_undegraded":"2024-03-21T14:06:28.670824+0000","last_fullsized":"2024-03-21T14:06:28.670824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.174851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.174851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.192729+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"119'9044","reported_seq":9114,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.270368+0000","last_change":"2024-03-21T14:04:28.369284+0000","last_active":"2024-03-21T14:06:41.270368+0000","last_peered":"2024-03-21T14:06:41.270368+0000","last_clean":"2024-03-21T14:06:41.270368+0000","last_became_active":"2024-03-21T14:04:28.368956+0000","last_became_peered":"2024-03-21T14:04:28.368956+0000","last_unstale":"2024-03-21T14:06:41.270368+0000","last_undegraded":"2024-03-21T14:06:41.270368+0000","last_fullsized":"2024-03-21T14:06:41.270368+0000","mapping_epoch":89,"log_start":"97'2700","ondisk_log_start":"97'2700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381681664,"num_objects":5824,"num_object_clones":0,"num_object_copies":17472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5824,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9044,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.684105+0000","last_change":"2024-03-21T14:05:24.417631+0000","last_active":"2024-03-21T14:06:28.684105+0000","last_peered":"2024-03-21T14:06:28.684105+0000","last_clean":"2024-03-21T14:06:28.684105+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:06:28.684105+0000","last_undegraded":"2024-03-21T14:06:28.684105+0000","last_fullsized":"2024-03-21T14:06:28.684105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.417495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.417495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:33.538419+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":186,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.537572+0000","last_change":"2024-03-21T14:05:37.280589+0000","last_active":"2024-03-21T14:06:33.537572+0000","last_peered":"2024-03-21T14:06:33.537572+0000","last_clean":"2024-03-21T14:06:33.537572+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:06:33.537572+0000","last_undegraded":"2024-03-21T14:06:33.537572+0000","last_fullsized":"2024-03-21T14:06:33.537572+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:37.280472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.280472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:55.817892+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.326248+0000","last_change":"2024-03-21T14:06:34.326281+0000","last_active":"2024-03-21T14:06:34.326248+0000","last_peered":"2024-03-21T14:06:34.326248+0000","last_clean":"2024-03-21T14:06:34.326248+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:06:34.326248+0000","last_undegraded":"2024-03-21T14:06:34.326248+0000","last_fullsized":"2024-03-21T14:06:34.326248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.326168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.326168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.255180+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":157,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.682539+0000","last_change":"2024-03-21T14:05:29.265158+0000","last_active":"2024-03-21T14:06:28.682539+0000","last_peered":"2024-03-21T14:06:28.682539+0000","last_clean":"2024-03-21T14:06:28.682539+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:06:28.682539+0000","last_undegraded":"2024-03-21T14:06:28.682539+0000","last_fullsized":"2024-03-21T14:06:28.682539+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:29.265009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.265009+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:50.852216+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"119'9127","reported_seq":9276,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.236129+0000","last_change":"2024-03-21T14:06:21.944486+0000","last_active":"2024-03-21T14:06:41.236129+0000","last_peered":"2024-03-21T14:06:41.236129+0000","last_clean":"2024-03-21T14:06:41.236129+0000","last_became_active":"2024-03-21T14:04:28.371337+0000","last_became_peered":"2024-03-21T14:04:28.371337+0000","last_unstale":"2024-03-21T14:06:41.236129+0000","last_undegraded":"2024-03-21T14:06:41.236129+0000","last_fullsized":"2024-03-21T14:06:41.236129+0000","mapping_epoch":89,"log_start":"97'2800","ondisk_log_start":"97'2800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8126","last_scrub_stamp":"2024-03-21T14:06:21.944346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.944346+0000","objects_scrubbed":6902,"log_size":6327,"log_dups_size":0,"ondisk_log_size":6327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.534040+0000","scrub_duration":1949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382074904,"num_objects":5831,"num_object_clones":0,"num_object_copies":17493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5831,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9127,"num_write_kb":478593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.696750+0000","last_change":"2024-03-21T14:05:10.657107+0000","last_active":"2024-03-21T14:06:28.696750+0000","last_peered":"2024-03-21T14:06:28.696750+0000","last_clean":"2024-03-21T14:06:28.696750+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:06:28.696750+0000","last_undegraded":"2024-03-21T14:06:28.696750+0000","last_fullsized":"2024-03-21T14:06:28.696750+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:10.656780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:10.656780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.396348+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:34.396207+0000","last_change":"2024-03-21T14:06:34.396244+0000","last_active":"2024-03-21T14:06:34.396207+0000","last_peered":"2024-03-21T14:06:34.396207+0000","last_clean":"2024-03-21T14:06:34.396207+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:06:34.396207+0000","last_undegraded":"2024-03-21T14:06:34.396207+0000","last_fullsized":"2024-03-21T14:06:34.396207+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.396131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.396131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:00.169873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.670921+0000","last_change":"2024-03-21T14:05:33.267089+0000","last_active":"2024-03-21T14:06:28.670921+0000","last_peered":"2024-03-21T14:06:28.670921+0000","last_clean":"2024-03-21T14:06:28.670921+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:06:28.670921+0000","last_undegraded":"2024-03-21T14:06:28.670921+0000","last_fullsized":"2024-03-21T14:06:28.670921+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:33.266985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.266985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:42.374468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":151,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671061+0000","last_change":"2024-03-21T14:06:21.108018+0000","last_active":"2024-03-21T14:06:28.671061+0000","last_peered":"2024-03-21T14:06:28.671061+0000","last_clean":"2024-03-21T14:06:28.671061+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:06:28.671061+0000","last_undegraded":"2024-03-21T14:06:28.671061+0000","last_fullsized":"2024-03-21T14:06:28.671061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:06:21.107886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.107886+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.591227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"119'9353","reported_seq":9486,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.392716+0000","last_change":"2024-03-21T14:05:36.765363+0000","last_active":"2024-03-21T14:06:41.392716+0000","last_peered":"2024-03-21T14:06:41.392716+0000","last_clean":"2024-03-21T14:06:41.392716+0000","last_became_active":"2024-03-21T14:04:28.365174+0000","last_became_peered":"2024-03-21T14:04:28.365174+0000","last_unstale":"2024-03-21T14:06:41.392716+0000","last_undegraded":"2024-03-21T14:06:41.392716+0000","last_fullsized":"2024-03-21T14:06:41.392716+0000","mapping_epoch":89,"log_start":"97'2500","ondisk_log_start":"97'2500","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'5496","last_scrub_stamp":"2024-03-21T14:05:36.764945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.764945+0000","objects_scrubbed":5416,"log_size":6853,"log_dups_size":0,"ondisk_log_size":6853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:44.153675+0000","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383975424,"num_objects":5859,"num_object_clones":0,"num_object_copies":17577,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5859,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":125,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.687207+0000","last_change":"2024-03-21T14:05:33.294251+0000","last_active":"2024-03-21T14:06:28.687207+0000","last_peered":"2024-03-21T14:06:28.687207+0000","last_clean":"2024-03-21T14:06:28.687207+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:06:28.687207+0000","last_undegraded":"2024-03-21T14:06:28.687207+0000","last_fullsized":"2024-03-21T14:06:28.687207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:33.294119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.294119+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:38.044466+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.683130+0000","last_change":"2024-03-21T14:05:30.277978+0000","last_active":"2024-03-21T14:06:28.683130+0000","last_peered":"2024-03-21T14:06:28.683130+0000","last_clean":"2024-03-21T14:06:28.683130+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:06:28.683130+0000","last_undegraded":"2024-03-21T14:06:28.683130+0000","last_fullsized":"2024-03-21T14:06:28.683130+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:30.277645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:30.277645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.448928+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:31.951691+0000","last_change":"2024-03-21T14:06:31.951712+0000","last_active":"2024-03-21T14:06:31.951691+0000","last_peered":"2024-03-21T14:06:31.951691+0000","last_clean":"2024-03-21T14:06:31.951691+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:06:31.951691+0000","last_undegraded":"2024-03-21T14:06:31.951691+0000","last_fullsized":"2024-03-21T14:06:31.951691+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:31.951638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:31.951638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:34.454617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":160,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671341+0000","last_change":"2024-03-21T14:06:16.191465+0000","last_active":"2024-03-21T14:06:28.671341+0000","last_peered":"2024-03-21T14:06:28.671341+0000","last_clean":"2024-03-21T14:06:28.671341+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:06:28.671341+0000","last_undegraded":"2024-03-21T14:06:28.671341+0000","last_fullsized":"2024-03-21T14:06:28.671341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:06:16.191249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.191249+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.656713+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"119'9450","reported_seq":9580,"reported_epoch":119,"state":"active+clean+scrubbing","last_fresh":"2024-03-21T14:06:41.409409+0000","last_change":"2024-03-21T14:06:33.994650+0000","last_active":"2024-03-21T14:06:41.409409+0000","last_peered":"2024-03-21T14:06:41.409409+0000","last_clean":"2024-03-21T14:06:41.409409+0000","last_became_active":"2024-03-21T14:04:28.364570+0000","last_became_peered":"2024-03-21T14:04:28.364570+0000","last_unstale":"2024-03-21T14:06:41.409409+0000","last_undegraded":"2024-03-21T14:06:41.409409+0000","last_fullsized":"2024-03-21T14:06:41.409409+0000","mapping_epoch":89,"log_start":"99'3800","ondisk_log_start":"99'3800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":5674,"log_size":5650,"log_dups_size":0,"ondisk_log_size":5650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 7s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216000,"num_objects":6000,"num_object_clones":0,"num_object_copies":18000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9450,"num_write_kb":494400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:41.193439+0000","last_change":"2024-03-21T14:06:41.193471+0000","last_active":"2024-03-21T14:06:41.193439+0000","last_peered":"2024-03-21T14:06:41.193439+0000","last_clean":"2024-03-21T14:06:41.193439+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:06:41.193439+0000","last_undegraded":"2024-03-21T14:06:41.193439+0000","last_fullsized":"2024-03-21T14:06:41.193439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:41.193360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:41.193360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:45.867638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":187,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:33.537617+0000","last_change":"2024-03-21T14:05:32.346872+0000","last_active":"2024-03-21T14:06:33.537617+0000","last_peered":"2024-03-21T14:06:33.537617+0000","last_clean":"2024-03-21T14:06:33.537617+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:06:33.537617+0000","last_undegraded":"2024-03-21T14:06:33.537617+0000","last_fullsized":"2024-03-21T14:06:33.537617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.346553+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:43.290171+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:39.275357+0000","last_change":"2024-03-21T14:06:39.275391+0000","last_active":"2024-03-21T14:06:39.275357+0000","last_peered":"2024-03-21T14:06:39.275357+0000","last_clean":"2024-03-21T14:06:39.275357+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:06:39.275357+0000","last_undegraded":"2024-03-21T14:06:39.275357+0000","last_fullsized":"2024-03-21T14:06:39.275357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:39.275282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:39.275282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:52.086851+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":139,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-21T14:06:28.671601+0000","last_change":"2024-03-21T14:05:14.162485+0000","last_active":"2024-03-21T14:06:28.671601+0000","last_peered":"2024-03-21T14:06:28.671601+0000","last_clean":"2024-03-21T14:06:28.671601+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:06:28.671601+0000","last_undegraded":"2024-03-21T14:06:28.671601+0000","last_fullsized":"2024-03-21T14:06:28.671601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.162391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.162391+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:06:37.569569+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6141968408,"num_objects":93720,"num_object_clones":0,"num_object_copies":281160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93720,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148310,"num_write_kb":7744897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18425589760,"data_stored":18425577544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90118,"ondisk_log_size":90118,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":241,"num_read_kb":236,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103222,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1953220,"kb_used_data":1877628,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91791932,"statfs":{"total":95995035648,"available":93994938368,"internally_reserved":0,"allocated":1922691072,"data_stored":1922327973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430267,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1939696,"kb_used_data":1872424,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91805456,"statfs":{"total":95995035648,"available":94008786944,"internally_reserved":0,"allocated":1917362176,"data_stored":1917017618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349212,"kb_used_data":2263892,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91395940,"statfs":{"total":95995035648,"available":93589442560,"internally_reserved":0,"allocated":2318225408,"data_stored":2317837804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738459,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3110408,"kb_used_data":3007040,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":90634744,"statfs":{"total":95995035648,"available":92809977856,"internally_reserved":0,"allocated":3079208960,"data_stored":3078840620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2335984,"kb_used_data":2254824,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":91409168,"statfs":{"total":95995035648,"available":93602988032,"internally_reserved":0,"allocated":2308939776,"data_stored":2308594998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3499028,"kb_used_data":3377420,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":90246124,"statfs":{"total":95995035648,"available":92412030976,"internally_reserved":0,"allocated":3458478080,"data_stored":3458097430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124320199},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018950,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1950092,"kb_used_data":1878660,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91795060,"statfs":{"total":95995035648,"available":93998141440,"internally_reserved":0,"allocated":1923747840,"data_stored":1923375096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691905,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1582208,"kb_used_data":1514936,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92162944,"statfs":{"total":95995035648,"available":94374854656,"internally_reserved":0,"allocated":1551294464,"data_stored":1550934549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3069054976,"data_stored":3069050904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2309165056,"data_stored":2309160984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2298544128,"data_stored":2298544128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3454730240,"data_stored":3454730240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1922105344,"data_stored":1922105344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1541603328,"data_stored":1541603328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1911296000,"data_stored":1911291928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1919090688,"data_stored":1919090688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:06:47.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-21T14:06:47.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:47.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:47.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:47.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:47.965 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:48.025 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:48.051 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:48.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:47 smithi003 ceph-mon[29576]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:48.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:48.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:47 smithi003 ceph-mon[40510]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:48.111 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:48.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:47 smithi118 ceph-mon[33862]: from='client.24812 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:06:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:48.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:48.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:48.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:48.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:48.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:48.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:48.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:48.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:48.733 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:48.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:48.835 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:48.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:48.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:48.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:48.994 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:48.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:48.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:49.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:06:49.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:49.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:48 smithi003 ceph-mon[29576]: pgmap v152: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s 2024-03-21T14:06:49.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:48 smithi003 ceph-mon[40510]: pgmap v152: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s 2024-03-21T14:06:49.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:49.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:48 smithi118 ceph-mon[33862]: pgmap v152: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s 2024-03-21T14:06:49.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:49.269 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:49.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:49.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:49.400 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:49.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:49.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:49.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:49.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:49.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:21329 7.1 7:8d6c4f9f:::benchmark_data_smithi003_80362_object21328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:48.833492+0000", 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 120.52760032, 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.064000908999999995, 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.600 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 21329 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833492+0000", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833492+0000", 2024-03-21T14:06:49.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833492+0000", 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2299999999999999e-07 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833563+0000", 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0772000000000006e-05 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833573+0000", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9240000000000002e-06 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833580+0000", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9519999999999997e-06 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833625+0000", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5371000000000003e-05 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833740+0000", 2024-03-21T14:06:49.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011438300000000001 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.833872+0000", 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013219300000000001 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.834615+0000", 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00074261300000000004 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.835531+0000", 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00091635100000000004 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.897417+0000", 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.061885655999999997 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.897462+0000", 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4957999999999999e-05 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:48.897492+0000", 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0812999999999998e-05 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:21607 7.1 7:8e031e0a:::benchmark_data_smithi003_80362_object21606:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:49.044204+0000", 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 120.316887815, 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057443650999999998, 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 21607 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044204+0000", 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044204+0000", 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044205+0000", 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0359999999999999e-06 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044263+0000", 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7705999999999999e-05 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044272+0000", 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0420000000000008e-06 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044277+0000", 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3179999999999997e-06 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044314+0000", 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6875999999999997e-05 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044398+0000", 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4338999999999993e-05 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.044520+0000", 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012166 2024-03-21T14:06:49.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.045487+0000", 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096658300000000003 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.045747+0000", 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025996100000000001 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.101574+0000", 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055827433000000003 2024-03-21T14:06:49.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.101611+0000", 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7366000000000001e-05 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.101648+0000", 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6331000000000002e-05 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:22034 7.1 7:840edf8a:::benchmark_data_smithi003_80362_object22033:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-03-21T14:06:49.611 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:49.317158+0000", 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 120.04393337400001, 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.064090737999999994, 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 22034 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317158+0000", 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317158+0000", 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317160+0000", 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2750000000000001e-06 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317220+0000", 2024-03-21T14:06:49.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0705999999999998e-05 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317233+0000", 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2917e-05 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317242+0000", 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4549999999999995e-06 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.614 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317282+0000", 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0054e-05 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317360+0000", 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8422000000000002e-05 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.317441+0000", 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0742999999999996e-05 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.318179+0000", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073775800000000003 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.318460+0000", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028094300000000002 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.381158+0000", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062697928 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.381211+0000", 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3065000000000001e-05 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:49.381249+0000", 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8472e-05 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:24700 7.1 7:883db035:::benchmark_data_smithi003_80362_object24699:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:50.761562+0000", 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 118.599530284, 2024-03-21T14:06:49.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.073515602999999999, 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 24700 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761562+0000", 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.618 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761562+0000", 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761563+0000", 2024-03-21T14:06:49.619 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.248e-06 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761601+0000", 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7756000000000002e-05 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761612+0000", 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1311999999999999e-05 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761619+0000", 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5100000000000001e-06 2024-03-21T14:06:49.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761669+0000", 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0083000000000001e-05 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761748+0000", 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8215999999999998e-05 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.761867+0000", 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119003 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.621 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763099+0000", 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001232749 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763643+0000", 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054397600000000005 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.834982+0000", 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.071338611999999996 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.622 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.835035+0000", 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3313000000000003e-05 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.835077+0000", 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1825e-05 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.623 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:24704 7.1 7:881999aa:::benchmark_data_smithi003_80362_object24703:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:50.763820+0000", 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 118.597272273, 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.073745805999999997, 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 24704 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.624 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763820+0000", 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763820+0000", 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763821+0000", 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5680000000000001e-06 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.625 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763930+0000", 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010933 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763944+0000", 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3366999999999999e-05 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.763952+0000", 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0080000000000002e-06 2024-03-21T14:06:49.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.764010+0000", 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8471999999999999e-05 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.764127+0000", 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116896 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.764305+0000", 2024-03-21T14:06:49.627 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017792700000000001 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.765435+0000", 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001129498 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.766112+0000", 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00067739000000000005 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.837426+0000", 2024-03-21T14:06:49.628 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.071314361000000007 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.837517+0000", 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0388000000000003e-05 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.837565+0000", 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8600999999999998e-05 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.629 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:24714 7.1 7:87410db8:::benchmark_data_smithi003_80362_object24713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:50.767726+0000", 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 118.593365402, 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.070040843000000005, 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 24714 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.630 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767726+0000", 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767726+0000", 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767728+0000", 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.46e-06 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.631 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767778+0000", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9750999999999999e-05 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767786+0000", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0520000000000002e-06 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767791+0000", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9470000000000002e-06 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767829+0000", 2024-03-21T14:06:49.632 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8472e-05 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.767979+0000", 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015008900000000001 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.768124+0000", 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000144628 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.768988+0000", 2024-03-21T14:06:49.633 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086377100000000005 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.769236+0000", 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000248302 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.837602+0000", 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.068365984000000005 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.837726+0000", 2024-03-21T14:06:49.634 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012364099999999999 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:50.837767+0000", 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1746000000000003e-05 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:27566 7.9 e96/89 7:9e8ba9bd:::benchmark_data_smithi003_80362_object27565:head v 96'1776, mlcod=96'1774)", 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:52.205548+0000", 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 117.155544222, 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056164306999999997, 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.635 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.205548+0000", 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.205548+0000", 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.205548+0000", 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8000000000000005e-07 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.636 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.206435+0000", 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00088617199999999996 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.206444+0000", 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5719999999999992e-06 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.206451+0000", 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4960000000000001e-06 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.637 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.206622+0000", 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017171800000000001 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.206680+0000", 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7552000000000001e-05 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207613+0000", 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093263899999999997 2024-03-21T14:06:49.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261712+0000", 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054099377999999997 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:27569 7.8 e96/89 7:1d9708c8:::benchmark_data_smithi003_80362_object27568:head v 96'1796, mlcod=96'1792)", 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:52.207520+0000", 2024-03-21T14:06:49.639 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 117.153571504, 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057584968, 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207520+0000", 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207520+0000", 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.640 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207521+0000", 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.75e-07 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207580+0000", 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8984000000000003e-05 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207590+0000", 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0139999999999999e-05 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.641 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.207597+0000", 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6640000000000001e-06 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261668+0000", 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054071523000000003 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.262242+0000", 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00057352100000000001 2024-03-21T14:06:49.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.265025+0000", 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027831819999999999 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.265105+0000", 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0378999999999997e-05 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.643 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:27578 7.4 e96/89 7:2e05acf7:::benchmark_data_smithi003_80362_object27577:head v 96'1718, mlcod=96'1716)", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:52.208895+0000", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 117.15219714600001, 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056322504000000002, 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.208895+0000", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.208895+0000", 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.208896+0000", 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0299999999999998e-07 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.208951+0000", 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5677000000000002e-05 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.645 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261675+0000", 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052723528999999998 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261682+0000", 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0940000000000002e-06 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.262101+0000", 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041911900000000002 2024-03-21T14:06:49.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.262119+0000", 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8318999999999999e-05 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.265168+0000", 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030490130000000002 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.265217+0000", 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8949999999999997e-05 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.647 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:27582 7.3 e96/89 7:c93663e8:::benchmark_data_smithi003_80362_object27581:head v 96'1721, mlcod=96'1719)", 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:52.211492+0000", 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 117.149599381, 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052672176000000001, 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.211492+0000", 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.648 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.211492+0000", 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.211493+0000", 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0999999999999998e-07 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.649 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.211550+0000", 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6913000000000002e-05 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261590+0000", 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.050039814000000002 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261605+0000", 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4711e-05 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.650 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261655+0000", 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0766000000000001e-05 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.261720+0000", 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4739999999999993e-05 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.264127+0000", 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024065079999999999 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.651 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:52.264165+0000", 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8013999999999999e-05 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:34255 7.1 7:879f17e0:::benchmark_data_smithi003_80362_object34254:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:55.659260+0000", 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 113.701832275, 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.060976519999999999, 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.652 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 34255 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659260+0000", 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659260+0000", 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.653 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659264+0000", 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6489999999999999e-06 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659296+0000", 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1399999999999998e-05 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659310+0000", 2024-03-21T14:06:49.654 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4637999999999999e-05 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659318+0000", 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1699999999999997e-06 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.655 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.659974+0000", 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0006556 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.660088+0000", 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011395699999999999 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.657 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.660211+0000", 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012323400000000001 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.663125+0000", 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029140149999999998 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.663943+0000", 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00081817200000000004 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.658 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.720153+0000", 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056209797999999998 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.720202+0000", 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8356000000000003e-05 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.720236+0000", 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4530999999999999e-05 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.659 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:34260 7.1 7:876ccf26:::benchmark_data_smithi003_80362_object34259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e97)", 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:04:55.661617+0000", 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 113.699474874, 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.061940964000000001, 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 34260 2024-03-21T14:06:49.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661617+0000", 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661617+0000", 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661618+0000", 2024-03-21T14:06:49.661 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.429e-06 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661665+0000", 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6717e-05 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661679+0000", 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4131000000000001e-05 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.662 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661688+0000", 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5059999999999995e-06 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661761+0000", 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3663000000000003e-05 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.661889+0000", 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012750399999999999 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.663 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.662037+0000", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000148326 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.663273+0000", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001235819 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.664062+0000", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078901999999999998 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.723485+0000", 2024-03-21T14:06:49.664 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.059422629999999997 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.723526+0000", 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1585000000000002e-05 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:04:55.723558+0000", 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1634000000000002e-05 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.665 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:83153 7.1 7:8207c333:::benchmark_data_smithi003_80362_object83152:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:05:32.583943+0000", 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 76.777148808000007, 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057156753999999997, 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83153 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.666 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.583943+0000", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.583943+0000", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.583944+0000", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9999999999999995e-07 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584015+0000", 2024-03-21T14:06:49.667 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0753999999999994e-05 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584027+0000", 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2383e-05 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584033+0000", 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2740000000000004e-06 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.668 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584093+0000", 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9571000000000003e-05 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584216+0000", 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012339800000000001 2024-03-21T14:06:49.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584357+0000", 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014087400000000001 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.588605+0000", 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004247575 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.588644+0000", 2024-03-21T14:06:49.670 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9039999999999999e-05 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.640977+0000", 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052333445999999999 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.641046+0000", 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8382999999999999e-05 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.641100+0000", 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4055999999999999e-05 2024-03-21T14:06:49.671 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:83155 7.1 7:85eaf609:::benchmark_data_smithi003_80362_object83154:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:05:32.584987+0000", 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 76.776104928999999, 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.061524521999999998, 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83155 2024-03-21T14:06:49.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584987+0000", 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584987+0000", 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.584988+0000", 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.054e-06 2024-03-21T14:06:49.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.585031+0000", 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2556999999999999e-05 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.585041+0000", 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0818999999999999e-05 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.674 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.585049+0000", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4580000000000004e-06 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.586693+0000", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001644319 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.586805+0000", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011149700000000001 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.586962+0000", 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015763800000000001 2024-03-21T14:06:49.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.588726+0000", 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017634020000000001 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.588947+0000", 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000221542 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.646372+0000", 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057425262999999997 2024-03-21T14:06:49.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.646466+0000", 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3688000000000002e-05 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:32.646511+0000", 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5284999999999998e-05 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:90090 7.1 7:8989d1b8:::benchmark_data_smithi003_80362_object90089:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-03-21T14:06:49.677 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:05:39.459000+0000", 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 69.902092314000001, 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053749927000000003, 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 90090 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459000+0000", 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459000+0000", 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459001+0000", 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.46e-06 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459072+0000", 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1008999999999997e-05 2024-03-21T14:06:49.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459085+0000", 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2836999999999999e-05 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459093+0000", 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1820000000000006e-06 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459148+0000", 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4673000000000003e-05 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.680 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459298+0000", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015071099999999999 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.459501+0000", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020296600000000001 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.461658+0000", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021570019999999999 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.681 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.461895+0000", 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023615100000000001 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.512617+0000", 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.050722578999999997 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.512697+0000", 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9420999999999997e-05 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.682 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:05:39.512749+0000", 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2936000000000003e-05 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:111957 7.1 7:86199786:::benchmark_data_smithi003_80362_object111956:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:01.544253+0000", 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.816838494000002, 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056292189999999999, 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.683 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 111957 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544253+0000", 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544253+0000", 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.684 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544255+0000", 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.39e-06 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544293+0000", 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8441e-05 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544307+0000", 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3399e-05 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.685 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544314+0000", 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1960000000000002e-06 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544370+0000", 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6282000000000001e-05 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544495+0000", 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012500400000000001 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.686 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544645+0000", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000150321 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.549012+0000", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043669859999999998 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.549330+0000", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031806800000000002 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.687 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.600441+0000", 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.051110831000000002 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.600498+0000", 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6434000000000002e-05 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.600546+0000", 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7837999999999998e-05 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.688 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:111958 7.1 7:89c57164:::benchmark_data_smithi003_80362_object111957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:01.544329+0000", 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.816762400000002, 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056443846999999998, 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 111958 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.689 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544329+0000", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544329+0000", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544330+0000", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.82e-07 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544357+0000", 2024-03-21T14:06:49.690 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6948e-05 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544365+0000", 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7100000000000007e-06 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.544370+0000", 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9790000000000001e-06 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.546730+0000", 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002360006 2024-03-21T14:06:49.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.546869+0000", 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013972600000000001 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.547046+0000", 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000176987 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.549200+0000", 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00215372 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.692 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.549425+0000", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000225357 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.600679+0000", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.051253146999999999 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.600732+0000", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3585000000000003e-05 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.600773+0000", 2024-03-21T14:06:49.693 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1100000000000003e-05 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:112187 7.1 7:82b6784a:::benchmark_data_smithi003_80362_object112186:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:01.789270+0000", 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.571821550000003, 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056606554000000003, 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 112187 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.694 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789270+0000", 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789270+0000", 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789272+0000", 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.42e-06 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.695 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789373+0000", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010134899999999999 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789386+0000", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3045e-05 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789392+0000", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.105e-06 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.696 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789496+0000", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010396299999999999 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789629+0000", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000132869 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.789774+0000", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014479 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.792456+0000", 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026817569999999999 2024-03-21T14:06:49.697 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.792502+0000", 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6199000000000003e-05 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.845770+0000", 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.053268358000000002 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.845820+0000", 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9832e-05 2024-03-21T14:06:49.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.845877+0000", 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6867000000000003e-05 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:112196 7.1 7:8471483f:::benchmark_data_smithi003_80362_object112195:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:01.797499+0000", 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 47.563592534999998, 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.058787724, 2024-03-21T14:06:49.699 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 112196 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797499+0000", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797499+0000", 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797501+0000", 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5889999999999999e-06 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797530+0000", 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8810999999999999e-05 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797543+0000", 2024-03-21T14:06:49.701 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2945000000000001e-05 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797551+0000", 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8839999999999994e-06 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797606+0000", 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5526000000000003e-05 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797735+0000", 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128885 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.702 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.797872+0000", 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013672700000000001 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.799780+0000", 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019080189999999999 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.800072+0000", 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029220400000000001 2024-03-21T14:06:49.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.856173+0000", 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056101129 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.856245+0000", 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1587000000000004e-05 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:01.856287+0000", 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2417999999999999e-05 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.704 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:152493 7.1 7:86240b90:::benchmark_data_smithi003_80362_object31476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:48.244577+0000", 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.1165149999999999, 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055068536000000001, 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 152493 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:49.705 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244577+0000", 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244577+0000", 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244579+0000", 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.66e-06 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.706 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244585+0000", 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0920000000000003e-06 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244596+0000", 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1719000000000001e-05 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244602+0000", 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5659999999999997e-06 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.707 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244643+0000", 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1057999999999999e-05 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244723+0000", 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0552999999999999e-05 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.244790+0000", 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6917999999999999e-05 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.708 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.246945+0000", 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021548470000000001 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.248649+0000", 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001703242 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.299522+0000", 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.050873552000000002 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.709 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.299583+0000", 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1024999999999999e-05 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.299645+0000", 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2304000000000005e-05 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:49.710 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:06:49.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:49.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:49.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:49.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:49.879 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:49.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:49.981 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:50.010 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:50.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:50.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:50.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:50.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:50.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:50.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:50.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:50.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:50.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:50.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:50.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:50.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:50.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:06:50.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:50.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:50.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:51.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:51.124 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:51.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:50 smithi118 ceph-mon[33862]: pgmap v153: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s 2024-03-21T14:06:51.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:50 smithi003 ceph-mon[40510]: pgmap v153: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s 2024-03-21T14:06:51.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:50 smithi003 ceph-mon[29576]: pgmap v153: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s 2024-03-21T14:06:51.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:51.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:51.412 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:51.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:51.543 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:51.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:51.668 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:06:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:51.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:51.741 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:06:51.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:51.851 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:51.884 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:06:51.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:51.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:51.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:51.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:52.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:52.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:52.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3562206975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:52.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:51 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:52.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:51 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3562206975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:52.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:51 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:52.140 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:51 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3562206975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:52.140 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:51 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:06:52.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:52.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:52.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:52.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:52.405 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:52.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:52.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:52.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:52.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:52.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.813 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:52.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:52.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:52.945 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:52.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:53.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:53.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:53.134 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:52 smithi118 ceph-mon[33862]: pgmap v154: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 603 op/s 2024-03-21T14:06:53.135 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:52 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:53.135 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:52 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:53.135 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:52 smithi118 ceph-mon[33862]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T14:06:53.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:53.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[40510]: pgmap v154: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 603 op/s 2024-03-21T14:06:53.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:53.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:53.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[40510]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T14:06:53.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[29576]: pgmap v154: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 603 op/s 2024-03-21T14:06:53.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:06:53.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:06:53.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:52 smithi003 ceph-mon[29576]: osdmap e121: 8 total, 8 up, 8 in 2024-03-21T14:06:53.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:53.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:53.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:53.372 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:53.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:53.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:53.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:53.754 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:156704 7.0 7:0e23a85d:::benchmark_data_smithi003_80362_object35687:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:53.739355+0000", 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.013679116999999999, 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013713912999999999, 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 156704 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:53.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739355+0000", 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739355+0000", 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739356+0000", 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3850000000000001e-06 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739362+0000", 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.835e-06 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739374+0000", 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2584e-05 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739382+0000", 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4070000000000004e-06 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739429+0000", 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7068999999999999e-05 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739554+0000", 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012488500000000001 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,3", 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.739673+0000", 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011874 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.743092+0000", 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034190280000000002 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.746215+0000", 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031233709999999998 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:156714 7.a e121/89 7:5cac49a0:::benchmark_data_smithi003_80362_object35697:head v 121'9832, mlcod=121'9830)", 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:53.752102+0000", 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00093175300000000001, 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010622400000000001, 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752102+0000", 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:53.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752102+0000", 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752103+0000", 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0330000000000001e-06 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752115+0000", 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1622e-05 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752129+0000", 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4131000000000001e-05 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752137+0000", 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5790000000000004e-06 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752221+0000", 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3546000000000003e-05 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752247+0000", 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5831999999999999e-05 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:156713 7.e e121/89 7:7ab585e7:::benchmark_data_smithi003_80362_object35696:head v 121'9877, mlcod=121'9874)", 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:53.752172+0000", 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00086149499999999999, 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010522179999999999, 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752172+0000", 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752172+0000", 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752173+0000", 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2399999999999998e-07 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752181+0000", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1510000000000004e-06 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752192+0000", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0621999999999999e-05 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752198+0000", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9699999999999996e-06 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752286+0000", 2024-03-21T14:06:53.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8456999999999997e-05 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.752307+0000", 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1471000000000002e-05 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.24692.0:156704 7.0 e121/89 ondisk, result = 0)", 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:53.753087+0000", 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4294967295.9999471, 2024-03-21T14:06:53.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019766400000000001, 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753087+0000", 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753087+0000", 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:53.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753087+0000", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8800000000000002e-07 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753096+0000", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4349999999999997e-06 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753111+0000", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5495e-05 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753118+0000", 2024-03-21T14:06:53.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1609999999999997e-06 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753214+0000", 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5336999999999999e-05 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753221+0000", 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.769 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.24692.0:156715 7.b e121/89 7:dae31a74:::benchmark_data_smithi003_80362_object35698:head v 121'9862, mlcod=121'9859)", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:53.753139+0000", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4294967295.9998946, 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020703099999999999, 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753139+0000", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753139+0000", 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:06:53.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753140+0000", 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.91e-07 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753144+0000", 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5549999999999996e-06 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753151+0000", 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8210000000000004e-06 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753156+0000", 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5909999999999998e-06 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753216+0000", 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0783999999999999e-05 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:53.753235+0000", 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8765000000000001e-05 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.772 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:06:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:06:53.773 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 5 2024-03-21T14:06:53.773 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:06:53.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:53.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:53.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:53.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:53.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:53.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:54.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:54.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:54.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:54.224 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:54.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:54.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:54.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:54.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.519 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:54.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:54.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:06:54.620 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:54.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:54.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:54.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:54.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:54.918 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pgp_num to 1 2024-03-21T14:06:54.928 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:54.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:55.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:55.061 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:55.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:06:55.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:55.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:54 smithi003 ceph-mon[29576]: pgmap v156: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 617 op/s 2024-03-21T14:06:55.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:54 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1050980725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T14:06:55.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:54 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T14:06:55.178 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:54 smithi003 ceph-mon[40510]: pgmap v156: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 617 op/s 2024-03-21T14:06:55.178 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:54 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1050980725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T14:06:55.178 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:54 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T14:06:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:54 smithi118 ceph-mon[33862]: pgmap v156: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 617 op/s 2024-03-21T14:06:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:54 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1050980725' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T14:06:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:54 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-21T14:06:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:55.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:55.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:55.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:55.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:55.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:55.646 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:06:55.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:55.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:55.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:55.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.825 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:55.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:55.926 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:55.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:55.958 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:55.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:56.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:56.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:56.135 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:55 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-21T14:06:56.135 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:55 smithi118 ceph-mon[33862]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T14:06:56.177 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:55 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-21T14:06:56.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:55 smithi003 ceph-mon[29576]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T14:06:56.178 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:55 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-21T14:06:56.178 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:55 smithi003 ceph-mon[40510]: osdmap e122: 8 total, 8 up, 8 in 2024-03-21T14:06:56.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:56.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:56.336 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:56.360 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:56.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:56.489 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:56.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:56.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:56.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:06:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:06:56.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:06:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:06:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:06:56.767 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:56.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:56.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:56.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:57.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:57.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:57.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:57.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:56 smithi118 ceph-mon[33862]: pgmap v158: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 398 op/s 2024-03-21T14:06:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:56 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:57.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:57.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:57.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:56 smithi003 ceph-mon[29576]: pgmap v158: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 398 op/s 2024-03-21T14:06:57.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:56 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:57.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:56 smithi003 ceph-mon[40510]: pgmap v158: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 398 op/s 2024-03-21T14:06:57.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:56 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:06:57.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:06:57.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:57.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:06:57.440 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:57.473 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:57.575 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:57.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.602 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:57.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:57.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:57.731 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:57.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:57.860 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:57.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:06:57.996 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:57.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:57.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:06:58.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:06:58.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:06:58.069 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:06:58.070 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:06:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:06:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:06:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:06:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:06:58.071 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:06:58.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:57 smithi003 ceph-mon[29576]: pgmap v159: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T14:06:58.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:06:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2718549937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:58.071 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:57 smithi003 ceph-mon[40510]: pgmap v159: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T14:06:58.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:06:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2718549937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:58.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:58.126 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:57 smithi118 ceph-mon[33862]: pgmap v159: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T14:06:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:06:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2718549937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:06:58.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:06:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:58.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:06:58.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:58.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:58.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:58.547 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:58.642 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:06:58.645 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:06:58.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:58.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:58.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:58.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:58.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:58.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:58.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:59.046 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:06:59.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:59.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:06:59.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:59.203 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:59.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:59.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:59.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:06:59.480 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:06:59.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:06:59.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:06:59.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:06:59.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:06:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:06:59.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:06:59.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:06:59.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:06:59.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:06:59.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:00.024 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:00.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:00.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.225 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:00.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:00.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:07:00.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:00.331 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:00.353 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:00.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:00.434 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:00 smithi118 ceph-mon[33862]: pgmap v160: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 872 op/s 2024-03-21T14:07:00.434 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:00 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:00.434 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:00 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:00.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:00.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:00 smithi003 ceph-mon[29576]: pgmap v160: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 872 op/s 2024-03-21T14:07:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:00 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:00 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:00.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:00 smithi003 ceph-mon[40510]: pgmap v160: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 872 op/s 2024-03-21T14:07:00.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:00 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:00.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:00 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:00.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:00.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:00.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:00.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:00.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:00.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:00.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:01.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:01.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:01.160 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:01.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:01.180 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:07:01.180 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:07:01.180 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-21T14:07:01.180 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-21T14:07:01.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:07:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:01.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:01.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:01.357 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:01.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:01.459 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:01.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:01.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:01.554 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:01 smithi118 ceph-mon[33862]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T14:07:01.554 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:01.554 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:01 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:01.554 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:01 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:01.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:01.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:01.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[29576]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[40510]: osdmap e123: 8 total, 8 up, 8 in 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:01 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:01.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:01.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:01.901 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:01.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:02.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:02.037 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:02.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:02.167 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:02.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:02.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:02.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:07:02.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:02.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:02.421 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:02.454 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:02.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:02.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:02 smithi003 ceph-mon[29576]: pgmap v162: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 851 op/s 2024-03-21T14:07:02.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:02 smithi003 ceph-mon[40510]: pgmap v162: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 851 op/s 2024-03-21T14:07:02.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:02.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:02 smithi118 ceph-mon[33862]: pgmap v162: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 851 op/s 2024-03-21T14:07:02.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:02.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:02.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:02.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:02.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:02.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:02.982 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:02.982 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:03.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:03.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:03.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:03.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:03.178 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:03.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:03.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:03.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:03.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:03.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:03.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:03.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:03.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:03.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:03.607 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:03.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:03.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:03.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:07:03.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:03.871 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:03.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:03.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:04.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:04.099 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:04.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:04.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:04.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:04.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:04.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:04.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:04.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[29576]: pgmap v163: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-21T14:07:04.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:04.304 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:04.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:04.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:04.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[40510]: pgmap v163: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-21T14:07:04.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:04.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:04 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:04.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:04 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:04 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:04 smithi118 ceph-mon[33862]: pgmap v163: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-21T14:07:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:04 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:04 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:04.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:04.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:04.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:04.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:04.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:04.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:04.852 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:04.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:04.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:04.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:05.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:05.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:05.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:05.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:05.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[29576]: osdmap e124: 8 total, 8 up, 8 in 2024-03-21T14:07:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:07:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:07:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4115682293' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:07:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[40510]: osdmap e124: 8 total, 8 up, 8 in 2024-03-21T14:07:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:07:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:07:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:05 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4115682293' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:07:05.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:05.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:05.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:05 smithi118 ceph-mon[33862]: osdmap e124: 8 total, 8 up, 8 in 2024-03-21T14:07:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:05 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:07:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:05 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:07:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:05 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:05 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4115682293' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:07:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:05.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:05.498 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:07:05.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:05.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:05.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:05.736 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:05.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:05.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:05.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:05.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:06.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:06.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:06.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[29576]: pgmap v165: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s 2024-03-21T14:07:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4115682293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:07:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[29576]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T14:07:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:07:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:07:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[40510]: pgmap v165: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s 2024-03-21T14:07:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4115682293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:07:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[40510]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T14:07:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:07:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:07:06.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:06.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:06.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-mon[33862]: pgmap v165: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s 2024-03-21T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4115682293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-mon[33862]: osdmap e125: 8 total, 8 up, 8 in 2024-03-21T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:07:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:07:06.432 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:06.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:06.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:06.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:06.594 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:07:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:07:06.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:06.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:06.800 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:07:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:07:06.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:06.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:06.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:06.871 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:06.871 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:06.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:06.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:06.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:07.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:07.126 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:07.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:07.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:07.278 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:07:07.322 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:07:07.365 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:07:07.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:07.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:07.409 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:07:07.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:07.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:07.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:07.451 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:07:07.489 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:07:07.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:07.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:07.534 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:07:07.578 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:07:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:07:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:07:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:07:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:07:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:07:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:07:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:07:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:07:07.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:07.657 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:07.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T14:07:07.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:07.803 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:07.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:07.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:07.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:08.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:08.054 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:08.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:08.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:08.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:08.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:08.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:08.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:08.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:08.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: 7.6 scrub starts 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2561636413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:07:08.476 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:08 smithi118 ceph-mon[33862]: pgmap v167: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: 7.6 scrub starts 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2561636413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:07:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[29576]: pgmap v167: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: 7.6 scrub starts 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2561636413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:07:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:08 smithi003 ceph-mon[40510]: pgmap v167: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.46k op/s 2024-03-21T14:07:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:08.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:08.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:08.754 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:08.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:08.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:08.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:08.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:09.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:09.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:09.214 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 7.5 scrub starts 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:07:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 7.5 scrub starts 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:07:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:09 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:07:09.319 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:09.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:09.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:09.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:09.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:09.623 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 7.5 scrub starts 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:07:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:09 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:07:09.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:09.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.766 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:09.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:09.792 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:07:09.805 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:07:09.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:09.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:09.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:09.998 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:10.028 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:10.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:10.157 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:10.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:10.292 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:10.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: 7.6 scrub ok 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: 7.5 scrub ok 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: pgmap v168: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:07:10.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:10 smithi118 ceph-mon[33862]: from='client.24739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: 7.6 scrub ok 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: 7.5 scrub ok 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: pgmap v168: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[29576]: from='client.24739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:07:10.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:07:10.362 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: 7.6 scrub ok 2024-03-21T14:07:10.362 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: 7.5 scrub ok 2024-03-21T14:07:10.362 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: pgmap v168: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-21T14:07:10.362 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:10 smithi003 ceph-mon[40510]: from='client.24739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:07:10.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:10.425 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:10.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:10.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:10.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:10.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:10.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:07:10.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:10.847 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:10.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:10.984 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:10.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:11.085 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:11.118 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:11.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:11.144 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:07:11.144 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:07:11.144 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:07:11.144 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:07:11.145 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:07:11.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:11.252 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:11.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:11.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:11 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:11 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:11 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:11 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:11 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:07:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:11 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:07:11.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:11.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:07:11.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:07:11.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:07:11.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:07:11.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:07:11.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:07:11.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:07:11.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:07:11.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:07:11.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:07:11.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:07:11.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:11 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:07:11.526 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:11.549 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":168,"stamp":"2024-03-21T14:07:09.321497+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4749269440,"num_objects":72658,"num_object_clones":0,"num_object_copies":217974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72658,"num_whiteouts":0,"num_read":1201,"num_read_kb":1047,"num_write":170185,"num_write_kb":7747358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1771682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90619,"ondisk_log_size":90619,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":14839820,"kb_used_data":14010076,"kb_used_omap":12,"kb_used_meta":829619,"kb_avail":735121396,"statfs":{"total":767960285184,"available":752764309504,"internally_reserved":0,"allocated":14346317824,"data_stored":14343143088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":849530416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-542965760,"num_objects":-8285,"num_object_clones":0,"num_object_copies":-24855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8285,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8285,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002513"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500082+0000","last_change":"2024-03-21T14:05:24.217692+0000","last_active":"2024-03-21T14:07:05.500082+0000","last_peered":"2024-03-21T14:07:05.500082+0000","last_clean":"2024-03-21T14:07:05.500082+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:07:05.500082+0000","last_undegraded":"2024-03-21T14:07:05.500082+0000","last_fullsized":"2024-03-21T14:07:05.500082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.217564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.217564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004038+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498503+0000","last_change":"2024-03-21T14:06:30.718732+0000","last_active":"2024-03-21T14:07:05.498503+0000","last_peered":"2024-03-21T14:07:05.498503+0000","last_clean":"2024-03-21T14:07:05.498503+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:07:05.498503+0000","last_undegraded":"2024-03-21T14:07:05.498503+0000","last_fullsized":"2024-03-21T14:07:05.498503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:06:30.718605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:47.380214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505538+0000","last_change":"2024-03-21T14:05:37.038645+0000","last_active":"2024-03-21T14:07:05.505538+0000","last_peered":"2024-03-21T14:07:05.505538+0000","last_clean":"2024-03-21T14:07:05.505538+0000","last_became_active":"2024-03-21T14:02:33.442430+0000","last_became_peered":"2024-03-21T14:02:33.442430+0000","last_unstale":"2024-03-21T14:07:05.505538+0000","last_undegraded":"2024-03-21T14:07:05.505538+0000","last_fullsized":"2024-03-21T14:07:05.505538+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:37.038531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.038531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.964335+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500054+0000","last_change":"2024-03-21T14:06:38.152523+0000","last_active":"2024-03-21T14:07:05.500054+0000","last_peered":"2024-03-21T14:07:05.500054+0000","last_clean":"2024-03-21T14:07:05.500054+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:07:05.500054+0000","last_undegraded":"2024-03-21T14:07:05.500054+0000","last_fullsized":"2024-03-21T14:07:05.500054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:38.152382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:38.152382+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:53.181079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498030+0000","last_change":"2024-03-21T14:05:09.665071+0000","last_active":"2024-03-21T14:07:05.498030+0000","last_peered":"2024-03-21T14:07:05.498030+0000","last_clean":"2024-03-21T14:07:05.498030+0000","last_became_active":"2024-03-21T14:02:39.517140+0000","last_became_peered":"2024-03-21T14:02:39.517140+0000","last_unstale":"2024-03-21T14:07:05.498030+0000","last_undegraded":"2024-03-21T14:07:05.498030+0000","last_fullsized":"2024-03-21T14:07:05.498030+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:09.664919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.664919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296679+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502249+0000","last_change":"2024-03-21T14:05:32.256870+0000","last_active":"2024-03-21T14:07:05.502249+0000","last_peered":"2024-03-21T14:07:05.502249+0000","last_clean":"2024-03-21T14:07:05.502249+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:07:05.502249+0000","last_undegraded":"2024-03-21T14:07:05.502249+0000","last_fullsized":"2024-03-21T14:07:05.502249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.256757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499794+0000","last_change":"2024-03-21T14:05:32.643259+0000","last_active":"2024-03-21T14:07:05.499794+0000","last_peered":"2024-03-21T14:07:05.499794+0000","last_clean":"2024-03-21T14:07:05.499794+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:07:05.499794+0000","last_undegraded":"2024-03-21T14:07:05.499794+0000","last_fullsized":"2024-03-21T14:07:05.499794+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:32.643133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.643133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:07.242341+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":176,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497356+0000","last_change":"2024-03-21T14:05:24.664374+0000","last_active":"2024-03-21T14:07:05.497356+0000","last_peered":"2024-03-21T14:07:05.497356+0000","last_clean":"2024-03-21T14:07:05.497356+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:07:05.497356+0000","last_undegraded":"2024-03-21T14:07:05.497356+0000","last_fullsized":"2024-03-21T14:07:05.497356+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:05:24.664099+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.664099+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296817+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503704+0000","last_change":"2024-03-21T14:06:34.967518+0000","last_active":"2024-03-21T14:07:05.503704+0000","last_peered":"2024-03-21T14:07:05.503704+0000","last_clean":"2024-03-21T14:07:05.503704+0000","last_became_active":"2024-03-21T14:02:39.469165+0000","last_became_peered":"2024-03-21T14:02:39.469165+0000","last_unstale":"2024-03-21T14:07:05.503704+0000","last_undegraded":"2024-03-21T14:07:05.503704+0000","last_fullsized":"2024-03-21T14:07:05.503704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.967385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.967385+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:55.339956+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498238+0000","last_change":"2024-03-21T14:06:14.360406+0000","last_active":"2024-03-21T14:07:05.498238+0000","last_peered":"2024-03-21T14:07:05.498238+0000","last_clean":"2024-03-21T14:07:05.498238+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:07:05.498238+0000","last_undegraded":"2024-03-21T14:07:05.498238+0000","last_fullsized":"2024-03-21T14:07:05.498238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.360300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.360300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.401246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498388+0000","last_change":"2024-03-21T14:05:39.329035+0000","last_active":"2024-03-21T14:07:05.498388+0000","last_peered":"2024-03-21T14:07:05.498388+0000","last_clean":"2024-03-21T14:07:05.498388+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:07:05.498388+0000","last_undegraded":"2024-03-21T14:07:05.498388+0000","last_fullsized":"2024-03-21T14:07:05.498388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:39.328812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:39.328812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500236+0000","last_change":"2024-03-21T14:05:19.183943+0000","last_active":"2024-03-21T14:07:05.500236+0000","last_peered":"2024-03-21T14:07:05.500236+0000","last_clean":"2024-03-21T14:07:05.500236+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:07:05.500236+0000","last_undegraded":"2024-03-21T14:07:05.500236+0000","last_fullsized":"2024-03-21T14:07:05.500236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:05:19.183805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.183805+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004303+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497918+0000","last_change":"2024-03-21T14:05:26.392444+0000","last_active":"2024-03-21T14:07:05.497918+0000","last_peered":"2024-03-21T14:07:05.497918+0000","last_clean":"2024-03-21T14:07:05.497918+0000","last_became_active":"2024-03-21T14:02:39.466594+0000","last_became_peered":"2024-03-21T14:02:39.466594+0000","last_unstale":"2024-03-21T14:07:05.497918+0000","last_undegraded":"2024-03-21T14:07:05.497918+0000","last_fullsized":"2024-03-21T14:07:05.497918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:26.392328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:26.392328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506638+0000","last_change":"2024-03-21T14:05:12.880436+0000","last_active":"2024-03-21T14:07:05.506638+0000","last_peered":"2024-03-21T14:07:05.506638+0000","last_clean":"2024-03-21T14:07:05.506638+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:07:05.506638+0000","last_undegraded":"2024-03-21T14:07:05.506638+0000","last_fullsized":"2024-03-21T14:07:05.506638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:12.880320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.880320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498255+0000","last_change":"2024-03-21T14:05:14.427489+0000","last_active":"2024-03-21T14:07:05.498255+0000","last_peered":"2024-03-21T14:07:05.498255+0000","last_clean":"2024-03-21T14:07:05.498255+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:07:05.498255+0000","last_undegraded":"2024-03-21T14:07:05.498255+0000","last_fullsized":"2024-03-21T14:07:05.498255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:14.427346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.427346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":146,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501565+0000","last_change":"2024-03-21T14:05:13.203259+0000","last_active":"2024-03-21T14:07:05.501565+0000","last_peered":"2024-03-21T14:07:05.501565+0000","last_clean":"2024-03-21T14:07:05.501565+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:07:05.501565+0000","last_undegraded":"2024-03-21T14:07:05.501565+0000","last_fullsized":"2024-03-21T14:07:05.501565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:05:13.203132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:13.203132+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004081+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":129,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501347+0000","last_change":"2024-03-21T14:05:21.139383+0000","last_active":"2024-03-21T14:07:05.501347+0000","last_peered":"2024-03-21T14:07:05.501347+0000","last_clean":"2024-03-21T14:07:05.501347+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:07:05.501347+0000","last_undegraded":"2024-03-21T14:07:05.501347+0000","last_fullsized":"2024-03-21T14:07:05.501347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:21.139263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:21.139263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004386+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498120+0000","last_change":"2024-03-21T14:06:17.620381+0000","last_active":"2024-03-21T14:07:05.498120+0000","last_peered":"2024-03-21T14:07:05.498120+0000","last_clean":"2024-03-21T14:07:05.498120+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:07:05.498120+0000","last_undegraded":"2024-03-21T14:07:05.498120+0000","last_fullsized":"2024-03-21T14:07:05.498120+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.620252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.620252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:21.968626+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":156,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503171+0000","last_change":"2024-03-21T14:06:36.002443+0000","last_active":"2024-03-21T14:07:05.503171+0000","last_peered":"2024-03-21T14:07:05.503171+0000","last_clean":"2024-03-21T14:07:05.503171+0000","last_became_active":"2024-03-21T14:02:33.446710+0000","last_became_peered":"2024-03-21T14:02:33.446710+0000","last_unstale":"2024-03-21T14:07:05.503171+0000","last_undegraded":"2024-03-21T14:07:05.503171+0000","last_fullsized":"2024-03-21T14:07:05.503171+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:36.002333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.002333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:59.288742+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":177,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501140+0000","last_change":"2024-03-21T14:06:16.097848+0000","last_active":"2024-03-21T14:07:05.501140+0000","last_peered":"2024-03-21T14:07:05.501140+0000","last_clean":"2024-03-21T14:07:05.501140+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:07:05.501140+0000","last_undegraded":"2024-03-21T14:07:05.501140+0000","last_fullsized":"2024-03-21T14:07:05.501140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:06:16.097602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.097602+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.783390+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503608+0000","last_change":"2024-03-21T14:06:38.311505+0000","last_active":"2024-03-21T14:07:05.503608+0000","last_peered":"2024-03-21T14:07:05.503608+0000","last_clean":"2024-03-21T14:07:05.503608+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:07:05.503608+0000","last_undegraded":"2024-03-21T14:07:05.503608+0000","last_fullsized":"2024-03-21T14:07:05.503608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:38.311384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:38.311384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.066697+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":132,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501138+0000","last_change":"2024-03-21T14:05:22.170101+0000","last_active":"2024-03-21T14:07:05.501138+0000","last_peered":"2024-03-21T14:07:05.501138+0000","last_clean":"2024-03-21T14:07:05.501138+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:07:05.501138+0000","last_undegraded":"2024-03-21T14:07:05.501138+0000","last_fullsized":"2024-03-21T14:07:05.501138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.169947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.169947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004269+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500330+0000","last_change":"2024-03-21T14:05:31.213796+0000","last_active":"2024-03-21T14:07:05.500330+0000","last_peered":"2024-03-21T14:07:05.500330+0000","last_clean":"2024-03-21T14:07:05.500330+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:07:05.500330+0000","last_undegraded":"2024-03-21T14:07:05.500330+0000","last_fullsized":"2024-03-21T14:07:05.500330+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.213713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.213713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004331+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":167,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497311+0000","last_change":"2024-03-21T14:06:33.285958+0000","last_active":"2024-03-21T14:07:05.497311+0000","last_peered":"2024-03-21T14:07:05.497311+0000","last_clean":"2024-03-21T14:07:05.497311+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:07:05.497311+0000","last_undegraded":"2024-03-21T14:07:05.497311+0000","last_fullsized":"2024-03-21T14:07:05.497311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:33.285816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.285816+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:02.251092+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"72'1","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506921+0000","last_change":"2024-03-21T14:05:22.883709+0000","last_active":"2024-03-21T14:07:05.506921+0000","last_peered":"2024-03-21T14:07:05.506921+0000","last_clean":"2024-03-21T14:07:05.506921+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:07:05.506921+0000","last_undegraded":"2024-03-21T14:07:05.506921+0000","last_fullsized":"2024-03-21T14:07:05.506921+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:22.883594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.883594+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":210,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.494303+0000","last_change":"2024-03-21T14:05:09.894660+0000","last_active":"2024-03-21T14:07:05.494303+0000","last_peered":"2024-03-21T14:07:05.494303+0000","last_clean":"2024-03-21T14:07:05.494303+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:07:05.494303+0000","last_undegraded":"2024-03-21T14:07:05.494303+0000","last_fullsized":"2024-03-21T14:07:05.494303+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:09.894577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:09.894577+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":165,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499380+0000","last_change":"2024-03-21T14:06:25.667933+0000","last_active":"2024-03-21T14:07:05.499380+0000","last_peered":"2024-03-21T14:07:05.499380+0000","last_clean":"2024-03-21T14:07:05.499380+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:07:05.499380+0000","last_undegraded":"2024-03-21T14:07:05.499380+0000","last_fullsized":"2024-03-21T14:07:05.499380+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:06:25.667803+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.667803+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:51.379688+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.513180+0000","last_change":"2024-03-21T14:05:18.144792+0000","last_active":"2024-03-21T14:07:05.513180+0000","last_peered":"2024-03-21T14:07:05.513180+0000","last_clean":"2024-03-21T14:07:05.513180+0000","last_became_active":"2024-03-21T14:02:35.773300+0000","last_became_peered":"2024-03-21T14:02:35.773300+0000","last_unstale":"2024-03-21T14:07:05.513180+0000","last_undegraded":"2024-03-21T14:07:05.513180+0000","last_fullsized":"2024-03-21T14:07:05.513180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:18.144307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.144307+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497748+0000","last_change":"2024-03-21T14:05:35.356518+0000","last_active":"2024-03-21T14:07:05.497748+0000","last_peered":"2024-03-21T14:07:05.497748+0000","last_clean":"2024-03-21T14:07:05.497748+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:07:05.497748+0000","last_undegraded":"2024-03-21T14:07:05.497748+0000","last_fullsized":"2024-03-21T14:07:05.497748+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.356273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.356273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.513089+0000","last_change":"2024-03-21T14:05:16.155310+0000","last_active":"2024-03-21T14:07:05.513089+0000","last_peered":"2024-03-21T14:07:05.513089+0000","last_clean":"2024-03-21T14:07:05.513089+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:07:05.513089+0000","last_undegraded":"2024-03-21T14:07:05.513089+0000","last_fullsized":"2024-03-21T14:07:05.513089+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.155028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.155028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.496343+0000","last_change":"2024-03-21T14:05:28.539817+0000","last_active":"2024-03-21T14:07:05.496343+0000","last_peered":"2024-03-21T14:07:05.496343+0000","last_clean":"2024-03-21T14:07:05.496343+0000","last_became_active":"2024-03-21T14:02:33.445324+0000","last_became_peered":"2024-03-21T14:02:33.445324+0000","last_unstale":"2024-03-21T14:07:05.496343+0000","last_undegraded":"2024-03-21T14:07:05.496343+0000","last_fullsized":"2024-03-21T14:07:05.496343+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.539674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.539674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296785+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"72'11","reported_seq":164,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.494298+0000","last_change":"2024-03-21T14:05:12.953038+0000","last_active":"2024-03-21T14:07:05.494298+0000","last_peered":"2024-03-21T14:07:05.494298+0000","last_clean":"2024-03-21T14:07:05.494298+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:07:05.494298+0000","last_undegraded":"2024-03-21T14:07:05.494298+0000","last_fullsized":"2024-03-21T14:07:05.494298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:12.952946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.952946+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506544+0000","last_change":"2024-03-21T14:05:27.971209+0000","last_active":"2024-03-21T14:07:05.506544+0000","last_peered":"2024-03-21T14:07:05.506544+0000","last_clean":"2024-03-21T14:07:05.506544+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:07:05.506544+0000","last_undegraded":"2024-03-21T14:07:05.506544+0000","last_fullsized":"2024-03-21T14:07:05.506544+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.971081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.971081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.491329+0000","last_change":"2024-03-21T14:05:35.015627+0000","last_active":"2024-03-21T14:07:05.491329+0000","last_peered":"2024-03-21T14:07:05.491329+0000","last_clean":"2024-03-21T14:07:05.491329+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:07:05.491329+0000","last_undegraded":"2024-03-21T14:07:05.491329+0000","last_fullsized":"2024-03-21T14:07:05.491329+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:35.015517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:35.015517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885614+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505776+0000","last_change":"2024-03-21T14:05:31.027870+0000","last_active":"2024-03-21T14:07:05.505776+0000","last_peered":"2024-03-21T14:07:05.505776+0000","last_clean":"2024-03-21T14:07:05.505776+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:07:05.505776+0000","last_undegraded":"2024-03-21T14:07:05.505776+0000","last_fullsized":"2024-03-21T14:07:05.505776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:31.027754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.027754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":150,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.496434+0000","last_change":"2024-03-21T14:05:36.765565+0000","last_active":"2024-03-21T14:07:05.496434+0000","last_peered":"2024-03-21T14:07:05.496434+0000","last_clean":"2024-03-21T14:07:05.496434+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:07:05.496434+0000","last_undegraded":"2024-03-21T14:07:05.496434+0000","last_fullsized":"2024-03-21T14:07:05.496434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:05:36.765228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.765228+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296873+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498325+0000","last_change":"2024-03-21T14:06:27.194477+0000","last_active":"2024-03-21T14:07:05.498325+0000","last_peered":"2024-03-21T14:07:05.498325+0000","last_clean":"2024-03-21T14:07:05.498325+0000","last_became_active":"2024-03-21T14:02:39.516661+0000","last_became_peered":"2024-03-21T14:02:39.516661+0000","last_unstale":"2024-03-21T14:07:05.498325+0000","last_undegraded":"2024-03-21T14:07:05.498325+0000","last_fullsized":"2024-03-21T14:07:05.498325+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.194189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.194189+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:57.100324+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506783+0000","last_change":"2024-03-21T14:05:07.887048+0000","last_active":"2024-03-21T14:07:05.506783+0000","last_peered":"2024-03-21T14:07:05.506783+0000","last_clean":"2024-03-21T14:07:05.506783+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:07:05.506783+0000","last_undegraded":"2024-03-21T14:07:05.506783+0000","last_fullsized":"2024-03-21T14:07:05.506783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:07.886924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:07.886924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":154,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498496+0000","last_change":"2024-03-21T14:05:17.362797+0000","last_active":"2024-03-21T14:07:05.498496+0000","last_peered":"2024-03-21T14:07:05.498496+0000","last_clean":"2024-03-21T14:07:05.498496+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:07:05.498496+0000","last_undegraded":"2024-03-21T14:07:05.498496+0000","last_fullsized":"2024-03-21T14:07:05.498496+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:17.362665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:17.362665+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313663+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":163,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497146+0000","last_change":"2024-03-21T14:05:38.851695+0000","last_active":"2024-03-21T14:07:05.497146+0000","last_peered":"2024-03-21T14:07:05.497146+0000","last_clean":"2024-03-21T14:07:05.497146+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:07:05.497146+0000","last_undegraded":"2024-03-21T14:07:05.497146+0000","last_fullsized":"2024-03-21T14:07:05.497146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.849289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.849289+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:07.242311+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502312+0000","last_change":"2024-03-21T14:06:27.935782+0000","last_active":"2024-03-21T14:07:05.502312+0000","last_peered":"2024-03-21T14:07:05.502312+0000","last_clean":"2024-03-21T14:07:05.502312+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:07:05.502312+0000","last_undegraded":"2024-03-21T14:07:05.502312+0000","last_fullsized":"2024-03-21T14:07:05.502312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.935658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.935658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.122257+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497942+0000","last_change":"2024-03-21T14:06:29.146817+0000","last_active":"2024-03-21T14:07:05.497942+0000","last_peered":"2024-03-21T14:07:05.497942+0000","last_clean":"2024-03-21T14:07:05.497942+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:07:05.497942+0000","last_undegraded":"2024-03-21T14:07:05.497942+0000","last_fullsized":"2024-03-21T14:07:05.497942+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:06:29.146695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.757841+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.507139+0000","last_change":"2024-03-21T14:06:28.067288+0000","last_active":"2024-03-21T14:07:05.507139+0000","last_peered":"2024-03-21T14:07:05.507139+0000","last_clean":"2024-03-21T14:07:05.507139+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:07:05.507139+0000","last_undegraded":"2024-03-21T14:07:05.507139+0000","last_fullsized":"2024-03-21T14:07:05.507139+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.067196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.067196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.480869+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":168,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499761+0000","last_change":"2024-03-21T14:06:24.661616+0000","last_active":"2024-03-21T14:07:05.499761+0000","last_peered":"2024-03-21T14:07:05.499761+0000","last_clean":"2024-03-21T14:07:05.499761+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:07:05.499761+0000","last_undegraded":"2024-03-21T14:07:05.499761+0000","last_fullsized":"2024-03-21T14:07:05.499761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:24.661482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:24.661482+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.597062+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.496291+0000","last_change":"2024-03-21T14:04:27.572389+0000","last_active":"2024-03-21T14:07:05.496291+0000","last_peered":"2024-03-21T14:07:05.496291+0000","last_clean":"2024-03-21T14:07:05.496291+0000","last_became_active":"2024-03-21T14:02:39.471585+0000","last_became_peered":"2024-03-21T14:02:39.471585+0000","last_unstale":"2024-03-21T14:07:05.496291+0000","last_undegraded":"2024-03-21T14:07:05.496291+0000","last_fullsized":"2024-03-21T14:07:05.496291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.572309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.572309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499535+0000","last_change":"2024-03-21T14:05:38.341843+0000","last_active":"2024-03-21T14:07:05.499535+0000","last_peered":"2024-03-21T14:07:05.499535+0000","last_clean":"2024-03-21T14:07:05.499535+0000","last_became_active":"2024-03-21T14:02:37.455013+0000","last_became_peered":"2024-03-21T14:02:37.455013+0000","last_unstale":"2024-03-21T14:07:05.499535+0000","last_undegraded":"2024-03-21T14:07:05.499535+0000","last_fullsized":"2024-03-21T14:07:05.499535+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.341705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.341705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500927+0000","last_change":"2024-03-21T14:05:36.278697+0000","last_active":"2024-03-21T14:07:05.500927+0000","last_peered":"2024-03-21T14:07:05.500927+0000","last_clean":"2024-03-21T14:07:05.500927+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:07:05.500927+0000","last_undegraded":"2024-03-21T14:07:05.500927+0000","last_fullsized":"2024-03-21T14:07:05.500927+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.278592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.278592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"72'11","reported_seq":181,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501699+0000","last_change":"2024-03-21T14:05:38.294588+0000","last_active":"2024-03-21T14:07:05.501699+0000","last_peered":"2024-03-21T14:07:05.501699+0000","last_clean":"2024-03-21T14:07:05.501699+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:07:05.501699+0000","last_undegraded":"2024-03-21T14:07:05.501699+0000","last_fullsized":"2024-03-21T14:07:05.501699+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:05:38.294441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.294441+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004473+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505948+0000","last_change":"2024-03-21T14:05:38.046118+0000","last_active":"2024-03-21T14:07:05.505948+0000","last_peered":"2024-03-21T14:07:05.505948+0000","last_clean":"2024-03-21T14:07:05.505948+0000","last_became_active":"2024-03-21T14:02:39.472672+0000","last_became_peered":"2024-03-21T14:02:39.472672+0000","last_unstale":"2024-03-21T14:07:05.505948+0000","last_undegraded":"2024-03-21T14:07:05.505948+0000","last_fullsized":"2024-03-21T14:07:05.505948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:38.045998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:38.045998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.153632+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":211,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498368+0000","last_change":"2024-03-21T14:06:36.361002+0000","last_active":"2024-03-21T14:07:05.498368+0000","last_peered":"2024-03-21T14:07:05.498368+0000","last_clean":"2024-03-21T14:07:05.498368+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:07:05.498368+0000","last_undegraded":"2024-03-21T14:07:05.498368+0000","last_fullsized":"2024-03-21T14:07:05.498368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:36.360920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.360920+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:53.901675+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506542+0000","last_change":"2024-03-21T14:06:32.074638+0000","last_active":"2024-03-21T14:07:05.506542+0000","last_peered":"2024-03-21T14:07:05.506542+0000","last_clean":"2024-03-21T14:07:05.506542+0000","last_became_active":"2024-03-21T14:02:33.446594+0000","last_became_peered":"2024-03-21T14:02:33.446594+0000","last_unstale":"2024-03-21T14:07:05.506542+0000","last_undegraded":"2024-03-21T14:07:05.506542+0000","last_fullsized":"2024-03-21T14:07:05.506542+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:32.074556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:06:32.074556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:32.791813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":151,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499343+0000","last_change":"2024-03-21T14:05:14.713009+0000","last_active":"2024-03-21T14:07:05.499343+0000","last_peered":"2024-03-21T14:07:05.499343+0000","last_clean":"2024-03-21T14:07:05.499343+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:07:05.499343+0000","last_undegraded":"2024-03-21T14:07:05.499343+0000","last_fullsized":"2024-03-21T14:07:05.499343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.712091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.712091+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296760+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501864+0000","last_change":"2024-03-21T14:06:36.093312+0000","last_active":"2024-03-21T14:07:05.501864+0000","last_peered":"2024-03-21T14:07:05.501864+0000","last_clean":"2024-03-21T14:07:05.501864+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:07:05.501864+0000","last_undegraded":"2024-03-21T14:07:05.501864+0000","last_fullsized":"2024-03-21T14:07:05.501864+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:36.093213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.093213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:56.600582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"72'8","reported_seq":203,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499511+0000","last_change":"2024-03-21T14:06:20.079291+0000","last_active":"2024-03-21T14:07:05.499511+0000","last_peered":"2024-03-21T14:07:05.499511+0000","last_clean":"2024-03-21T14:07:05.499511+0000","last_became_active":"2024-03-21T14:02:37.461462+0000","last_became_peered":"2024-03-21T14:02:37.461462+0000","last_unstale":"2024-03-21T14:07:05.499511+0000","last_undegraded":"2024-03-21T14:07:05.499511+0000","last_fullsized":"2024-03-21T14:07:05.499511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:20.079142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.079142+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.032560+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502837+0000","last_change":"2024-03-21T14:06:20.884332+0000","last_active":"2024-03-21T14:07:05.502837+0000","last_peered":"2024-03-21T14:07:05.502837+0000","last_clean":"2024-03-21T14:07:05.502837+0000","last_became_active":"2024-03-21T14:02:33.439505+0000","last_became_peered":"2024-03-21T14:02:33.439505+0000","last_unstale":"2024-03-21T14:07:05.502837+0000","last_undegraded":"2024-03-21T14:07:05.502837+0000","last_fullsized":"2024-03-21T14:07:05.502837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:20.884210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.884210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.916333+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498562+0000","last_change":"2024-03-21T14:06:20.431407+0000","last_active":"2024-03-21T14:07:05.498562+0000","last_peered":"2024-03-21T14:07:05.498562+0000","last_clean":"2024-03-21T14:07:05.498562+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:07:05.498562+0000","last_undegraded":"2024-03-21T14:07:05.498562+0000","last_fullsized":"2024-03-21T14:07:05.498562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:20.431279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:20.431279+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.061041+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501441+0000","last_change":"2024-03-21T14:05:29.391237+0000","last_active":"2024-03-21T14:07:05.501441+0000","last_peered":"2024-03-21T14:07:05.501441+0000","last_clean":"2024-03-21T14:07:05.501441+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:07:05.501441+0000","last_undegraded":"2024-03-21T14:07:05.501441+0000","last_fullsized":"2024-03-21T14:07:05.501441+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.391091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.391091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.006503+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.504159+0000","last_change":"2024-03-21T14:06:40.314652+0000","last_active":"2024-03-21T14:07:05.504159+0000","last_peered":"2024-03-21T14:07:05.504159+0000","last_clean":"2024-03-21T14:07:05.504159+0000","last_became_active":"2024-03-21T14:02:37.456542+0000","last_became_peered":"2024-03-21T14:02:37.456542+0000","last_unstale":"2024-03-21T14:07:05.504159+0000","last_undegraded":"2024-03-21T14:07:05.504159+0000","last_fullsized":"2024-03-21T14:07:05.504159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:40.314528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:40.314528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:48.007862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506459+0000","last_change":"2024-03-21T14:05:16.811057+0000","last_active":"2024-03-21T14:07:05.506459+0000","last_peered":"2024-03-21T14:07:05.506459+0000","last_clean":"2024-03-21T14:07:05.506459+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:07:05.506459+0000","last_undegraded":"2024-03-21T14:07:05.506459+0000","last_fullsized":"2024-03-21T14:07:05.506459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.810928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.810928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.512925+0000","last_change":"2024-03-21T14:06:16.830628+0000","last_active":"2024-03-21T14:07:05.512925+0000","last_peered":"2024-03-21T14:07:05.512925+0000","last_clean":"2024-03-21T14:07:05.512925+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:07:05.512925+0000","last_undegraded":"2024-03-21T14:07:05.512925+0000","last_fullsized":"2024-03-21T14:07:05.512925+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:16.830503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.830503+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:17.447722+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.507137+0000","last_change":"2024-03-21T14:05:15.857954+0000","last_active":"2024-03-21T14:07:05.507137+0000","last_peered":"2024-03-21T14:07:05.507137+0000","last_clean":"2024-03-21T14:07:05.507137+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:07:05.507137+0000","last_undegraded":"2024-03-21T14:07:05.507137+0000","last_fullsized":"2024-03-21T14:07:05.507137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:15.857799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.857799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498556+0000","last_change":"2024-03-21T14:05:28.362497+0000","last_active":"2024-03-21T14:07:05.498556+0000","last_peered":"2024-03-21T14:07:05.498556+0000","last_clean":"2024-03-21T14:07:05.498556+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:07:05.498556+0000","last_undegraded":"2024-03-21T14:07:05.498556+0000","last_fullsized":"2024-03-21T14:07:05.498556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:28.362424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:28.362424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313783+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":148,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.490596+0000","last_change":"2024-03-21T14:05:08.883023+0000","last_active":"2024-03-21T14:07:05.490596+0000","last_peered":"2024-03-21T14:07:05.490596+0000","last_clean":"2024-03-21T14:07:05.490596+0000","last_became_active":"2024-03-21T14:02:33.443676+0000","last_became_peered":"2024-03-21T14:02:33.443676+0000","last_unstale":"2024-03-21T14:07:05.490596+0000","last_undegraded":"2024-03-21T14:07:05.490596+0000","last_fullsized":"2024-03-21T14:07:05.490596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:08.882903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:08.882903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":145,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.504003+0000","last_change":"2024-03-21T14:05:15.725635+0000","last_active":"2024-03-21T14:07:05.504003+0000","last_peered":"2024-03-21T14:07:05.504003+0000","last_clean":"2024-03-21T14:07:05.504003+0000","last_became_active":"2024-03-21T14:02:35.773287+0000","last_became_peered":"2024-03-21T14:02:35.773287+0000","last_unstale":"2024-03-21T14:07:05.504003+0000","last_undegraded":"2024-03-21T14:07:05.504003+0000","last_fullsized":"2024-03-21T14:07:05.504003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:05:15.725275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:15.725275+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296629+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"125'10720","reported_seq":10881,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.344767+0000","last_change":"2024-03-21T14:06:25.814602+0000","last_active":"2024-03-21T14:07:06.344767+0000","last_peered":"2024-03-21T14:07:06.344767+0000","last_clean":"2024-03-21T14:07:06.344767+0000","last_became_active":"2024-03-21T14:04:28.368403+0000","last_became_peered":"2024-03-21T14:04:28.368403+0000","last_unstale":"2024-03-21T14:07:06.344767+0000","last_undegraded":"2024-03-21T14:07:06.344767+0000","last_fullsized":"2024-03-21T14:07:06.344767+0000","mapping_epoch":89,"log_start":"106'5700","ondisk_log_start":"106'5700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8546","last_scrub_stamp":"2024-03-21T14:06:25.814421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.814421+0000","objects_scrubbed":6773,"log_size":5020,"log_dups_size":0,"ondisk_log_size":5020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.221261+0000","scrub_duration":2159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296747008,"num_objects":4528,"num_object_clones":0,"num_object_copies":13584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10720,"num_write_kb":487936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498201+0000","last_change":"2024-03-21T14:06:16.410140+0000","last_active":"2024-03-21T14:07:05.498201+0000","last_peered":"2024-03-21T14:07:05.498201+0000","last_clean":"2024-03-21T14:07:05.498201+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:07:05.498201+0000","last_undegraded":"2024-03-21T14:07:05.498201+0000","last_fullsized":"2024-03-21T14:07:05.498201+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:16.409991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.409991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.498428+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":161,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.509324+0000","last_change":"2024-03-21T14:06:26.071562+0000","last_active":"2024-03-21T14:07:05.509324+0000","last_peered":"2024-03-21T14:07:05.509324+0000","last_clean":"2024-03-21T14:07:05.509324+0000","last_became_active":"2024-03-21T14:02:33.448127+0000","last_became_peered":"2024-03-21T14:02:33.448127+0000","last_unstale":"2024-03-21T14:07:05.509324+0000","last_undegraded":"2024-03-21T14:07:05.509324+0000","last_fullsized":"2024-03-21T14:07:05.509324+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:06:26.071314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:26.071314+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.081185+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":170,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497965+0000","last_change":"2024-03-21T14:05:40.337551+0000","last_active":"2024-03-21T14:07:05.497965+0000","last_peered":"2024-03-21T14:07:05.497965+0000","last_clean":"2024-03-21T14:07:05.497965+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:07:05.497965+0000","last_undegraded":"2024-03-21T14:07:05.497965+0000","last_fullsized":"2024-03-21T14:07:05.497965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:40.337278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:40.337278+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313816+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506561+0000","last_change":"2024-03-21T14:06:35.127130+0000","last_active":"2024-03-21T14:07:05.506561+0000","last_peered":"2024-03-21T14:07:05.506561+0000","last_clean":"2024-03-21T14:07:05.506561+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:07:05.506561+0000","last_undegraded":"2024-03-21T14:07:05.506561+0000","last_fullsized":"2024-03-21T14:07:05.506561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:35.126990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:35.126990+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:00.060225+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"125'10651","reported_seq":10802,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.480533+0000","last_change":"2024-03-21T14:06:38.868454+0000","last_active":"2024-03-21T14:07:06.480533+0000","last_peered":"2024-03-21T14:07:06.480533+0000","last_clean":"2024-03-21T14:07:06.480533+0000","last_became_active":"2024-03-21T14:04:28.366208+0000","last_became_peered":"2024-03-21T14:04:28.366208+0000","last_unstale":"2024-03-21T14:07:06.480533+0000","last_undegraded":"2024-03-21T14:07:06.480533+0000","last_fullsized":"2024-03-21T14:07:06.480533+0000","mapping_epoch":89,"log_start":"108'6000","ondisk_log_start":"108'6000","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'9182","last_scrub_stamp":"2024-03-21T14:06:38.868297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:38.868297+0000","objects_scrubbed":6021,"log_size":4651,"log_dups_size":0,"ondisk_log_size":4651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:57.781353+0000","scrub_duration":1537,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296943616,"num_objects":4531,"num_object_clones":0,"num_object_copies":13593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10651,"num_write_kb":485824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498121+0000","last_change":"2024-03-21T14:05:23.704653+0000","last_active":"2024-03-21T14:07:05.498121+0000","last_peered":"2024-03-21T14:07:05.498121+0000","last_clean":"2024-03-21T14:07:05.498121+0000","last_became_active":"2024-03-21T14:02:39.471719+0000","last_became_peered":"2024-03-21T14:02:39.471719+0000","last_unstale":"2024-03-21T14:07:05.498121+0000","last_undegraded":"2024-03-21T14:07:05.498121+0000","last_fullsized":"2024-03-21T14:07:05.498121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.703365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.703365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296654+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.507331+0000","last_change":"2024-03-21T14:05:19.908957+0000","last_active":"2024-03-21T14:07:05.507331+0000","last_peered":"2024-03-21T14:07:05.507331+0000","last_clean":"2024-03-21T14:07:05.507331+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:07:05.507331+0000","last_undegraded":"2024-03-21T14:07:05.507331+0000","last_fullsized":"2024-03-21T14:07:05.507331+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.908814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.908814+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502343+0000","last_change":"2024-03-21T14:05:12.169866+0000","last_active":"2024-03-21T14:07:05.502343+0000","last_peered":"2024-03-21T14:07:05.502343+0000","last_clean":"2024-03-21T14:07:05.502343+0000","last_became_active":"2024-03-21T14:02:35.448438+0000","last_became_peered":"2024-03-21T14:02:35.448438+0000","last_unstale":"2024-03-21T14:07:05.502343+0000","last_undegraded":"2024-03-21T14:07:05.502343+0000","last_fullsized":"2024-03-21T14:07:05.502343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:05:12.169735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.169735+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505406+0000","last_change":"2024-03-21T14:06:34.915366+0000","last_active":"2024-03-21T14:07:05.505406+0000","last_peered":"2024-03-21T14:07:05.505406+0000","last_clean":"2024-03-21T14:07:05.505406+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:07:05.505406+0000","last_undegraded":"2024-03-21T14:07:05.505406+0000","last_fullsized":"2024-03-21T14:07:05.505406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.915163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.915163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.070587+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"125'10525","reported_seq":10678,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.646503+0000","last_change":"2024-03-21T14:06:33.954281+0000","last_active":"2024-03-21T14:07:06.646503+0000","last_peered":"2024-03-21T14:07:06.646503+0000","last_clean":"2024-03-21T14:07:06.646503+0000","last_became_active":"2024-03-21T14:04:28.367553+0000","last_became_peered":"2024-03-21T14:04:28.367553+0000","last_unstale":"2024-03-21T14:07:06.646503+0000","last_undegraded":"2024-03-21T14:07:06.646503+0000","last_fullsized":"2024-03-21T14:07:06.646503+0000","mapping_epoch":89,"log_start":"102'4900","ondisk_log_start":"102'4900","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'8984","last_scrub_stamp":"2024-03-21T14:06:33.954062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.954062+0000","objects_scrubbed":6105,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.888906+0000","scrub_duration":1976,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295501824,"num_objects":4509,"num_object_clones":0,"num_object_copies":13527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4509,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10525,"num_write_kb":481088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506233+0000","last_change":"2024-03-21T14:06:25.080381+0000","last_active":"2024-03-21T14:07:05.506233+0000","last_peered":"2024-03-21T14:07:05.506233+0000","last_clean":"2024-03-21T14:07:05.506233+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:07:05.506233+0000","last_undegraded":"2024-03-21T14:07:05.506233+0000","last_fullsized":"2024-03-21T14:07:05.506233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:25.080145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.080145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.576368+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.504184+0000","last_change":"2024-03-21T14:05:20.225369+0000","last_active":"2024-03-21T14:07:05.504184+0000","last_peered":"2024-03-21T14:07:05.504184+0000","last_clean":"2024-03-21T14:07:05.504184+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:07:05.504184+0000","last_undegraded":"2024-03-21T14:07:05.504184+0000","last_fullsized":"2024-03-21T14:07:05.504184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:20.225234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.225234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.006477+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.507286+0000","last_change":"2024-03-21T14:06:19.057440+0000","last_active":"2024-03-21T14:07:05.507286+0000","last_peered":"2024-03-21T14:07:05.507286+0000","last_clean":"2024-03-21T14:07:05.507286+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:07:05.507286+0000","last_undegraded":"2024-03-21T14:07:05.507286+0000","last_fullsized":"2024-03-21T14:07:05.507286+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:19.057319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:19.057319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.818134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"72'19","reported_seq":176,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.490937+0000","last_change":"2024-03-21T14:06:26.007667+0000","last_active":"2024-03-21T14:07:05.490937+0000","last_peered":"2024-03-21T14:07:05.490937+0000","last_clean":"2024-03-21T14:07:05.490937+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:07:05.490937+0000","last_undegraded":"2024-03-21T14:07:05.490937+0000","last_fullsized":"2024-03-21T14:07:05.490937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:06:26.007559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:26.007559+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.730733+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"125'10670","reported_seq":10832,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.446076+0000","last_change":"2024-03-21T14:06:21.462241+0000","last_active":"2024-03-21T14:07:06.446076+0000","last_peered":"2024-03-21T14:07:06.446076+0000","last_clean":"2024-03-21T14:07:06.446076+0000","last_became_active":"2024-03-21T14:04:28.360990+0000","last_became_peered":"2024-03-21T14:04:28.360990+0000","last_unstale":"2024-03-21T14:07:06.446076+0000","last_undegraded":"2024-03-21T14:07:06.446076+0000","last_fullsized":"2024-03-21T14:07:06.446076+0000","mapping_epoch":89,"log_start":"99'3800","ondisk_log_start":"99'3800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8224","last_scrub_stamp":"2024-03-21T14:06:21.462077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.462077+0000","objects_scrubbed":6973,"log_size":6870,"log_dups_size":0,"ondisk_log_size":6870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.847162+0000","scrub_duration":2328,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292814848,"num_objects":4468,"num_object_clones":0,"num_object_copies":13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10670,"num_write_kb":484416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505599+0000","last_change":"2024-03-21T14:05:34.000829+0000","last_active":"2024-03-21T14:07:05.505599+0000","last_peered":"2024-03-21T14:07:05.505599+0000","last_clean":"2024-03-21T14:07:05.505599+0000","last_became_active":"2024-03-21T14:02:39.465642+0000","last_became_peered":"2024-03-21T14:02:39.465642+0000","last_unstale":"2024-03-21T14:07:05.505599+0000","last_undegraded":"2024-03-21T14:07:05.505599+0000","last_fullsized":"2024-03-21T14:07:05.505599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.000689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502323+0000","last_change":"2024-03-21T14:06:30.047729+0000","last_active":"2024-03-21T14:07:05.502323+0000","last_peered":"2024-03-21T14:07:05.502323+0000","last_clean":"2024-03-21T14:07:05.502323+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:07:05.502323+0000","last_undegraded":"2024-03-21T14:07:05.502323+0000","last_fullsized":"2024-03-21T14:07:05.502323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:30.047613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:06:30.047613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.635296+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498339+0000","last_change":"2024-03-21T14:05:23.381853+0000","last_active":"2024-03-21T14:07:05.498339+0000","last_peered":"2024-03-21T14:07:05.498339+0000","last_clean":"2024-03-21T14:07:05.498339+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:07:05.498339+0000","last_undegraded":"2024-03-21T14:07:05.498339+0000","last_fullsized":"2024-03-21T14:07:05.498339+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:23.381726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:23.381726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":177,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503337+0000","last_change":"2024-03-21T14:06:15.846648+0000","last_active":"2024-03-21T14:07:05.503337+0000","last_peered":"2024-03-21T14:07:05.503337+0000","last_clean":"2024-03-21T14:07:05.503337+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:07:05.503337+0000","last_undegraded":"2024-03-21T14:07:05.503337+0000","last_fullsized":"2024-03-21T14:07:05.503337+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:06:15.846504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.846504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.520598+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"125'12974","reported_seq":13165,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.282182+0000","last_change":"2024-03-21T14:06:51.980257+0000","last_active":"2024-03-21T14:07:06.282182+0000","last_peered":"2024-03-21T14:07:06.282182+0000","last_clean":"2024-03-21T14:07:06.282182+0000","last_became_active":"2024-03-21T14:04:28.369020+0000","last_became_peered":"2024-03-21T14:04:28.369020+0000","last_unstale":"2024-03-21T14:07:06.282182+0000","last_undegraded":"2024-03-21T14:07:06.282182+0000","last_fullsized":"2024-03-21T14:07:06.282182+0000","mapping_epoch":89,"log_start":"109'6700","ondisk_log_start":"109'6700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":2599,"log_size":6274,"log_dups_size":0,"ondisk_log_size":6274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10393,"num_write_kb":475264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500167+0000","last_change":"2024-03-21T14:05:16.720441+0000","last_active":"2024-03-21T14:07:05.500167+0000","last_peered":"2024-03-21T14:07:05.500167+0000","last_clean":"2024-03-21T14:07:05.500167+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:07:05.500167+0000","last_undegraded":"2024-03-21T14:07:05.500167+0000","last_fullsized":"2024-03-21T14:07:05.500167+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:16.720331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:16.720331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":160,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498025+0000","last_change":"2024-03-21T14:05:11.492743+0000","last_active":"2024-03-21T14:07:05.498025+0000","last_peered":"2024-03-21T14:07:05.498025+0000","last_clean":"2024-03-21T14:07:05.498025+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:07:05.498025+0000","last_undegraded":"2024-03-21T14:07:05.498025+0000","last_fullsized":"2024-03-21T14:07:05.498025+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:05:11.492609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:05:11.492609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313622+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501396+0000","last_change":"2024-03-21T14:05:18.141165+0000","last_active":"2024-03-21T14:07:05.501396+0000","last_peered":"2024-03-21T14:07:05.501396+0000","last_clean":"2024-03-21T14:07:05.501396+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:07:05.501396+0000","last_undegraded":"2024-03-21T14:07:05.501396+0000","last_fullsized":"2024-03-21T14:07:05.501396+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:18.140955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:18.140955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"75'29","reported_seq":201,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498148+0000","last_change":"2024-03-21T14:05:12.482027+0000","last_active":"2024-03-21T14:07:05.498148+0000","last_peered":"2024-03-21T14:07:05.498148+0000","last_clean":"2024-03-21T14:07:05.498148+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:07:05.498148+0000","last_undegraded":"2024-03-21T14:07:05.498148+0000","last_fullsized":"2024-03-21T14:07:05.498148+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'29","last_scrub_stamp":"2024-03-21T14:05:12.481864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:12.481864+0000","objects_scrubbed":10,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313754+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505517+0000","last_change":"2024-03-21T14:06:14.840624+0000","last_active":"2024-03-21T14:07:05.505517+0000","last_peered":"2024-03-21T14:07:05.505517+0000","last_clean":"2024-03-21T14:07:05.505517+0000","last_became_active":"2024-03-21T14:02:37.456664+0000","last_became_peered":"2024-03-21T14:02:37.456664+0000","last_unstale":"2024-03-21T14:07:05.505517+0000","last_undegraded":"2024-03-21T14:07:05.505517+0000","last_fullsized":"2024-03-21T14:07:05.505517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:14.840509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:14.840509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.244776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"125'10816","reported_seq":10979,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.621687+0000","last_change":"2024-03-21T14:06:21.025026+0000","last_active":"2024-03-21T14:07:06.621687+0000","last_peered":"2024-03-21T14:07:06.621687+0000","last_clean":"2024-03-21T14:07:06.621687+0000","last_became_active":"2024-03-21T14:04:28.367387+0000","last_became_peered":"2024-03-21T14:04:28.367387+0000","last_unstale":"2024-03-21T14:07:06.621687+0000","last_undegraded":"2024-03-21T14:07:06.621687+0000","last_fullsized":"2024-03-21T14:07:06.621687+0000","mapping_epoch":89,"log_start":"103'5200","ondisk_log_start":"103'5200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8368","last_scrub_stamp":"2024-03-21T14:06:21.024886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.024886+0000","objects_scrubbed":7201,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:33.840880+0000","scrub_duration":3116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304742400,"num_objects":4650,"num_object_clones":0,"num_object_copies":13950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10816,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.507314+0000","last_change":"2024-03-21T14:06:11.955196+0000","last_active":"2024-03-21T14:07:05.507314+0000","last_peered":"2024-03-21T14:07:05.507314+0000","last_clean":"2024-03-21T14:07:05.507314+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:07:05.507314+0000","last_undegraded":"2024-03-21T14:07:05.507314+0000","last_fullsized":"2024-03-21T14:07:05.507314+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:11.955082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:11.955082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:41.410006+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498798+0000","last_change":"2024-03-21T14:05:22.226038+0000","last_active":"2024-03-21T14:07:05.498798+0000","last_peered":"2024-03-21T14:07:05.498798+0000","last_clean":"2024-03-21T14:07:05.498798+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:07:05.498798+0000","last_undegraded":"2024-03-21T14:07:05.498798+0000","last_fullsized":"2024-03-21T14:07:05.498798+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:22.225815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:05:22.225815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500906+0000","last_change":"2024-03-21T14:06:12.171064+0000","last_active":"2024-03-21T14:07:05.500906+0000","last_peered":"2024-03-21T14:07:05.500906+0000","last_clean":"2024-03-21T14:07:05.500906+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:07:05.500906+0000","last_undegraded":"2024-03-21T14:07:05.500906+0000","last_fullsized":"2024-03-21T14:07:05.500906+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:12.170919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:12.170919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:35.011481+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":185,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498113+0000","last_change":"2024-03-21T14:06:31.724307+0000","last_active":"2024-03-21T14:07:05.498113+0000","last_peered":"2024-03-21T14:07:05.498113+0000","last_clean":"2024-03-21T14:07:05.498113+0000","last_became_active":"2024-03-21T14:02:35.454955+0000","last_became_peered":"2024-03-21T14:02:35.454955+0000","last_unstale":"2024-03-21T14:07:05.498113+0000","last_undegraded":"2024-03-21T14:07:05.498113+0000","last_fullsized":"2024-03-21T14:07:05.498113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:06:31.724171+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:06:31.724171+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.792472+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501039+0000","last_change":"2024-03-21T14:06:32.979972+0000","last_active":"2024-03-21T14:07:05.501039+0000","last_peered":"2024-03-21T14:07:05.501039+0000","last_clean":"2024-03-21T14:07:05.501039+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:07:05.501039+0000","last_undegraded":"2024-03-21T14:07:05.501039+0000","last_fullsized":"2024-03-21T14:07:05.501039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:32.979871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:32.979871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:52.805427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"125'13654","reported_seq":13867,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.616969+0000","last_change":"2024-03-21T14:06:52.010201+0000","last_active":"2024-03-21T14:07:06.616969+0000","last_peered":"2024-03-21T14:07:06.616969+0000","last_clean":"2024-03-21T14:07:06.616969+0000","last_became_active":"2024-03-21T14:04:28.363923+0000","last_became_peered":"2024-03-21T14:04:28.363923+0000","last_unstale":"2024-03-21T14:07:06.616969+0000","last_undegraded":"2024-03-21T14:07:06.616969+0000","last_fullsized":"2024-03-21T14:07:06.616969+0000","mapping_epoch":89,"log_start":"117'8000","ondisk_log_start":"117'8000","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":3123,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298450944,"num_objects":4554,"num_object_clones":0,"num_object_copies":13662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4554,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10602,"num_write_kb":484992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498469+0000","last_change":"2024-03-21T14:06:15.375779+0000","last_active":"2024-03-21T14:07:05.498469+0000","last_peered":"2024-03-21T14:07:05.498469+0000","last_clean":"2024-03-21T14:07:05.498469+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:07:05.498469+0000","last_undegraded":"2024-03-21T14:07:05.498469+0000","last_fullsized":"2024-03-21T14:07:05.498469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:15.375667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:15.375667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:16.848357+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":403,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506350+0000","last_change":"2024-03-21T14:05:31.964155+0000","last_active":"2024-03-21T14:07:05.506350+0000","last_peered":"2024-03-21T14:07:05.506350+0000","last_clean":"2024-03-21T14:07:05.506350+0000","last_became_active":"2024-03-21T14:02:23.337256+0000","last_became_peered":"2024-03-21T14:02:23.337256+0000","last_unstale":"2024-03-21T14:07:05.506350+0000","last_undegraded":"2024-03-21T14:07:05.506350+0000","last_fullsized":"2024-03-21T14:07:05.506350+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-03-21T14:05:31.964052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:00:13.313784+0000","last_clean_scrub_stamp":"2024-03-21T14:05:31.964052+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497293+0000","last_change":"2024-03-21T14:06:28.186432+0000","last_active":"2024-03-21T14:07:05.497293+0000","last_peered":"2024-03-21T14:07:05.497293+0000","last_clean":"2024-03-21T14:07:05.497293+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:07:05.497293+0000","last_undegraded":"2024-03-21T14:07:05.497293+0000","last_fullsized":"2024-03-21T14:07:05.497293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.186306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.186306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:32.719009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":222,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.491493+0000","last_change":"2024-03-21T14:05:25.911867+0000","last_active":"2024-03-21T14:07:05.491493+0000","last_peered":"2024-03-21T14:07:05.491493+0000","last_clean":"2024-03-21T14:07:05.491493+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:07:05.491493+0000","last_undegraded":"2024-03-21T14:07:05.491493+0000","last_fullsized":"2024-03-21T14:07:05.491493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:05:25.911729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:25.911729+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506997+0000","last_change":"2024-03-21T14:06:37.087710+0000","last_active":"2024-03-21T14:07:05.506997+0000","last_peered":"2024-03-21T14:07:05.506997+0000","last_clean":"2024-03-21T14:07:05.506997+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:07:05.506997+0000","last_undegraded":"2024-03-21T14:07:05.506997+0000","last_fullsized":"2024-03-21T14:07:05.506997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:37.087593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:37.087593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:43.851819+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"125'10464","reported_seq":10547,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.230151+0000","last_change":"2024-03-21T14:04:28.363760+0000","last_active":"2024-03-21T14:07:06.230151+0000","last_peered":"2024-03-21T14:07:06.230151+0000","last_clean":"2024-03-21T14:07:06.230151+0000","last_became_active":"2024-03-21T14:04:28.363404+0000","last_became_peered":"2024-03-21T14:04:28.363404+0000","last_unstale":"2024-03-21T14:07:06.230151+0000","last_undegraded":"2024-03-21T14:07:06.230151+0000","last_fullsized":"2024-03-21T14:07:06.230151+0000","mapping_epoch":89,"log_start":"105'5200","ondisk_log_start":"105'5200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":5264,"log_dups_size":0,"ondisk_log_size":5264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293470208,"num_objects":4478,"num_object_clones":0,"num_object_copies":13434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10464,"num_write_kb":478144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497674+0000","last_change":"2024-03-21T14:05:36.379278+0000","last_active":"2024-03-21T14:07:05.497674+0000","last_peered":"2024-03-21T14:07:05.497674+0000","last_clean":"2024-03-21T14:07:05.497674+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:07:05.497674+0000","last_undegraded":"2024-03-21T14:07:05.497674+0000","last_fullsized":"2024-03-21T14:07:05.497674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:36.376091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.376091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.505835+0000","last_change":"2024-03-21T14:06:28.961386+0000","last_active":"2024-03-21T14:07:05.505835+0000","last_peered":"2024-03-21T14:07:05.505835+0000","last_clean":"2024-03-21T14:07:05.505835+0000","last_became_active":"2024-03-21T14:02:33.446487+0000","last_became_peered":"2024-03-21T14:02:33.446487+0000","last_unstale":"2024-03-21T14:07:05.505835+0000","last_undegraded":"2024-03-21T14:07:05.505835+0000","last_fullsized":"2024-03-21T14:07:05.505835+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:28.961272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:28.961272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:46.502960+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":427,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506873+0000","last_change":"2024-03-21T14:06:13.952697+0000","last_active":"2024-03-21T14:07:05.506873+0000","last_peered":"2024-03-21T14:07:05.506873+0000","last_clean":"2024-03-21T14:07:05.506873+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:07:05.506873+0000","last_undegraded":"2024-03-21T14:07:05.506873+0000","last_fullsized":"2024-03-21T14:07:05.506873+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:06:13.952449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.952449+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:39.957684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":163,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506044+0000","last_change":"2024-03-21T14:06:18.867126+0000","last_active":"2024-03-21T14:07:05.506044+0000","last_peered":"2024-03-21T14:07:05.506044+0000","last_clean":"2024-03-21T14:07:05.506044+0000","last_became_active":"2024-03-21T14:02:35.449479+0000","last_became_peered":"2024-03-21T14:02:35.449479+0000","last_unstale":"2024-03-21T14:07:05.506044+0000","last_undegraded":"2024-03-21T14:07:05.506044+0000","last_fullsized":"2024-03-21T14:07:05.506044+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:06:18.866994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.866994+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.123054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498676+0000","last_change":"2024-03-21T14:06:27.461385+0000","last_active":"2024-03-21T14:07:05.498676+0000","last_peered":"2024-03-21T14:07:05.498676+0000","last_clean":"2024-03-21T14:07:05.498676+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:07:05.498676+0000","last_undegraded":"2024-03-21T14:07:05.498676+0000","last_fullsized":"2024-03-21T14:07:05.498676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:27.461298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.461298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.789908+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"125'10613","reported_seq":10772,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.264682+0000","last_change":"2024-03-21T14:06:25.324945+0000","last_active":"2024-03-21T14:07:06.264682+0000","last_peered":"2024-03-21T14:07:06.264682+0000","last_clean":"2024-03-21T14:07:06.264682+0000","last_became_active":"2024-03-21T14:04:28.367882+0000","last_became_peered":"2024-03-21T14:04:28.367882+0000","last_unstale":"2024-03-21T14:07:06.264682+0000","last_undegraded":"2024-03-21T14:07:06.264682+0000","last_fullsized":"2024-03-21T14:07:06.264682+0000","mapping_epoch":89,"log_start":"102'4800","ondisk_log_start":"102'4800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8391","last_scrub_stamp":"2024-03-21T14:06:25.321852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:25.321852+0000","objects_scrubbed":6732,"log_size":5813,"log_dups_size":0,"ondisk_log_size":5813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.120366+0000","scrub_duration":1849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10613,"num_write_kb":482304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498426+0000","last_change":"2024-03-21T14:05:34.327891+0000","last_active":"2024-03-21T14:07:05.498426+0000","last_peered":"2024-03-21T14:07:05.498426+0000","last_clean":"2024-03-21T14:07:05.498426+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:07:05.498426+0000","last_undegraded":"2024-03-21T14:07:05.498426+0000","last_fullsized":"2024-03-21T14:07:05.498426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:34.327068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:34.327068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497913+0000","last_change":"2024-03-21T14:05:19.237932+0000","last_active":"2024-03-21T14:07:05.497913+0000","last_peered":"2024-03-21T14:07:05.497913+0000","last_clean":"2024-03-21T14:07:05.497913+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:07:05.497913+0000","last_undegraded":"2024-03-21T14:07:05.497913+0000","last_fullsized":"2024-03-21T14:07:05.497913+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:19.237452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:19.237452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":181,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503081+0000","last_change":"2024-03-21T14:05:24.049566+0000","last_active":"2024-03-21T14:07:05.503081+0000","last_peered":"2024-03-21T14:07:05.503081+0000","last_clean":"2024-03-21T14:07:05.503081+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:07:05.503081+0000","last_undegraded":"2024-03-21T14:07:05.503081+0000","last_fullsized":"2024-03-21T14:07:05.503081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:05:24.049421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.049421+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":141,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.491041+0000","last_change":"2024-03-21T14:06:18.996323+0000","last_active":"2024-03-21T14:07:05.491041+0000","last_peered":"2024-03-21T14:07:05.491041+0000","last_clean":"2024-03-21T14:07:05.491041+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:07:05.491041+0000","last_undegraded":"2024-03-21T14:07:05.491041+0000","last_fullsized":"2024-03-21T14:07:05.491041+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.996252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.996252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:36.294909+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"125'10504","reported_seq":10657,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.257477+0000","last_change":"2024-03-21T14:06:33.292754+0000","last_active":"2024-03-21T14:07:06.257477+0000","last_peered":"2024-03-21T14:07:06.257477+0000","last_clean":"2024-03-21T14:07:06.257477+0000","last_became_active":"2024-03-21T14:04:28.369002+0000","last_became_peered":"2024-03-21T14:04:28.369002+0000","last_unstale":"2024-03-21T14:07:06.257477+0000","last_undegraded":"2024-03-21T14:07:06.257477+0000","last_fullsized":"2024-03-21T14:07:06.257477+0000","mapping_epoch":89,"log_start":"107'5700","ondisk_log_start":"107'5700","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'8884","last_scrub_stamp":"2024-03-21T14:06:33.292611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:33.292611+0000","objects_scrubbed":6207,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:58.862461+0000","scrub_duration":1119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844160,"num_objects":4560,"num_object_clones":0,"num_object_copies":13680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4560,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10504,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503495+0000","last_change":"2024-03-21T14:06:42.348065+0000","last_active":"2024-03-21T14:07:05.503495+0000","last_peered":"2024-03-21T14:07:05.503495+0000","last_clean":"2024-03-21T14:07:05.503495+0000","last_became_active":"2024-03-21T14:02:39.473074+0000","last_became_peered":"2024-03-21T14:02:39.473074+0000","last_unstale":"2024-03-21T14:07:05.503495+0000","last_undegraded":"2024-03-21T14:07:05.503495+0000","last_fullsized":"2024-03-21T14:07:05.503495+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:42.347853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:42.347853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:10.814273+0000","scrub_duration":5295,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498480+0000","last_change":"2024-03-21T14:06:18.451847+0000","last_active":"2024-03-21T14:07:05.498480+0000","last_peered":"2024-03-21T14:07:05.498480+0000","last_clean":"2024-03-21T14:07:05.498480+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:07:05.498480+0000","last_undegraded":"2024-03-21T14:07:05.498480+0000","last_fullsized":"2024-03-21T14:07:05.498480+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:18.451772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.451772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:31.610702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"72'10","reported_seq":168,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499618+0000","last_change":"2024-03-21T14:06:27.736989+0000","last_active":"2024-03-21T14:07:05.499618+0000","last_peered":"2024-03-21T14:07:05.499618+0000","last_clean":"2024-03-21T14:07:05.499618+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:07:05.499618+0000","last_undegraded":"2024-03-21T14:07:05.499618+0000","last_fullsized":"2024-03-21T14:07:05.499618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:06:27.736695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:27.736695+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:30.734373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503391+0000","last_change":"2024-03-21T14:06:13.786619+0000","last_active":"2024-03-21T14:07:05.503391+0000","last_peered":"2024-03-21T14:07:05.503391+0000","last_clean":"2024-03-21T14:07:05.503391+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:07:05.503391+0000","last_undegraded":"2024-03-21T14:07:05.503391+0000","last_fullsized":"2024-03-21T14:07:05.503391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:13.786485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:13.786485+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:40.654794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"125'10681","reported_seq":10764,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.459071+0000","last_change":"2024-03-21T14:04:28.366919+0000","last_active":"2024-03-21T14:07:06.459071+0000","last_peered":"2024-03-21T14:07:06.459071+0000","last_clean":"2024-03-21T14:07:06.459071+0000","last_became_active":"2024-03-21T14:04:28.366634+0000","last_became_peered":"2024-03-21T14:04:28.366634+0000","last_unstale":"2024-03-21T14:07:06.459071+0000","last_undegraded":"2024-03-21T14:07:06.459071+0000","last_fullsized":"2024-03-21T14:07:06.459071+0000","mapping_epoch":89,"log_start":"108'6000","ondisk_log_start":"108'6000","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":4681,"log_dups_size":0,"ondisk_log_size":4681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294846464,"num_objects":4499,"num_object_clones":0,"num_object_copies":13497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10681,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506872+0000","last_change":"2024-03-21T14:06:34.155961+0000","last_active":"2024-03-21T14:07:05.506872+0000","last_peered":"2024-03-21T14:07:05.506872+0000","last_clean":"2024-03-21T14:07:05.506872+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:07:05.506872+0000","last_undegraded":"2024-03-21T14:07:05.506872+0000","last_fullsized":"2024-03-21T14:07:05.506872+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.155840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.155840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:49.184912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":150,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.506145+0000","last_change":"2024-03-21T14:05:20.137081+0000","last_active":"2024-03-21T14:07:05.506145+0000","last_peered":"2024-03-21T14:07:05.506145+0000","last_clean":"2024-03-21T14:07:05.506145+0000","last_became_active":"2024-03-21T14:02:33.446622+0000","last_became_peered":"2024-03-21T14:02:33.446622+0000","last_unstale":"2024-03-21T14:07:05.506145+0000","last_undegraded":"2024-03-21T14:07:05.506145+0000","last_fullsized":"2024-03-21T14:07:05.506145+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:05:20.136955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:20.136955+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.153604+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":166,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502232+0000","last_change":"2024-03-21T14:06:18.141733+0000","last_active":"2024-03-21T14:07:05.502232+0000","last_peered":"2024-03-21T14:07:05.502232+0000","last_clean":"2024-03-21T14:07:05.502232+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:07:05.502232+0000","last_undegraded":"2024-03-21T14:07:05.502232+0000","last_fullsized":"2024-03-21T14:07:05.502232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:06:18.141619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:18.141619+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:29.717483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":204,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.496706+0000","last_change":"2024-03-21T14:06:36.344441+0000","last_active":"2024-03-21T14:07:05.496706+0000","last_peered":"2024-03-21T14:07:05.496706+0000","last_clean":"2024-03-21T14:07:05.496706+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:07:05.496706+0000","last_undegraded":"2024-03-21T14:07:05.496706+0000","last_fullsized":"2024-03-21T14:07:05.496706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:06:36.344322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:36.344322+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:49.912913+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"125'10623","reported_seq":10729,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.441342+0000","last_change":"2024-03-21T14:06:42.835419+0000","last_active":"2024-03-21T14:07:06.441342+0000","last_peered":"2024-03-21T14:07:06.441342+0000","last_clean":"2024-03-21T14:07:06.441342+0000","last_became_active":"2024-03-21T14:04:28.364379+0000","last_became_peered":"2024-03-21T14:04:28.364379+0000","last_unstale":"2024-03-21T14:07:06.441342+0000","last_undegraded":"2024-03-21T14:07:06.441342+0000","last_fullsized":"2024-03-21T14:07:06.441342+0000","mapping_epoch":89,"log_start":"108'6000","ondisk_log_start":"108'6000","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":1999,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300482560,"num_objects":4585,"num_object_clones":0,"num_object_copies":13755,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4585,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.500059+0000","last_change":"2024-03-21T14:06:17.164352+0000","last_active":"2024-03-21T14:07:05.500059+0000","last_peered":"2024-03-21T14:07:05.500059+0000","last_clean":"2024-03-21T14:07:05.500059+0000","last_became_active":"2024-03-21T14:02:39.469270+0000","last_became_peered":"2024-03-21T14:02:39.469270+0000","last_unstale":"2024-03-21T14:07:05.500059+0000","last_undegraded":"2024-03-21T14:07:05.500059+0000","last_fullsized":"2024-03-21T14:07:05.500059+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:17.164232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:06:17.164232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:43.626287+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.503836+0000","last_change":"2024-03-21T14:05:29.550541+0000","last_active":"2024-03-21T14:07:05.503836+0000","last_peered":"2024-03-21T14:07:05.503836+0000","last_clean":"2024-03-21T14:07:05.503836+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:07:05.503836+0000","last_undegraded":"2024-03-21T14:07:05.503836+0000","last_fullsized":"2024-03-21T14:07:05.503836+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:29.548379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.548379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296843+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":177,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.496613+0000","last_change":"2024-03-21T14:06:16.045314+0000","last_active":"2024-03-21T14:07:05.496613+0000","last_peered":"2024-03-21T14:07:05.496613+0000","last_clean":"2024-03-21T14:07:05.496613+0000","last_became_active":"2024-03-21T14:02:35.773336+0000","last_became_peered":"2024-03-21T14:02:35.773336+0000","last_unstale":"2024-03-21T14:07:05.496613+0000","last_undegraded":"2024-03-21T14:07:05.496613+0000","last_fullsized":"2024-03-21T14:07:05.496613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:06:16.045170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.045170+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:28.072141+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":133,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502009+0000","last_change":"2024-03-21T14:05:27.174995+0000","last_active":"2024-03-21T14:07:05.502009+0000","last_peered":"2024-03-21T14:07:05.502009+0000","last_clean":"2024-03-21T14:07:05.502009+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:07:05.502009+0000","last_undegraded":"2024-03-21T14:07:05.502009+0000","last_fullsized":"2024-03-21T14:07:05.502009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:27.174851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:27.174851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"125'10339","reported_seq":10422,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.257638+0000","last_change":"2024-03-21T14:04:28.369284+0000","last_active":"2024-03-21T14:07:06.257638+0000","last_peered":"2024-03-21T14:07:06.257638+0000","last_clean":"2024-03-21T14:07:06.257638+0000","last_became_active":"2024-03-21T14:04:28.368956+0000","last_became_peered":"2024-03-21T14:04:28.368956+0000","last_unstale":"2024-03-21T14:07:06.257638+0000","last_undegraded":"2024-03-21T14:07:06.257638+0000","last_fullsized":"2024-03-21T14:07:06.257638+0000","mapping_epoch":89,"log_start":"100'4000","ondisk_log_start":"100'4000","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:04:27.337977+0000","objects_scrubbed":0,"log_size":6339,"log_dups_size":0,"ondisk_log_size":6339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.885490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296812544,"num_objects":4529,"num_object_clones":0,"num_object_copies":13587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10339,"num_write_kb":475776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498265+0000","last_change":"2024-03-21T14:05:24.417631+0000","last_active":"2024-03-21T14:07:05.498265+0000","last_peered":"2024-03-21T14:07:05.498265+0000","last_clean":"2024-03-21T14:07:05.498265+0000","last_became_active":"2024-03-21T14:02:39.470964+0000","last_became_peered":"2024-03-21T14:02:39.470964+0000","last_unstale":"2024-03-21T14:07:05.498265+0000","last_undegraded":"2024-03-21T14:07:05.498265+0000","last_fullsized":"2024-03-21T14:07:05.498265+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:24.417495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:05:24.417495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:06.313693+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":203,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501440+0000","last_change":"2024-03-21T14:05:37.280589+0000","last_active":"2024-03-21T14:07:05.501440+0000","last_peered":"2024-03-21T14:07:05.501440+0000","last_clean":"2024-03-21T14:07:05.501440+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:07:05.501440+0000","last_undegraded":"2024-03-21T14:07:05.501440+0000","last_fullsized":"2024-03-21T14:07:05.501440+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:05:37.280472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:05:37.280472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004442+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.496529+0000","last_change":"2024-03-21T14:06:34.326281+0000","last_active":"2024-03-21T14:07:05.496529+0000","last_peered":"2024-03-21T14:07:05.496529+0000","last_clean":"2024-03-21T14:07:05.496529+0000","last_became_active":"2024-03-21T14:02:33.443392+0000","last_became_peered":"2024-03-21T14:02:33.443392+0000","last_unstale":"2024-03-21T14:07:05.496529+0000","last_undegraded":"2024-03-21T14:07:05.496529+0000","last_fullsized":"2024-03-21T14:07:05.496529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.326168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.326168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:50.255180+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":169,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497998+0000","last_change":"2024-03-21T14:05:29.265158+0000","last_active":"2024-03-21T14:07:05.497998+0000","last_peered":"2024-03-21T14:07:05.497998+0000","last_clean":"2024-03-21T14:07:05.497998+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:07:05.497998+0000","last_undegraded":"2024-03-21T14:07:05.497998+0000","last_fullsized":"2024-03-21T14:07:05.497998+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:05:29.265009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:29.265009+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"125'10463","reported_seq":10624,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.275046+0000","last_change":"2024-03-21T14:06:21.944486+0000","last_active":"2024-03-21T14:07:06.275046+0000","last_peered":"2024-03-21T14:07:06.275046+0000","last_clean":"2024-03-21T14:07:06.275046+0000","last_became_active":"2024-03-21T14:04:28.371337+0000","last_became_peered":"2024-03-21T14:04:28.371337+0000","last_unstale":"2024-03-21T14:07:06.275046+0000","last_undegraded":"2024-03-21T14:07:06.275046+0000","last_fullsized":"2024-03-21T14:07:06.275046+0000","mapping_epoch":89,"log_start":"100'4200","ondisk_log_start":"100'4200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"118'8126","last_scrub_stamp":"2024-03-21T14:06:21.944346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.944346+0000","objects_scrubbed":6902,"log_size":6263,"log_dups_size":0,"ondisk_log_size":6263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.534040+0000","scrub_duration":1949,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294518808,"num_objects":4495,"num_object_clones":0,"num_object_copies":13485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10463,"num_write_kb":478593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":128,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.499130+0000","last_change":"2024-03-21T14:05:10.657107+0000","last_active":"2024-03-21T14:07:05.499130+0000","last_peered":"2024-03-21T14:07:05.499130+0000","last_clean":"2024-03-21T14:07:05.499130+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:07:05.499130+0000","last_undegraded":"2024-03-21T14:07:05.499130+0000","last_fullsized":"2024-03-21T14:07:05.499130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:10.656780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:10.656780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.296705+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":140,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.498666+0000","last_change":"2024-03-21T14:06:34.396244+0000","last_active":"2024-03-21T14:07:05.498666+0000","last_peered":"2024-03-21T14:07:05.498666+0000","last_clean":"2024-03-21T14:07:05.498666+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:07:05.498666+0000","last_undegraded":"2024-03-21T14:07:05.498666+0000","last_fullsized":"2024-03-21T14:07:05.498666+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:34.396131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:06:34.396131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:00.169873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502091+0000","last_change":"2024-03-21T14:05:33.267089+0000","last_active":"2024-03-21T14:07:05.502091+0000","last_peered":"2024-03-21T14:07:05.502091+0000","last_clean":"2024-03-21T14:07:05.502091+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:07:05.502091+0000","last_undegraded":"2024-03-21T14:07:05.502091+0000","last_fullsized":"2024-03-21T14:07:05.502091+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:05:33.266985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.266985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004238+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":164,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501296+0000","last_change":"2024-03-21T14:06:21.108018+0000","last_active":"2024-03-21T14:07:05.501296+0000","last_peered":"2024-03-21T14:07:05.501296+0000","last_clean":"2024-03-21T14:07:05.501296+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:07:05.501296+0000","last_undegraded":"2024-03-21T14:07:05.501296+0000","last_fullsized":"2024-03-21T14:07:05.501296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:06:21.107886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:21.107886+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:42.591227+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"125'10696","reported_seq":10840,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.444524+0000","last_change":"2024-03-21T14:05:36.765363+0000","last_active":"2024-03-21T14:07:06.444524+0000","last_peered":"2024-03-21T14:07:06.444524+0000","last_clean":"2024-03-21T14:07:06.444524+0000","last_became_active":"2024-03-21T14:04:28.365174+0000","last_became_peered":"2024-03-21T14:04:28.365174+0000","last_unstale":"2024-03-21T14:07:06.444524+0000","last_undegraded":"2024-03-21T14:07:06.444524+0000","last_fullsized":"2024-03-21T14:07:06.444524+0000","mapping_epoch":89,"log_start":"99'3800","ondisk_log_start":"99'3800","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"105'5496","last_scrub_stamp":"2024-03-21T14:05:36.764945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:05:36.764945+0000","objects_scrubbed":5416,"log_size":6896,"log_dups_size":0,"ondisk_log_size":6896,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.006448+0000","scrub_duration":2456,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295960576,"num_objects":4516,"num_object_clones":0,"num_object_copies":13548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10696,"num_write_kb":486784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":137,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501682+0000","last_change":"2024-03-21T14:05:33.294251+0000","last_active":"2024-03-21T14:07:05.501682+0000","last_peered":"2024-03-21T14:07:05.501682+0000","last_clean":"2024-03-21T14:07:05.501682+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:07:05.501682+0000","last_undegraded":"2024-03-21T14:07:05.501682+0000","last_fullsized":"2024-03-21T14:07:05.501682+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:05:33.294119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:05:33.294119+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.006376+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.690652+0000","last_change":"2024-03-21T14:07:05.690652+0000","last_active":"2024-03-21T14:07:05.690652+0000","last_peered":"2024-03-21T14:07:05.690652+0000","last_clean":"2024-03-21T14:07:05.690652+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:07:05.690652+0000","last_undegraded":"2024-03-21T14:07:05.690652+0000","last_fullsized":"2024-03-21T14:07:05.690652+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:07:05.690531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:07:05.690531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:08.557966+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.491132+0000","last_change":"2024-03-21T14:06:31.951712+0000","last_active":"2024-03-21T14:07:05.491132+0000","last_peered":"2024-03-21T14:07:05.491132+0000","last_clean":"2024-03-21T14:07:05.491132+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:07:05.491132+0000","last_undegraded":"2024-03-21T14:07:05.491132+0000","last_fullsized":"2024-03-21T14:07:05.491132+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:31.951638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:31.951638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:34.454617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":173,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.502132+0000","last_change":"2024-03-21T14:06:16.191465+0000","last_active":"2024-03-21T14:07:05.502132+0000","last_peered":"2024-03-21T14:07:05.502132+0000","last_clean":"2024-03-21T14:07:05.502132+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:07:05.502132+0000","last_undegraded":"2024-03-21T14:07:05.502132+0000","last_fullsized":"2024-03-21T14:07:05.502132+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:06:16.191249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:06:16.191249+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:44.656713+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"125'10810","reported_seq":10964,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:06.620802+0000","last_change":"2024-03-21T14:06:42.341735+0000","last_active":"2024-03-21T14:07:06.620802+0000","last_peered":"2024-03-21T14:07:06.620802+0000","last_clean":"2024-03-21T14:07:06.620802+0000","last_became_active":"2024-03-21T14:04:28.364570+0000","last_became_peered":"2024-03-21T14:04:28.364570+0000","last_unstale":"2024-03-21T14:07:06.620802+0000","last_undegraded":"2024-03-21T14:07:06.620802+0000","last_fullsized":"2024-03-21T14:07:06.620802+0000","mapping_epoch":89,"log_start":"103'5200","ondisk_log_start":"103'5200","created":89,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'9480","last_scrub_stamp":"2024-03-21T14:06:42.341399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:27.337977+0000","last_clean_scrub_stamp":"2024-03-21T14:06:42.341399+0000","objects_scrubbed":6153,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:08:07.311401+0000","scrub_duration":8346,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304087040,"num_objects":4640,"num_object_clones":0,"num_object_copies":13920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4640,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10810,"num_write_kb":494400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":136,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501537+0000","last_change":"2024-03-21T14:06:41.193471+0000","last_active":"2024-03-21T14:07:05.501537+0000","last_peered":"2024-03-21T14:07:05.501537+0000","last_clean":"2024-03-21T14:07:05.501537+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:07:05.501537+0000","last_undegraded":"2024-03-21T14:07:05.501537+0000","last_fullsized":"2024-03-21T14:07:05.501537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:41.193360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:06:41.193360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:45.867638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":205,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501690+0000","last_change":"2024-03-21T14:05:32.346872+0000","last_active":"2024-03-21T14:07:05.501690+0000","last_peered":"2024-03-21T14:07:05.501690+0000","last_clean":"2024-03-21T14:07:05.501690+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:07:05.501690+0000","last_undegraded":"2024-03-21T14:07:05.501690+0000","last_fullsized":"2024-03-21T14:07:05.501690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:05:32.346553+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.006413+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":149,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.497601+0000","last_change":"2024-03-21T14:06:39.275391+0000","last_active":"2024-03-21T14:07:05.497601+0000","last_peered":"2024-03-21T14:07:05.497601+0000","last_clean":"2024-03-21T14:07:05.497601+0000","last_became_active":"2024-03-21T14:02:33.441741+0000","last_became_peered":"2024-03-21T14:02:33.441741+0000","last_unstale":"2024-03-21T14:07:05.497601+0000","last_undegraded":"2024-03-21T14:07:05.497601+0000","last_fullsized":"2024-03-21T14:07:05.497601+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:06:39.275282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:06:39.275282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:52.086851+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"72'10","reported_seq":152,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-03-21T14:07:05.501094+0000","last_change":"2024-03-21T14:05:14.162485+0000","last_active":"2024-03-21T14:07:05.501094+0000","last_peered":"2024-03-21T14:07:05.501094+0000","last_clean":"2024-03-21T14:07:05.501094+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:07:05.501094+0000","last_undegraded":"2024-03-21T14:07:05.501094+0000","last_fullsized":"2024-03-21T14:07:05.501094+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:05:14.162391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:05:14.162391+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:07:09.004144+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4748673048,"num_objects":72460,"num_object_clones":0,"num_object_copies":217380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72460,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":169570,"num_write_kb":7744897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14249897984,"data_stored":14249885768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90003,"ondisk_log_size":90003,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":266,"num_read_kb":261,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103227,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1555216,"kb_used_data":1461572,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":92189936,"statfs":{"total":95995035648,"available":94402494464,"internally_reserved":0,"allocated":1496649728,"data_stored":1496254315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":95877578},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430272,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1540288,"kb_used_data":1454960,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92204864,"statfs":{"total":95995035648,"available":94417780736,"internally_reserved":0,"allocated":1489879040,"data_stored":1489502168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87357894},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1856812,"kb_used_data":1757596,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91888340,"statfs":{"total":95995035648,"available":94093660160,"internally_reserved":0,"allocated":1799778304,"data_stored":1799358386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2473048,"kb_used_data":2333576,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91272104,"statfs":{"total":95995035648,"available":93462634496,"internally_reserved":0,"allocated":2389581824,"data_stored":2389181170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858368,"kb_used_data":1746672,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91886784,"statfs":{"total":95995035648,"available":94092066816,"internally_reserved":0,"allocated":1788592128,"data_stored":1788215036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182479,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2758240,"kb_used_data":2614612,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":90986912,"statfs":{"total":95995035648,"available":93170597888,"internally_reserved":0,"allocated":2677362688,"data_stored":2676949724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018955,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1552408,"kb_used_data":1467084,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92192744,"statfs":{"total":95995035648,"available":94405369856,"internally_reserved":0,"allocated":1502294016,"data_stored":1501888958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691910,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1245440,"kb_used_data":1174004,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92499712,"statfs":{"total":95995035648,"available":94719705088,"internally_reserved":0,"allocated":1202180096,"data_stored":1201793331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2372866048,"data_stored":2372861976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1781010432,"data_stored":1781006360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1774190592,"data_stored":1774190592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2671771648,"data_stored":2671771648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1497300992,"data_stored":1497300992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1183973376,"data_stored":1183973376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1479217152,"data_stored":1479213080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1489567744,"data_stored":1489567744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:07:11.555 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-21T14:07:11.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:11.645 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:11.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:11.727 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:11.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:11.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:11.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:11.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:11.933 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:11.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:12.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:12.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:12.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:12.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:12.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 7.3 scrub starts 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: pgmap v169: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 845 op/s 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:07:12.326 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 7.3 scrub starts 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: pgmap v169: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 845 op/s 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:07:12.327 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:12 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:07:12.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:12.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 7.3 scrub starts 2024-03-21T14:07:12.417 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: pgmap v169: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 845 op/s 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:07:12.418 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:12 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:07:12.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T14:07:12.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:12.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:12.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:12.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:12.771 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:12.779 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:12.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:12.901 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:12.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:13.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:13.032 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:13.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:13.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:13.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:13.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:13.307 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:13.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:13.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:13 smithi118 ceph-mon[33862]: 7.2 scrub starts 2024-03-21T14:07:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:13 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:07:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:13 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:07:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:13 smithi118 ceph-mon[33862]: 7.3 scrub ok 2024-03-21T14:07:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:13 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:07:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:13 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:07:13.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:13.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[29576]: 7.2 scrub starts 2024-03-21T14:07:13.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:07:13.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[29576]: 7.3 scrub ok 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[40510]: 7.2 scrub starts 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[40510]: 7.3 scrub ok 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:07:13.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:13 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:07:13.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:13.579 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:13.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:13.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:13.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:13.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:13.966 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:13.993 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:13.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:14.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:14.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:14.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:14.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:14.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:14.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:14.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:14.402 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:14.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:14.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: 7.2 scrub ok 2024-03-21T14:07:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:14 smithi118 ceph-mon[33862]: pgmap v171: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:07:14.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:07:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: 7.2 scrub ok 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[29576]: pgmap v171: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: osdmap e126: 8 total, 8 up, 8 in 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: 7.2 scrub ok 2024-03-21T14:07:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:14 smithi003 ceph-mon[40510]: pgmap v171: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:07:14.559 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:14.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:14.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:14.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:14.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:14.755 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:07:14.755 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:151054 7.7 7:e7985f45:::benchmark_data_smithi003_80362_object30037:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:45.913695+0000", 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.839720164999999, 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.132171765, 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 151054 2024-03-21T14:07:14.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.913695+0000", 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.913695+0000", 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.913696+0000", 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.879999999999999e-07 2024-03-21T14:07:14.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.913700+0000", 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2139999999999998e-06 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.913712+0000", 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2418e-05 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.913722+0000", 2024-03-21T14:07:14.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5009999999999993e-06 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.936554+0000", 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022832577 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.936636+0000", 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1487000000000001e-05 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.936712+0000", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.627e-05 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:45.938051+0000", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001339371 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.023697+0000", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.085645233000000001 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.045745+0000", 2024-03-21T14:07:14.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022048264000000001 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.045806+0000", 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1079000000000003e-05 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.045866+0000", 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0362999999999998e-05 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.761 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:151431 7.7 7:eed88c31:::benchmark_data_smithi003_80362_object30414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:46.589240+0000", 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 28.164175200999999, 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13263275499999999, 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 151431 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.589240+0000", 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.589240+0000", 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.589241+0000", 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.17e-06 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.589245+0000", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5419999999999999e-06 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.589254+0000", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4810000000000006e-06 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.589259+0000", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7869999999999998e-06 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.662485+0000", 2024-03-21T14:07:14.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.073225686999999998 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.662622+0000", 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013692500000000001 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.662751+0000", 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012915999999999999 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.665771+0000", 2024-03-21T14:07:14.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030195819999999998 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.718410+0000", 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052638933999999998 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.721763+0000", 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033530410000000002 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.721818+0000", 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5405999999999998e-05 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.721872+0000", 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4039999999999998e-05 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:151565 7.7 7:e7267da2:::benchmark_data_smithi003_80362_object30548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:46.776441+0000", 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.976974201000001, 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12088291299999999, 2024-03-21T14:07:14.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 151565 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.776441+0000", 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.776441+0000", 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.776442+0000", 2024-03-21T14:07:14.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.559e-06 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.776446+0000", 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6119999999999999e-06 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.776457+0000", 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1615e-05 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.776464+0000", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3910000000000003e-06 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.824070+0000", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.047606086999999998 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.824194+0000", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012445300000000001 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.824318+0000", 2024-03-21T14:07:14.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012365499999999999 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.826858+0000", 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025398030000000002 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.877145+0000", 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.050286995000000001 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.897224+0000", 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020079079 2024-03-21T14:07:14.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.897259+0000", 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5577999999999997e-05 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:46.897323+0000", 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4085999999999996e-05 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:151929 7.7 7:e4559b19:::benchmark_data_smithi003_80362_object30912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.773 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:47.351645+0000", 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.401770078999999, 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12524893400000001, 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 151929 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351645+0000", 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351645+0000", 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351645+0000", 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.63e-07 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351648+0000", 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4990000000000001e-06 2024-03-21T14:07:14.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351655+0000", 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1040000000000001e-06 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351661+0000", 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9769999999999997e-06 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351707+0000", 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6752000000000002e-05 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351909+0000", 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000201838 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.352045+0000", 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013554499999999999 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.357021+0000", 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049759970000000002 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.476772+0000", 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.119751148 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.476812+0000", 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9761000000000001e-05 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.476847+0000", 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5101000000000002e-05 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.476894+0000", 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6749000000000003e-05 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:151932 7.7 7:e5b02c7d:::benchmark_data_smithi003_80362_object30915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:47.351676+0000", 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.401739082999999, 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12557858599999999, 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.779 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 151932 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351676+0000", 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351676+0000", 2024-03-21T14:07:14.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351676+0000", 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3300000000000003e-07 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351679+0000", 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5229999999999998e-06 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351685+0000", 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.613e-06 2024-03-21T14:07:14.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.351691+0000", 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6960000000000002e-06 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.354574+0000", 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028831659999999999 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.354721+0000", 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014661899999999999 2024-03-21T14:07:14.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.354865+0000", 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014453400000000001 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.357125+0000", 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022597730000000001 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.477095+0000", 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.119969944 2024-03-21T14:07:14.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.477133+0000", 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8152999999999999e-05 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.477195+0000", 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1525999999999993e-05 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.477254+0000", 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9605999999999998e-05 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.784 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:152010 7.7 7:ed4f4c2e:::benchmark_data_smithi003_80362_object30993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:47.478125+0000", 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 27.275290200000001, 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12289370300000001, 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 152010 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.478125+0000", 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.478125+0000", 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.478126+0000", 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.263e-06 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.478131+0000", 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4659999999999998e-06 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.478143+0000", 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2068e-05 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.478151+0000", 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4200000000000001e-06 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.492691+0000", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014540187 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.492818+0000", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012685499999999999 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.492946+0000", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012823399999999999 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.495724+0000", 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027784490000000001 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.516007+0000", 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.020282664999999998 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.600887+0000", 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.084879668000000005 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.600955+0000", 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7812999999999996e-05 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.601018+0000", 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3615000000000007e-05 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:152284 7.7 7:e3974573:::benchmark_data_smithi003_80362_object31267:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:47.876150+0000", 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.877264793999998, 2024-03-21T14:07:14.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.129903981, 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 152284 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.876150+0000", 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.876150+0000", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.876151+0000", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.314e-06 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.876155+0000", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2579999999999997e-06 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.876166+0000", 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.063e-05 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.876172+0000", 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.319e-06 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.898767+0000", 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.022594916 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.898914+0000", 2024-03-21T14:07:14.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014699 2024-03-21T14:07:14.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.899040+0000", 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012555300000000001 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.903199+0000", 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004158681 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:47.973531+0000", 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.070332550999999993 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.005972+0000", 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.032440946999999998 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.006011+0000", 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9062e-05 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.006054+0000", 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2759999999999997e-05 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:152440 7.7 7:ee265198:::benchmark_data_smithi003_80362_object31423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:48.176169+0000", 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.577245914999999, 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.1224759, 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 152440 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.176169+0000", 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.176169+0000", 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.176170+0000", 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2780000000000001e-06 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.176176+0000", 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1770000000000003e-06 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.176187+0000", 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0369e-05 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.176195+0000", 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4260000000000003e-06 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.234451+0000", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.058255452999999999 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.234531+0000", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0069999999999997e-05 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.234604+0000", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3242000000000001e-05 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.237770+0000", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031663949999999998 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298029+0000", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.060258352000000001 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298579+0000", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055022600000000001 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298613+0000", 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3794000000000002e-05 2024-03-21T14:07:14.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298645+0000", 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2118e-05 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:152442 7.7 7:e08475dc:::benchmark_data_smithi003_80362_object31425:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:48.177932+0000", 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 26.575482992000001, 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.120822055, 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.802 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 152442 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.177932+0000", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.177932+0000", 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.177933+0000", 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.068e-06 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.177941+0000", 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5250000000000005e-06 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.177953+0000", 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1895999999999999e-05 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.177962+0000", 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5140000000000001e-06 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.238371+0000", 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.060408811999999999 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.238449+0000", 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8255000000000002e-05 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.238526+0000", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7220999999999998e-05 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.242989+0000", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044629090000000001 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298122+0000", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.055133505999999999 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298677+0000", 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055491299999999998 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298708+0000", 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0567000000000001e-05 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:48.298754+0000", 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5868999999999997e-05 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:153910 7.7 7:eca5b09c:::benchmark_data_smithi003_80362_object32893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:50.428960+0000", 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 24.324454961000001, 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12896205999999999, 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 153910 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.428960+0000", 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.428960+0000", 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.428961+0000", 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2589999999999999e-06 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.428967+0000", 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6330000000000001e-06 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.428988+0000", 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0947000000000001e-05 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.428996+0000", 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4870000000000002e-06 2024-03-21T14:07:14.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.483663+0000", 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054666834999999997 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.483745+0000", 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2305000000000006e-05 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.483819+0000", 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3937999999999996e-05 2024-03-21T14:07:14.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.488077+0000", 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042582849999999997 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.542878+0000", 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.054800508999999997 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.557820+0000", 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014942217000000001 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.557871+0000", 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0507999999999998e-05 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.557922+0000", 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1137000000000002e-05 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154023 7.7 7:ec8384af:::benchmark_data_smithi003_80362_object33006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:07:14.813 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 24.132311297000001, 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.14729518, 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154023 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621105+0000", 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.155e-06 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621110+0000", 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5790000000000003e-06 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621119+0000", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1670000000000005e-06 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621125+0000", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9370000000000001e-06 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665294+0000", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044168488999999998 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665370+0000", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6055000000000003e-05 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665441+0000", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1031000000000005e-05 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.668869+0000", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034284250000000001 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.767406+0000", 2024-03-21T14:07:14.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.098536532999999996 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768291+0000", 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00088539800000000002 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768347+0000", 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5769000000000001e-05 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768399+0000", 2024-03-21T14:07:14.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1641999999999998e-05 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154294 7.7 7:ee951786:::benchmark_data_smithi003_80362_object33277:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.073485+0000", 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.679929712, 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12139833799999999, 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.819 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154294 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.073485+0000", 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.073485+0000", 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.073486+0000", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.221e-06 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.073491+0000", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9509999999999997e-06 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.073501+0000", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9410000000000002e-06 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.073508+0000", 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8809999999999998e-06 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.135713+0000", 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.062204697000000003 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.135804+0000", 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0826000000000001e-05 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.135884+0000", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0863000000000002e-05 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.138002+0000", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021179200000000001 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.183640+0000", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.045637829999999997 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.194758+0000", 2024-03-21T14:07:14.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011117564999999999 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.194819+0000", 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0850999999999997e-05 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.194883+0000", 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4791999999999999e-05 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154388 7.7 7:ef0c72e7:::benchmark_data_smithi003_80362_object33371:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.241743+0000", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.511671794000002, 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12573778599999999, 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154388 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.241743+0000", 2024-03-21T14:07:14.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.241743+0000", 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.241744+0000", 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1540000000000001e-06 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.241749+0000", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-06 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.241758+0000", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.55e-06 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.241765+0000", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9609999999999999e-06 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.288550+0000", 2024-03-21T14:07:14.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.046784572000000003 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.288634+0000", 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4363000000000005e-05 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.288710+0000", 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6104000000000002e-05 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.290483+0000", 2024-03-21T14:07:14.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001772441 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.342974+0000", 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.052490857000000002 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.367385+0000", 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024411581000000002 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.367430+0000", 2024-03-21T14:07:14.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4975000000000002e-05 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.367481+0000", 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0547999999999998e-05 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154391 7.7 7:e00b968f:::benchmark_data_smithi003_80362_object33374:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.244471+0000", 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.508943548000001, 2024-03-21T14:07:14.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12332003599999999, 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154391 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.244471+0000", 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.244471+0000", 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.244472+0000", 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9800000000000002e-07 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.244476+0000", 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.946e-06 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.244486+0000", 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7310000000000005e-06 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.244494+0000", 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5790000000000004e-06 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.291164+0000", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.046669430999999997 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.291234+0000", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0129000000000006e-05 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.291305+0000", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0757e-05 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.294287+0000", 2024-03-21T14:07:14.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029828329999999998 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.343026+0000", 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.048738510999999998 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.367646+0000", 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.024620362999999999 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.367714+0000", 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7251999999999999e-05 2024-03-21T14:07:14.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.367791+0000", 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7606000000000002e-05 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154490 7.7 7:eb9688d9:::benchmark_data_smithi003_80362_object33473:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.414938+0000", 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.338476802999999, 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12848531699999999, 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.837 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154490 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.414938+0000", 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.414938+0000", 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.414940+0000", 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7689999999999999e-06 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.414947+0000", 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3710000000000002e-06 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.414961+0000", 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3913999999999999e-05 2024-03-21T14:07:14.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.414971+0000", 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7070000000000004e-06 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.480231+0000", 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.065260116000000007 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.480351+0000", 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119796 2024-03-21T14:07:14.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.480471+0000", 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120011 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.485496+0000", 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050251280000000002 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.527637+0000", 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042141290999999997 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.543223+0000", 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015585617 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.543349+0000", 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012596800000000001 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.543423+0000", 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4629000000000007e-05 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154494 7.7 7:ec8c7b03:::benchmark_data_smithi003_80362_object33477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.419179+0000", 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.334235953, 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.124407138, 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154494 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.419179+0000", 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.419179+0000", 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.419181+0000", 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.88e-06 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.419189+0000", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5650000000000002e-06 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.419204+0000", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4728e-05 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.419213+0000", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7169999999999997e-06 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.483259+0000", 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.064046622999999997 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.483345+0000", 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6058999999999997e-05 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.483424+0000", 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8254e-05 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.485500+0000", 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020765060000000001 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.542949+0000", 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.057448792999999998 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.543475+0000", 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052660100000000002 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.543528+0000", 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2416000000000001e-05 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.543586+0000", 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7995999999999998e-05 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154649 7.7 7:e07b5996:::benchmark_data_smithi003_80362_object33632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.677403+0000", 2024-03-21T14:07:14.848 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 23.076012101, 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13170960500000001, 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154649 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677403+0000", 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677403+0000", 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677403+0000", 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8500000000000002e-07 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677406+0000", 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8200000000000001e-06 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677413+0000", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8650000000000003e-06 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677418+0000", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6670000000000001e-06 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677457+0000", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8828999999999997e-05 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677534+0000", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6558999999999997e-05 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.677615+0000", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1210000000000003e-05 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.679960+0000", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023449399999999998 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.691651+0000", 2024-03-21T14:07:14.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011691181 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.808986+0000", 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.117334841 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.809048+0000", 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2416999999999995e-05 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.809112+0000", 2024-03-21T14:07:14.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3791e-05 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154716 7.7 7:eb3d6bd8:::benchmark_data_smithi003_80362_object33699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.979581261, 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.135279182, 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.854 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154716 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773834+0000", 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0399999999999995e-07 2024-03-21T14:07:14.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773838+0000", 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9910000000000004e-06 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773846+0000", 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2209999999999997e-06 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773853+0000", 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3890000000000002e-06 2024-03-21T14:07:14.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814039+0000", 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040186573000000003 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814159+0000", 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012002600000000001 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814282+0000", 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000123091 2024-03-21T14:07:14.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817930+0000", 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036476630000000002 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831508+0000", 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013578365 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.908985+0000", 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077476415000000007 2024-03-21T14:07:14.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909037+0000", 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1929999999999999e-05 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909113+0000", 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5814000000000003e-05 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154717 7.7 7:e048d7ab:::benchmark_data_smithi003_80362_object33700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.979320565999998, 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13536177999999999, 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154717 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774095+0000", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.861 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774098+0000", 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1640000000000001e-06 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774106+0000", 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3059999999999998e-06 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774112+0000", 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5740000000000003e-06 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816429+0000", 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042316843 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816497+0000", 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8421000000000001e-05 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816574+0000", 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6460999999999999e-05 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820900+0000", 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043267039999999998 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831554+0000", 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010653673000000001 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909389+0000", 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077835369000000001 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909424+0000", 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4275e-05 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909456+0000", 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2254e-05 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154718 7.7 7:eaddd76b:::benchmark_data_smithi003_80362_object33701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.978575627000001, 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13475282299999999, 2024-03-21T14:07:14.865 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154718 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:07:14.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774840+0000", 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774844+0000", 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.783e-06 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:07:14.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774853+0000", 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5459999999999997e-06 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774859+0000", 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2639999999999997e-06 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817968+0000", 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043108377000000003 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:07:14.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818046+0000", 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8118e-05 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818119+0000", 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2712000000000004e-05 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820930+0000", 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028118449999999999 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909502+0000", 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.088571516000000003 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909536+0000", 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3813999999999998e-05 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909563+0000", 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7297e-05 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:07:14.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909592+0000", 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8881e-05 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:07:14.871 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:07:14.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:14.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:15.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:15.033 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:15.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:15.164 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:15.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:15.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:15.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:07:15.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:07:15.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:07:15.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:07:15.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:07:15.388 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:07:15.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:07:15.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:07:15.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:07:15.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:07:15.390 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:07:15.391 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:15 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:07:15.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:15.425 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:07:15.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:15 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:07:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:15 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:07:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:15 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:07:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:15 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:07:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:15 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:07:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:15 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:07:15.437 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:15.472 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:07:15.516 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:07:15.539 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:15.566 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:15.574 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:07:15.617 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:07:15.663 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:07:15.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:15.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.699 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:15.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:15.708 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:07:15.748 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:07:15.753 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:07:15.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:15.829 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:15.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:15.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:15.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:15.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:16.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:16.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:16.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:16.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:16.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:16.278 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:07:16.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:16.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:16.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:07:16.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:07:16.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: pgmap v172: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 987 op/s 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/277820866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/345502447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[29576]: 7.7 scrub starts 2024-03-21T14:07:16.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: pgmap v172: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 987 op/s 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/277820866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/345502447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:16.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-mon[40510]: 7.7 scrub starts 2024-03-21T14:07:16.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:16.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:16.556 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:07:16.556 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:07:16.556 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-21T14:07:16.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-03-21T14:07:16.623 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:16.649 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: pgmap v172: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 987 op/s 2024-03-21T14:07:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/277820866' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:16.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:16.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/345502447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:16.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:16.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-mon[33862]: 7.7 scrub starts 2024-03-21T14:07:16.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:07:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:07:16.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:07:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:07:16.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:16.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:16.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:16.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:17.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:17.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:17.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:17.261 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:07:17.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:17.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:17.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:07:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:07:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:07:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[29576]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T14:07:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:17.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[29576]: 7.c scrub starts 2024-03-21T14:07:17.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:07:17.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:07:17.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:07:17.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[40510]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T14:07:17.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:17.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:17 smithi003 ceph-mon[40510]: 7.c scrub starts 2024-03-21T14:07:17.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:17.664 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:17.671 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:17 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:07:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:17 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:07:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:17 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:07:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:17 smithi118 ceph-mon[33862]: osdmap e127: 8 total, 8 up, 8 in 2024-03-21T14:07:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:17.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:17 smithi118 ceph-mon[33862]: 7.c scrub starts 2024-03-21T14:07:17.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:17.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:17.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:17.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:18.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:18.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:18.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:18.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:18.222 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:18.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:18.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:18.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: pgmap v174: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: 7.7 scrub ok 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: 7.c scrub ok 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:07:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:18 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:07:18.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:18.492 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:18.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:18.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:07:18.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:07:18.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:07:18.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: pgmap v174: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:07:18.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: 7.7 scrub ok 2024-03-21T14:07:18.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: 7.c scrub ok 2024-03-21T14:07:18.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: pgmap v174: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: 7.7 scrub ok 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: 7.c scrub ok 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:07:18.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:18 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:07:18.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:18.623 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:18.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:18.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:18.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:18.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:18.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:18.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:19.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:19.034 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:19.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:19.166 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:19.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:19.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:19 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:07:19.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:19 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:07:19.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:19 smithi118 ceph-mon[33862]: 4.1f deep-scrub starts 2024-03-21T14:07:19.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:19 smithi118 ceph-mon[33862]: 7.e scrub starts 2024-03-21T14:07:19.410 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:19 smithi118 ceph-mon[33862]: 4.1f deep-scrub ok 2024-03-21T14:07:19.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:19.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:19.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:19.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:07:19.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:07:19.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[29576]: 4.1f deep-scrub starts 2024-03-21T14:07:19.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[29576]: 7.e scrub starts 2024-03-21T14:07:19.548 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[29576]: 4.1f deep-scrub ok 2024-03-21T14:07:19.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:07:19.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:07:19.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[40510]: 4.1f deep-scrub starts 2024-03-21T14:07:19.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[40510]: 7.e scrub starts 2024-03-21T14:07:19.550 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:19 smithi003 ceph-mon[40510]: 4.1f deep-scrub ok 2024-03-21T14:07:19.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:07:19.584 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:19.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:19.705 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:19.705 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:19.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:19.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:19.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:19.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:19.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:19.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:19.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:20.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:20.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:20.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:20.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:20.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T14:07:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 839 B/s rd, 0 B/s wr, 827 op/s 2024-03-21T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: 3.0 scrub starts 2024-03-21T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: 3.0 scrub ok 2024-03-21T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:07:20.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:20 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:07:20.453 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:20.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:20.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:07:20.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:20.591 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:20.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:20.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:20.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 839 B/s rd, 0 B/s wr, 827 op/s 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: 3.0 scrub starts 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: 3.0 scrub ok 2024-03-21T14:07:20.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: osdmap e128: 8 total, 8 up, 8 in 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: pgmap v176: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 839 B/s rd, 0 B/s wr, 827 op/s 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: 3.0 scrub starts 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: 3.0 scrub ok 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:07:20.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:20 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:07:20.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:20.837 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:20.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:20.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:20.973 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:21.006 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:21.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:21.136 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:21.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:21.294 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:21.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:21.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:21.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:21.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:21.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:21 smithi003 ceph-mon[29576]: 7.e scrub ok 2024-03-21T14:07:21.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:21 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:07:21.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:21 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:07:21.498 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:21 smithi003 ceph-mon[40510]: 7.e scrub ok 2024-03-21T14:07:21.498 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:21 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:07:21.498 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:21 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:07:21.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:21.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:21.633 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:21.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:21 smithi118 ceph-mon[33862]: 7.e scrub ok 2024-03-21T14:07:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:21 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:07:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:21 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:07:21.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:21.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:21.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:21.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:21.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:21.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:22.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:22.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:22.126 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:22.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:22.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:moved 41 / 444 (9.23423%) 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:avg 55.5 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 6.85565 -> 11.2138 (expected baseline 6.96868) 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.2 with 44 -> 52 pgs (0.792793 -> 0.936937 * mean) 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.3 with 65 -> 49 pgs (1.17117 -> 0.882883 * mean) 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0169 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0185 2024-03-21T14:07:22.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 weight 1.0000 -> 0.6562 2024-03-21T14:07:22.347 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 1.0000 -> 0.7302 2024-03-21T14:07:22.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:22.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:22.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:22.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:07:22.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:07:22.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:07:22.500 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 6.1c deep-scrub starts 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 6.1c deep-scrub ok 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 4.4 deep-scrub starts 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: pgmap v177: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: 4.4 deep-scrub ok 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]: dispatch 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]: dispatch 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:07:22.501 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 6.1c deep-scrub starts 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 6.1c deep-scrub ok 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 4.4 deep-scrub starts 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: pgmap v177: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: 4.4 deep-scrub ok 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]: dispatch 2024-03-21T14:07:22.502 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:22 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]: dispatch 2024-03-21T14:07:22.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:22.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:22.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:22.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:22.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 6.1c deep-scrub starts 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 6.1c deep-scrub ok 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 4.4 deep-scrub starts 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: pgmap v177: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 860 op/s 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: 4.4 deep-scrub ok 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: from='client.24748 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:22.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]: dispatch 2024-03-21T14:07:22.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:22 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]: dispatch 2024-03-21T14:07:22.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:22.801 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:22.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:22.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:22.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:22.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:22.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:23.029 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:23.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:23.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:23.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:23.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:23.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:23.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:23.312 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:23.312 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:23.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:23.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:23.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:23.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:23.644 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:23 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:07:23.644 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:23 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:07:23.644 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]': finished 2024-03-21T14:07:23.644 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:23 smithi118 ceph-mon[33862]: osdmap e129: 8 total, 8 up, 8 in 2024-03-21T14:07:23.644 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:23 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:07:23.644 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:23 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:07:23.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:23.685 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]': finished 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[29576]: osdmap e129: 8 total, 8 up, 8 in 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:07:23.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:07:23.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'47852','3':'43005'}"}]': finished 2024-03-21T14:07:23.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[40510]: osdmap e129: 8 total, 8 up, 8 in 2024-03-21T14:07:23.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:07:23.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:23 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:07:23.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:23.809 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:23.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:23.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:23.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:23.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:24.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:07:24.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:24.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:24.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:24.198 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:24.246 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:07:24.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:24.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:24.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:24.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:24.505 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-21T14:07:24.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd test-reweight-by-utilization 110 1.0 2024-03-21T14:07:24.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:24.592 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:24.624 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:24 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:07:24.624 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:24 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:07:24.624 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:24 smithi118 ceph-mon[33862]: pgmap v179: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-21T14:07:24.624 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:24 smithi118 ceph-mon[33862]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T14:07:24.624 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:24 smithi118 ceph-mon[33862]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T14:07:24.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:24.724 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:24.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:07:24.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:07:24.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[29576]: pgmap v179: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-21T14:07:24.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[29576]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T14:07:24.726 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[29576]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T14:07:24.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:07:24.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:07:24.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[40510]: pgmap v179: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-21T14:07:24.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[40510]: osdmap e130: 8 total, 8 up, 8 in 2024-03-21T14:07:24.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:24 smithi003 ceph-mon[40510]: osdmap e131: 8 total, 8 up, 8 in 2024-03-21T14:07:24.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:24.857 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:24.958 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:24.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:24.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:24.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:25.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:25.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:25.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:25.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:25.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:25.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:25.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:25.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:25.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:25.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:25.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:25.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:25.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:25.673 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:07:25.673 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:07:25.673 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'11472 2024-03-21T14:07:25.673 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 128'11783 2024-03-21T14:07:25.673 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 128'11489 2024-03-21T14:07:25.673 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 128'11430 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'11357 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 128'11835 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'13941 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 128'11690 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 128'14595 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:25.674 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:25 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:25.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:25.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:25.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'11472 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 128'11783 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 128'11489 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 128'11430 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'11357 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 128'11835 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'13941 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 128'11690 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 128'14595 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:25.723 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.1 starting backfill to osd.1 from (0'0,0'0] MAX to 128'11472 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 128'11783 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 128'11489 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 128'11430 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.c starting backfill to osd.2 from (0'0,0'0] MAX to 128'11357 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.f starting backfill to osd.4 from (0'0,0'0] MAX to 128'11835 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.7 starting backfill to osd.1 from (0'0,0'0] MAX to 128'13941 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.e starting backfill to osd.3 from (0'0,0'0] MAX to 128'11690 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 128'14595 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: osdmap e132: 8 total, 8 up, 8 in 2024-03-21T14:07:25.724 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:25.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:25 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:25.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:25.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:25.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:26.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:26.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:26.137 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:26.137 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:26.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:26.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:26.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:26.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:26.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:26.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:26.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:26.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:26.597 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:26.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:26.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:07:26.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:07:26.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[29576]: pgmap v183: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 779 op/s 2024-03-21T14:07:26.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[29576]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T14:07:26.666 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:07:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:07:26.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:07:26.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:07:26.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[40510]: pgmap v183: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 779 op/s 2024-03-21T14:07:26.666 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:26 smithi003 ceph-mon[40510]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T14:07:26.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:07:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:07:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:26 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:07:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:26 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:07:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:26 smithi118 ceph-mon[33862]: pgmap v183: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 0 B/s wr, 779 op/s 2024-03-21T14:07:26.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:26 smithi118 ceph-mon[33862]: osdmap e133: 8 total, 8 up, 8 in 2024-03-21T14:07:26.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:26.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:26.782 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:07:26.793 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:26.798 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:07:26.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:07:26.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:26.854 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:07:26.879 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:26.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:26.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:26.905 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:07:26.956 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:07:26.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:27.000 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:07:27.053 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:07:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:27.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:27.108 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:07:27.122 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:07:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:07:27.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:27.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:27.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:27.347 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:27.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:27.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:27.460 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:07:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:27.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:27.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:27.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:27.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:27.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1594097414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2086209508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1594097414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:27.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:27.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:27.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2086209508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:07:27.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:27 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:07:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:27.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:27.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:27.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:27.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:27 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1594097414' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:27 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:27 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:27 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2086209508' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:07:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:27 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:07:28.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:28.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:28.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:28.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:28.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:28.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:28.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:28.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:28.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.474 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:28.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:28.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:28.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:28.729 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: pgmap v185: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 657 op/s; 29815/166179 objects misplaced (17.941%); 21 MiB/s, 341 objects/s recovering 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:07:28.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[29576]: osdmap e134: 8 total, 8 up, 8 in 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: pgmap v185: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 657 op/s; 29815/166179 objects misplaced (17.941%); 21 MiB/s, 341 objects/s recovering 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:07:28.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:28 smithi003 ceph-mon[40510]: osdmap e134: 8 total, 8 up, 8 in 2024-03-21T14:07:28.762 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:28.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:28.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:28.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: pgmap v185: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 657 op/s; 29815/166179 objects misplaced (17.941%); 21 MiB/s, 341 objects/s recovering 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:07:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:28 smithi118 ceph-mon[33862]: osdmap e134: 8 total, 8 up, 8 in 2024-03-21T14:07:29.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:29.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:29.168 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:29.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:29.304 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:29.406 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:29.437 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:29.539 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:29.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:29.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:29.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:29.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:29.835 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:29.861 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:29.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:29.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:29.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:30.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:30.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:30.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.138 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:30.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:30.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:30.272 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:30.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:30.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:30.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:30.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:30.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:30.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:30.608 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:30.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:07:30.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:30.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:30 smithi003 ceph-mon[29576]: pgmap v187: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 410 B/s rd, 0 B/s wr, 528 op/s; 29815/166179 objects misplaced (17.941%); 17 MiB/s, 274 objects/s recovering 2024-03-21T14:07:30.715 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:30 smithi003 ceph-mon[40510]: pgmap v187: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 410 B/s rd, 0 B/s wr, 528 op/s; 29815/166179 objects misplaced (17.941%); 17 MiB/s, 274 objects/s recovering 2024-03-21T14:07:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:30.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:30.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:30.929 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:30.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:30 smithi118 ceph-mon[33862]: pgmap v187: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 410 B/s rd, 0 B/s wr, 528 op/s; 29815/166179 objects misplaced (17.941%); 17 MiB/s, 274 objects/s recovering 2024-03-21T14:07:31.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:31.064 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:31.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:31.199 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:31.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:31.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:31.333 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:31.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:31.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:31.576 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:31.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:31.680 INFO:teuthology.orchestra.run.smithi003.stdout:moved 31 / 444 (6.98198%) 2024-03-21T14:07:31.680 INFO:teuthology.orchestra.run.smithi003.stdout:avg 55.5 2024-03-21T14:07:31.680 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.2138 -> 17.9722 (expected baseline 6.96868) 2024-03-21T14:07:31.680 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.0 with 33 -> 27 pgs (0.594595 -> 0.486486 * mean) 2024-03-21T14:07:31.680 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 72 -> 77 pgs (1.2973 -> 1.38739 * mean) 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 1 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0153 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0168 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 weight 0.6562 -> 0.4303 2024-03-21T14:07:31.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 0.7302 -> 0.5468 2024-03-21T14:07:31.688 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-03-21T14:07:31.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:31.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:31.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:31 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:31.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:31.837 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:31 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:07:31.675+0000 7fa1bb657700 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-21T14:07:31.838 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:31 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:31.838 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:31.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:31.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:31.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:31.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:32.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:32.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.060 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:32.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:32.161 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:32.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:31 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:07:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:32.193 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:32.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:32.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:32.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:32.440 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:32.473 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:32.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:32.529 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-03-21T14:07:32.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight-by-utilization 1000 3.0 2024-03-21T14:07:32.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:32.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:07:32.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:32.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[29576]: pgmap v188: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 658 op/s; 25383/160440 objects misplaced (15.821%); 41 MiB/s, 665 objects/s recovering 2024-03-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[29576]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:07:32.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:07:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[40510]: pgmap v188: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 658 op/s; 25383/160440 objects misplaced (15.821%); 41 MiB/s, 665 objects/s recovering 2024-03-21T14:07:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[40510]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:07:32.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:32 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:07:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:32.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:32.907 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:32.938 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:32.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:33.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:33.086 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:33.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:33.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:32 smithi118 ceph-mon[33862]: pgmap v188: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 658 op/s; 25383/160440 objects misplaced (15.821%); 41 MiB/s, 665 objects/s recovering 2024-03-21T14:07:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:32 smithi118 ceph-mon[33862]: from='client.24851 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:32 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:07:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:32 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:07:33.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:33.286 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:33.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:33.388 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:33.416 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:33.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:33.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:33.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:33.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:33.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:33.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:33.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:33.782 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:07:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:07:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-03-21T14:07:33.782 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:07:33.808 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:33 smithi003 ceph-mon[29576]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T14:07:33.808 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:33 smithi003 ceph-mon[40510]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T14:07:33.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:33.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:33.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:33.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:33.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:34.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:34.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:34.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:34.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:34.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:33 smithi118 ceph-mon[33862]: osdmap e135: 8 total, 8 up, 8 in 2024-03-21T14:07:34.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:34.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:34.416 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:34.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:34.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:34.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:34.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:34.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:34.814 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:34.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:34.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:34.948 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:34.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:34.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:34 smithi003 ceph-mon[29576]: pgmap v190: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 907 B/s rd, 0 B/s wr, 883 op/s; 23509/153147 objects misplaced (15.351%); 43 MiB/s, 687 objects/s recovering 2024-03-21T14:07:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:34 smithi003 ceph-mon[29576]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T14:07:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:34 smithi003 ceph-mon[40510]: pgmap v190: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 907 B/s rd, 0 B/s wr, 883 op/s; 23509/153147 objects misplaced (15.351%); 43 MiB/s, 687 objects/s recovering 2024-03-21T14:07:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:34 smithi003 ceph-mon[40510]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T14:07:35.049 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:35.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:35.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:34 smithi118 ceph-mon[33862]: pgmap v190: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 907 B/s rd, 0 B/s wr, 883 op/s; 23509/153147 objects misplaced (15.351%); 43 MiB/s, 687 objects/s recovering 2024-03-21T14:07:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:34 smithi118 ceph-mon[33862]: osdmap e136: 8 total, 8 up, 8 in 2024-03-21T14:07:35.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:35.217 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:35.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:35.319 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:35.350 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:35.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:35.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:35.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:35.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:35.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:35.754 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:35.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:07:35.890 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:35.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:35.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:35.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:35 smithi003 ceph-mon[29576]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T14:07:36.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:35 smithi003 ceph-mon[40510]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T14:07:36.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:36.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:36.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:35 smithi118 ceph-mon[33862]: osdmap e137: 8 total, 8 up, 8 in 2024-03-21T14:07:36.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:36.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:36.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:36.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:36.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:36.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:36.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:36.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:36.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:36.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:07:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:07:36.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:07:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:07:36.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:36.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:36.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:36.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:36.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:37.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:37.034 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:37.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:37.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:36 smithi118 ceph-mon[33862]: pgmap v193: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 724 op/s; 23509/153147 objects misplaced (15.351%); 42 MiB/s, 677 objects/s recovering 2024-03-21T14:07:37.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:36 smithi003 ceph-mon[29576]: pgmap v193: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 724 op/s; 23509/153147 objects misplaced (15.351%); 42 MiB/s, 677 objects/s recovering 2024-03-21T14:07:37.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:36 smithi003 ceph-mon[40510]: pgmap v193: 148 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 139 active+clean; 3.1 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 724 op/s; 23509/153147 objects misplaced (15.351%); 42 MiB/s, 677 objects/s recovering 2024-03-21T14:07:37.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:37.317 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:37.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:37.448 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:37.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:37.578 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:37.680 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:37.710 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:37.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:37.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:37.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:37.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:37.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:37.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:38.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:38.098 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:38.137 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:38.138 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:38.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:38.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:38.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:38.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:38.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:38.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:38.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:38.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:38.654 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:38.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:38.798 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 444 (0%) 2024-03-21T14:07:38.798 INFO:teuthology.orchestra.run.smithi003.stdout:avg 55.5 2024-03-21T14:07:38.798 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.2138 -> 11.2138 (expected baseline 6.96868) 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.0 with 33 -> 33 pgs (0.594595 -> 0.594595 * mean) 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 72 -> 72 pgs (1.2973 -> 1.2973 * mean) 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 3 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0140 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.1398 2024-03-21T14:07:38.799 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-03-21T14:07:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:38.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:38.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:38.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:38.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:38.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:39.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:38 smithi003 ceph-mon[29576]: pgmap v194: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 17641/141966 objects misplaced (12.426%); 54 MiB/s, 856 objects/s recovering 2024-03-21T14:07:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:38 smithi003 ceph-mon[40510]: pgmap v194: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 17641/141966 objects misplaced (12.426%); 54 MiB/s, 856 objects/s recovering 2024-03-21T14:07:39.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:39.081 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:38 smithi118 ceph-mon[33862]: pgmap v194: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 17641/141966 objects misplaced (12.426%); 54 MiB/s, 856 objects/s recovering 2024-03-21T14:07:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:39.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:39.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:39.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:39.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:39.394 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:39.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:39.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:39.558 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:39.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:39.687 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:07:39.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:39.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:39.728 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:07:39.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:07:39.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:39.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:39 smithi003 ceph-mon[29576]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:39.810 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/117845656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:39.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:39 smithi003 ceph-mon[40510]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:39.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/117845656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:39.824 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:07:39.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:39.869 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:07:39.908 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:07:39.930 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:39.950 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:07:39.965 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:39.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:39.996 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:07:40.000 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:07:40.067 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:40.101 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:40.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:40.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:39 smithi118 ceph-mon[33862]: from='client.24857 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/117845656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:40.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:40.247 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-03-21T14:07:40.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd test-reweight-by-utilization 110 0.05 2024-03-21T14:07:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:40.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:40.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:40.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:40.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:40.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:40.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:40.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:40.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:40.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:40.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:40.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:40.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:40.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:40.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:40.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:40.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:40.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:40 smithi003 ceph-mon[29576]: pgmap v195: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 563 op/s; 17641/141966 objects misplaced (12.426%); 41 MiB/s, 650 objects/s recovering 2024-03-21T14:07:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:40 smithi003 ceph-mon[40510]: pgmap v195: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 563 op/s; 17641/141966 objects misplaced (12.426%); 41 MiB/s, 650 objects/s recovering 2024-03-21T14:07:41.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:41.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:41.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:41.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:40 smithi118 ceph-mon[33862]: pgmap v195: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 772 B/s rd, 0 B/s wr, 563 op/s; 17641/141966 objects misplaced (12.426%); 41 MiB/s, 650 objects/s recovering 2024-03-21T14:07:41.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:41.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:41.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:41.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:41.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:41.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:41.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:41.817 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:07:41.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:41.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:41.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:41.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:41.986 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:41.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:41.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:42.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:42.121 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:42.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2682280398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:41 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:42.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2682280398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:41 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:42.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2682280398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:41 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:07:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:42.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:42.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:42.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:42.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:42.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:42.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:42.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.668 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:42.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:42.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:42.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:42.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:42.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:42.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:43.046 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:43.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:42 smithi118 ceph-mon[33862]: pgmap v196: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.8 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s; 15828/136665 objects misplaced (11.582%); 43 MiB/s, 683 objects/s recovering 2024-03-21T14:07:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:42 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:07:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:42 smithi118 ceph-mon[33862]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T14:07:43.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[29576]: pgmap v196: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.8 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s; 15828/136665 objects misplaced (11.582%); 43 MiB/s, 683 objects/s recovering 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[29576]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[40510]: pgmap v196: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.8 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s; 15828/136665 objects misplaced (11.582%); 43 MiB/s, 683 objects/s recovering 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:07:43.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:42 smithi003 ceph-mon[40510]: osdmap e138: 8 total, 8 up, 8 in 2024-03-21T14:07:43.220 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:43.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:43.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:43.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:43.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:43.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:43.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:43.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:43.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:43.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:43.913 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:43.943 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:43.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:44.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:44.072 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:44.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:44.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:44.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:44.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:44.307 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:44.332 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:44.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:44.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:44.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:44.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:44.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:44.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:44.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.852 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:44.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:44.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:44.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:44.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:45.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:45.120 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:45.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:44 smithi003 ceph-mon[29576]: pgmap v198: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.6 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 912 op/s; 14227/129615 objects misplaced (10.976%); 46 MiB/s, 729 objects/s recovering 2024-03-21T14:07:45.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:44 smithi003 ceph-mon[29576]: osdmap e139: 8 total, 8 up, 8 in 2024-03-21T14:07:45.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:44 smithi003 ceph-mon[40510]: pgmap v198: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.6 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 912 op/s; 14227/129615 objects misplaced (10.976%); 46 MiB/s, 729 objects/s recovering 2024-03-21T14:07:45.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:44 smithi003 ceph-mon[40510]: osdmap e139: 8 total, 8 up, 8 in 2024-03-21T14:07:45.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:44 smithi118 ceph-mon[33862]: pgmap v198: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.6 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 912 op/s; 14227/129615 objects misplaced (10.976%); 46 MiB/s, 729 objects/s recovering 2024-03-21T14:07:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:44 smithi118 ceph-mon[33862]: osdmap e139: 8 total, 8 up, 8 in 2024-03-21T14:07:45.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:45.247 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:45.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:07:45.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:45.372 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:45.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:45.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:45.497 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:45.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:45.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:45.622 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:45.724 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:45.748 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:45.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:45.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:45.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:45.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:46.008 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:46.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:46.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:46.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:45 smithi118 ceph-mon[33862]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T14:07:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:45 smithi118 ceph-mon[33862]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T14:07:46.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:45 smithi003 ceph-mon[29576]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T14:07:46.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:46.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:45 smithi003 ceph-mon[29576]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T14:07:46.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:45 smithi003 ceph-mon[40510]: osdmap e140: 8 total, 8 up, 8 in 2024-03-21T14:07:46.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:07:46.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:45 smithi003 ceph-mon[40510]: osdmap e141: 8 total, 8 up, 8 in 2024-03-21T14:07:46.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:46.275 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:46.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:46.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.529 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:46.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:46.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:46.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.654 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:46.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:07:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:07:46.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:46.778 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:46.843 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:07:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:07:46.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:46.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:46.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:47.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:47.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:47.115 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:46 smithi003 ceph-mon[29576]: pgmap v201: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.6 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 14227/129615 objects misplaced (10.976%); 20 MiB/s, 327 objects/s recovering 2024-03-21T14:07:47.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:46 smithi003 ceph-mon[40510]: pgmap v201: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.6 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 14227/129615 objects misplaced (10.976%); 20 MiB/s, 327 objects/s recovering 2024-03-21T14:07:47.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:47.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:47.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:46 smithi118 ceph-mon[33862]: pgmap v201: 148 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 140 active+clean; 2.6 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 686 op/s; 14227/129615 objects misplaced (10.976%); 20 MiB/s, 327 objects/s recovering 2024-03-21T14:07:47.212 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:47.212 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:47.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:47.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:47.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:47.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:47.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:47.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:47.575 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:47.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:47.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:47.736 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:47.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:47.847 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:47.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:47.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:48.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:48.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:47 smithi118 ceph-mon[33862]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T14:07:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:48.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:47 smithi003 ceph-mon[29576]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T14:07:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:47 smithi003 ceph-mon[40510]: osdmap e142: 8 total, 8 up, 8 in 2024-03-21T14:07:48.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.331 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:48.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:48.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:07:48.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:48.459 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:48.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:48.592 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:48.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:48.729 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:48.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:48.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:48.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:48.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:48.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:48.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:49.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:49.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:49.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:49.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:49.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:48 smithi118 ceph-mon[33862]: pgmap v204: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 692 op/s; 10336/121317 objects misplaced (8.520%); 47 MiB/s, 756 objects/s recovering 2024-03-21T14:07:49.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:48 smithi003 ceph-mon[29576]: pgmap v204: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 692 op/s; 10336/121317 objects misplaced (8.520%); 47 MiB/s, 756 objects/s recovering 2024-03-21T14:07:49.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:48 smithi003 ceph-mon[40510]: pgmap v204: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 692 op/s; 10336/121317 objects misplaced (8.520%); 47 MiB/s, 756 objects/s recovering 2024-03-21T14:07:49.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:49.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:49.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:49.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:49.508 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:49.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:49.591 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:49.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:49.630 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:49.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:49.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:49.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:49.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:49.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:49.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:07:49.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:50.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:50.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:49 smithi003 ceph-mon[29576]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T14:07:50.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:49 smithi003 ceph-mon[40510]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T14:07:50.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.174 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:50.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:49 smithi118 ceph-mon[33862]: osdmap e143: 8 total, 8 up, 8 in 2024-03-21T14:07:50.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:50.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:50.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:50.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:50.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:50.592 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:50.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:50.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:50.723 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:50.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:moved 7 / 444 (1.57658%) 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:avg 55.5 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.2138 -> 12.4197 (expected baseline 6.96868) 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.0 with 33 -> 32 pgs (0.594595 -> 0.576577 * mean) 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 72 -> 73 pgs (1.2973 -> 1.31532 * mean) 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:oload 110 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0123 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0135 2024-03-21T14:07:50.841 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 weight 0.6562 -> 0.6062 2024-03-21T14:07:50.842 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 weight 0.7302 -> 0.6802 2024-03-21T14:07:50.854 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-03-21T14:07:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:50.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:50.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:50.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.019 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:51.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:51.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:51.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:51.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:50 smithi118 ceph-mon[33862]: pgmap v206: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 614 op/s; 10336/121317 objects misplaced (8.520%); 42 MiB/s, 671 objects/s recovering 2024-03-21T14:07:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:50 smithi118 ceph-mon[33862]: osdmap e144: 8 total, 8 up, 8 in 2024-03-21T14:07:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:50 smithi003 ceph-mon[29576]: pgmap v206: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 614 op/s; 10336/121317 objects misplaced (8.520%); 42 MiB/s, 671 objects/s recovering 2024-03-21T14:07:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:50 smithi003 ceph-mon[29576]: osdmap e144: 8 total, 8 up, 8 in 2024-03-21T14:07:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:50 smithi003 ceph-mon[40510]: pgmap v206: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.5 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 614 op/s; 10336/121317 objects misplaced (8.520%); 42 MiB/s, 671 objects/s recovering 2024-03-21T14:07:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:50 smithi003 ceph-mon[40510]: osdmap e144: 8 total, 8 up, 8 in 2024-03-21T14:07:51.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:50 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:07:50.840+0000 7fa1bb657700 -1 mgr.server reply reply (2) No such file or directory no change 2024-03-21T14:07:51.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:51.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:51.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:51.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:51.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:51.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:51.582 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:51.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:51.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:51.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:51.855 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:51.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:51.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:51.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:51.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:52.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:52.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:51 smithi118 ceph-mon[33862]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:52.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:51 smithi003 ceph-mon[29576]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:52.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:51 smithi003 ceph-mon[40510]: from='client.15009 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:07:52.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:52.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:52.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:52.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:52.504 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:52.539 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:52.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:52.663 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:52.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:52.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:52.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:52.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:52.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:52.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:52.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:53.041 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:53.073 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:53.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:53.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:52 smithi118 ceph-mon[33862]: pgmap v208: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 797 op/s; 9896/116466 objects misplaced (8.497%); 34 MiB/s, 549 objects/s recovering 2024-03-21T14:07:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:53.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:52 smithi003 ceph-mon[29576]: pgmap v208: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 797 op/s; 9896/116466 objects misplaced (8.497%); 34 MiB/s, 549 objects/s recovering 2024-03-21T14:07:53.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:52 smithi003 ceph-mon[40510]: pgmap v208: 148 pgs: 1 active+remapped+backfilling, 1 peering, 4 active+remapped+backfill_toofull, 142 active+clean; 2.4 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 797 op/s; 9896/116466 objects misplaced (8.497%); 34 MiB/s, 549 objects/s recovering 2024-03-21T14:07:53.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:53.335 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:53.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:53.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:53.468 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:53.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:53.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:53.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:53.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:53.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:53.857 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:53.888 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:53.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:53.990 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:54.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:54.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:54.121 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:54.152 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:54.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:54.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:54.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:54.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:54.415 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:54.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:54.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:54.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:54.668 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-21T14:07:54.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-03-21T14:07:54.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:54.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:54.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:54.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:54.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:54.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:54.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:55.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:55.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:55.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:55.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:54 smithi118 ceph-mon[33862]: pgmap v209: 148 pgs: 4 active+remapped+backfill_toofull, 144 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 788 B/s rd, 0 B/s wr, 598 op/s; 9044/109659 objects misplaced (8.247%); 3.1 MiB/s, 49 objects/s recovering 2024-03-21T14:07:55.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:55.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:54 smithi003 ceph-mon[29576]: pgmap v209: 148 pgs: 4 active+remapped+backfill_toofull, 144 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 788 B/s rd, 0 B/s wr, 598 op/s; 9044/109659 objects misplaced (8.247%); 3.1 MiB/s, 49 objects/s recovering 2024-03-21T14:07:55.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:54 smithi003 ceph-mon[40510]: pgmap v209: 148 pgs: 4 active+remapped+backfill_toofull, 144 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 788 B/s rd, 0 B/s wr, 598 op/s; 9044/109659 objects misplaced (8.247%); 3.1 MiB/s, 49 objects/s recovering 2024-03-21T14:07:55.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:55.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:55.362 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:55.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:55.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.496 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:55.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:55.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:55.633 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:55.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:55.764 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:55.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:55.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:55.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:55.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:56.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:56.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:56.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:56.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:56.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:56.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:56.319 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:07:56.319 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:07:56.338 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:07:56.387 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:07:56.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:56.430 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:07:56.448 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:56.474 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:07:56.515 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:07:56.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:56.559 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:07:56.582 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:56.604 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:07:56.605 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:07:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:07:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:07:56.646 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:07:56.652 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:07:56.672 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:56.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:56.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:56.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:56.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:56.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:56.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:07:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:07:56.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:57.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:57.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:57.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:56 smithi003 ceph-mon[29576]: pgmap v210: 148 pgs: 4 active+remapped+backfill_toofull, 144 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 9044/109659 objects misplaced (8.247%); 2.5 MiB/s, 40 objects/s recovering 2024-03-21T14:07:57.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2404834301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:57.168 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:56 smithi003 ceph-mon[40510]: pgmap v210: 148 pgs: 4 active+remapped+backfill_toofull, 144 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 9044/109659 objects misplaced (8.247%); 2.5 MiB/s, 40 objects/s recovering 2024-03-21T14:07:57.168 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2404834301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:57.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:07:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:56 smithi118 ceph-mon[33862]: pgmap v210: 148 pgs: 4 active+remapped+backfill_toofull, 144 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 486 op/s; 9044/109659 objects misplaced (8.247%); 2.5 MiB/s, 40 objects/s recovering 2024-03-21T14:07:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2404834301' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:07:57.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:57.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:57.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:57.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:07:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:57.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:07:57.570 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:57.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:57.718 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:57.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:57.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:57.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:57.883 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:57.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:57.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:58.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:58.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:58.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.162 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:58.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:58.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:58.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:58.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:07:58.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:58.293 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:58.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:07:58.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:07:58.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:07:58.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:58.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:07:58.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:58.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:58.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:07:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:07:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:07:58.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:58.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:58.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:58.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:59.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:07:59.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:07:59.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:58 smithi118 ceph-mon[33862]: pgmap v211: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 7055/94305 objects misplaced (7.481%); 8.3 MiB/s, 133 objects/s recovering 2024-03-21T14:07:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:07:58 smithi118 ceph-mon[33862]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-03-21T14:07:59.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:59.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:58 smithi003 ceph-mon[29576]: pgmap v211: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 7055/94305 objects misplaced (7.481%); 8.3 MiB/s, 133 objects/s recovering 2024-03-21T14:07:59.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:07:58 smithi003 ceph-mon[29576]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-03-21T14:07:59.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:58 smithi003 ceph-mon[40510]: pgmap v211: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s; 7055/94305 objects misplaced (7.481%); 8.3 MiB/s, 133 objects/s recovering 2024-03-21T14:07:59.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:07:58 smithi003 ceph-mon[40510]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-03-21T14:07:59.257 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:59.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:07:59.394 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:07:59.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:59.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:59.658 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:07:59.676 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:07:59.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:07:59.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:07:59.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:07:59.924 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:07:59.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:07:59.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:00.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:00.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:00.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:00.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:08:00.159 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:00.187 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:00.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:00.192 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:00.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:00.318 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:00.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:00.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:00.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:00.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:00.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:00.756 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:00.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:00.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:00.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:00.919 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:00.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:01.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:01.053 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:01.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:01.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:01.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:00 smithi003 ceph-mon[29576]: pgmap v212: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s; 7055/94305 objects misplaced (7.481%); 7.0 MiB/s, 112 objects/s recovering 2024-03-21T14:08:01.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:01.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:00 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:01.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:00 smithi003 ceph-mon[40510]: pgmap v212: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s; 7055/94305 objects misplaced (7.481%); 7.0 MiB/s, 112 objects/s recovering 2024-03-21T14:08:01.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:01.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:00 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:01.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:00 smithi118 ceph-mon[33862]: pgmap v212: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.9 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 901 op/s; 7055/94305 objects misplaced (7.481%); 7.0 MiB/s, 112 objects/s recovering 2024-03-21T14:08:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:00 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:01.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:01.481 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:01.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:01.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:01.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:01.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:01.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:01.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:01.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:01.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:01.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:01.946 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:08:02.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:02.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:02.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:02.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:02.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:01 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3738300729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:08:02.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:08:02.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:01 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3738300729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:08:02.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:01 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:08:02.332 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:02.362 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:01 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3738300729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:08:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:01 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:08:02.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:02.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:02.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:02.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:02.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:02.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:02.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:02.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:02.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:02.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:02.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:02.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[29576]: pgmap v213: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 972 op/s; 5409/87984 objects misplaced (6.148%); 13 MiB/s, 203 objects/s recovering 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[29576]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[40510]: pgmap v213: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 972 op/s; 5409/87984 objects misplaced (6.148%); 13 MiB/s, 203 objects/s recovering 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:08:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:02 smithi003 ceph-mon[40510]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T14:08:03.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:03.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:03.175 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:03.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:03.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:03.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:03.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:02 smithi118 ceph-mon[33862]: pgmap v213: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.8 GiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 972 op/s; 5409/87984 objects misplaced (6.148%); 13 MiB/s, 203 objects/s recovering 2024-03-21T14:08:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:02 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:02 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:08:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:02 smithi118 ceph-mon[33862]: osdmap e145: 8 total, 8 up, 8 in 2024-03-21T14:08:03.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:03.637 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:03.668 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:03.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:03.800 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:03.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:03.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:03.939 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:03.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:04.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:04.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:04.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:04.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:03 smithi003 ceph-mon[29576]: pgmap v215: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s; 3798/80025 objects misplaced (4.746%); 20 MiB/s, 316 objects/s recovering 2024-03-21T14:08:04.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:03 smithi003 ceph-mon[40510]: pgmap v215: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s; 3798/80025 objects misplaced (4.746%); 20 MiB/s, 316 objects/s recovering 2024-03-21T14:08:04.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:04.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:04.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:04.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:03 smithi118 ceph-mon[33862]: pgmap v215: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s; 3798/80025 objects misplaced (4.746%); 20 MiB/s, 316 objects/s recovering 2024-03-21T14:08:04.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:04.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:04.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:moved 0 / 444 (0%) 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:avg 55.5 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:stddev 11.2138 -> 11.2138 (expected baseline 6.96868) 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:min osd.0 with 33 -> 33 pgs (0.594595 -> 0.594595 * mean) 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:max osd.1 with 72 -> 72 pgs (1.2973 -> 1.2973 * mean) 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:oload 1000 2024-03-21T14:08:04.654 INFO:teuthology.orchestra.run.smithi003.stdout:max_change 0.05 2024-03-21T14:08:04.655 INFO:teuthology.orchestra.run.smithi003.stdout:max_change_osds 4 2024-03-21T14:08:04.655 INFO:teuthology.orchestra.run.smithi003.stdout:average_utilization 0.0086 2024-03-21T14:08:04.655 INFO:teuthology.orchestra.run.smithi003.stdout:overload_utilization 0.0857 2024-03-21T14:08:04.660 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:04.662 INFO:teuthology.orchestra.run.smithi003.stderr:no change 2024-03-21T14:08:04.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:04.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:04.952 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:04.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:04.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:05.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:05.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:05.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:08:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:08:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:08:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:08:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:08:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[29576]: from='client.24781 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:08:05.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:08:05.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:08:05.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:08:05.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:08:05.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:08:05.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:04 smithi003 ceph-mon[40510]: from='client.24781 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:08:05.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:05.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:05.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:05.395 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:05.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:04 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:08:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:04 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:08:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:04 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:08:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:04 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:08:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:04 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:08:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:04 smithi118 ceph-mon[33862]: from='client.24781 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:08:05.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:05.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:05.637 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:05.665 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:05.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:05.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:05.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:05.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:05.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:05.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:05.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:08:06.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:06.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:06.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:06.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[29576]: 7.8 scrub starts 2024-03-21T14:08:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[29576]: pgmap v216: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s; 3798/80025 objects misplaced (4.746%); 20 MiB/s, 316 objects/s recovering 2024-03-21T14:08:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:06.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:06.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[40510]: 7.8 scrub starts 2024-03-21T14:08:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[40510]: pgmap v216: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s; 3798/80025 objects misplaced (4.746%); 20 MiB/s, 316 objects/s recovering 2024-03-21T14:08:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:06.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:05 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:06.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:06.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219781 7.d e145/89 7:b1a384cb:::benchmark_data_smithi003_80362_object98764:head v 145'13517, mlcod=145'13515)", 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.380716+0000", 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0066490569999999999, 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066794560000000003, 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380716+0000", 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380716+0000", 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380716+0000", 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4799999999999997e-07 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380726+0000", 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9199999999999999e-06 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380736+0000", 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8390000000000002e-06 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380743+0000", 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2910000000000002e-06 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380844+0000", 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100408 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380866+0000", 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2188000000000002e-05 2024-03-21T14:08:06.394 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219782 7.d e145/89 7:bafc148a:::benchmark_data_smithi003_80362_object98765:head v 145'13518, mlcod=145'13515)", 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.382837+0000", 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0045278369999999998, 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046297780000000002, 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.382837+0000", 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.382837+0000", 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.382838+0000", 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0100000000000004e-07 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.382846+0000", 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.422e-06 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.382856+0000", 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6749999999999997e-06 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.382863+0000", 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2590000000000003e-06 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.384560+0000", 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001696534 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.384589+0000", 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9139999999999999e-05 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:219771 7.9 7:9be2a61e:::benchmark_data_smithi003_80362_object98754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.363917+0000", 2024-03-21T14:08:06.398 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.023447454999999999, 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.023620842999999999, 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219771 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363917+0000", 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363917+0000", 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363919+0000", 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.945e-06 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363932+0000", 2024-03-21T14:08:06.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2909e-05 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363945+0000", 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2425999999999999e-05 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363953+0000", 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6240000000000008e-06 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.366705+0000", 2024-03-21T14:08:06.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027520449999999998 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.366840+0000", 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013475800000000001 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.366988+0000", 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014790199999999999 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380602+0000", 2024-03-21T14:08:06.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013614574000000001 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380681+0000", 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8992999999999994e-05 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.383802+0000", 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031209010000000001 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPushReply(7.9 145/130 [PushReplyOp(7:9ce6b04a:::benchmark_data_smithi003_80362_object106353:head)])", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.385119+0000", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0022455050000000001, 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025145340000000001, 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385119+0000", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385119+0000", 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385120+0000", 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7599999999999998e-07 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385123+0000", 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4460000000000001e-06 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385130+0000", 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5570000000000001e-06 2024-03-21T14:08:06.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385137+0000", 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5820000000000004e-06 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219785 7.d e145/89)", 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.385403+0000", 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0019614089999999999, 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022786849999999999, 2024-03-21T14:08:06.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385403+0000", 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385403+0000", 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385404+0000", 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8100000000000002e-07 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385413+0000", 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1740000000000006e-06 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385424+0000", 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0458e-05 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.385431+0000", 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7060000000000004e-06 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(7.7 145/130 [PushOp(7:edf6b872:::benchmark_data_smithi003_80362_object107858:head, version: 109'6653, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:edf6b872:::benchmark_data_smithi003_80362_object107858:head@109'6653, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.386393+0000", 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00097170999999999998, 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001365525, 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386393+0000", 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386393+0000", 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386393+0000", 2024-03-21T14:08:06.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8000000000000001e-07 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386468+0000", 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4634000000000004e-05 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386488+0000", 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9933999999999999e-05 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386506+0000", 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7725000000000001e-05 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.411 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219786 7.7 e145/130 7:e9401fac:::benchmark_data_smithi003_80362_object98769:head v 145'16077, mlcod=145'16074)", 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.386624+0000", 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00074043100000000005, 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001184376, 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386624+0000", 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386624+0000", 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386625+0000", 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6000000000000003e-07 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386631+0000", 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5820000000000001e-06 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386641+0000", 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0198e-05 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386647+0000", 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4320000000000004e-06 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386834+0000", 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000187295 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.386945+0000", 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110522 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219763 7.1 e145/140 7:88668862:::benchmark_data_smithi003_80362_object98746:head v 145'13710, mlcod=145'13707)", 2024-03-21T14:08:06.415 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.367087+0000", 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.020277671000000001, 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020783564000000001, 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367087+0000", 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367087+0000", 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367088+0000", 2024-03-21T14:08:06.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8999999999999997e-07 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367092+0000", 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6020000000000002e-06 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367098+0000", 2024-03-21T14:08:06.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7989999999999999e-06 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367103+0000", 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7369999999999997e-06 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.376744+0000", 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096412100000000008 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.376762+0000", 2024-03-21T14:08:06.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7720999999999999e-05 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219764 7.1 e145/140 7:80eb1f13:::benchmark_data_smithi003_80362_object98747:head v 145'13711, mlcod=145'13707)", 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.367111+0000", 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.020254030999999999, 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020831505, 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367111+0000", 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367111+0000", 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367111+0000", 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.04e-07 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367114+0000", 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.027e-06 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367118+0000", 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.476e-06 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367122+0000", 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9770000000000002e-06 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.378708+0000", 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01158583 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.378727+0000", 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8663000000000001e-05 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219770 7.1 e145/140 7:8e1c0bcd:::benchmark_data_smithi003_80362_object98753:head v 145'13712, mlcod=145'13707)", 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.367128+0000", 2024-03-21T14:08:06.423 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.020236971999999999, 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020888275000000001, 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367128+0000", 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367128+0000", 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367128+0000", 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.49e-07 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367134+0000", 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7359999999999998e-06 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367138+0000", 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.386e-06 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.367142+0000", 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7890000000000001e-06 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380710+0000", 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013568349 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380727+0000", 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7204999999999999e-05 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.426 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:219761 7.9 7:97ea6196:::benchmark_data_smithi003_80362_object98744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.355093+0000", 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.032272092000000002, 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032988667999999999, 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219761 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.355093+0000", 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.355093+0000", 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.355095+0000", 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8619999999999999e-06 2024-03-21T14:08:06.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.355102+0000", 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0029999999999997e-06 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.355115+0000", 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3617e-05 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.355124+0000", 2024-03-21T14:08:06.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0599999999999997e-06 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.361365+0000", 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062406379999999997 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.361498+0000", 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133459 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.363059+0000", 2024-03-21T14:08:06.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015602039999999999 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.366661+0000", 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036020169999999999 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380643+0000", 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013982919 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.383763+0000", 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003119494 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219777 7.1 e145/140 7:8ab3af74:::benchmark_data_smithi003_80362_object98760:head v 145'13713, mlcod=145'13708)", 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.375138+0000", 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.012227103, 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013036791000000001, 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:06.432 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.375138+0000", 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.375138+0000", 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.375138+0000", 2024-03-21T14:08:06.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3999999999999998e-07 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.375141+0000", 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1250000000000001e-06 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.375146+0000", 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9420000000000003e-06 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.375150+0000", 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4700000000000004e-06 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.383845+0000", 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0086947380000000005 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.383864+0000", 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9012e-05 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:219779 7.b 7:d8425d30:::benchmark_data_smithi003_80362_object98762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:06.380205+0000", 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0071595260000000003, 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080330230000000002, 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219779 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:06.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380205+0000", 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380205+0000", 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380206+0000", 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1820000000000001e-06 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380211+0000", 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7219999999999999e-06 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380220+0000", 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0070000000000003e-06 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380227+0000", 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6800000000000004e-06 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380467+0000", 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000239814 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380608+0000", 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014107099999999999 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.380733+0000", 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124838 2024-03-21T14:08:06.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.383258+0000", 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002525411 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:06.383580+0000", 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032228699999999999 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 13 2024-03-21T14:08:06.440 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:08:06.441 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:05 smithi118 ceph-mon[33862]: 7.8 scrub starts 2024-03-21T14:08:06.441 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:05 smithi118 ceph-mon[33862]: pgmap v216: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 988 op/s; 3798/80025 objects misplaced (4.746%); 20 MiB/s, 316 objects/s recovering 2024-03-21T14:08:06.441 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:05 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:06.441 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:05 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:06.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:06.474 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:06.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:06.575 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:06.606 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:06.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:06.653 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:08:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:08:06.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:06.741 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:06.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:06.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:06.875 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:06.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:06.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:08:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:08:06.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:07.009 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:07.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:07.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:07.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:07.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:07.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[29576]: 7.8 scrub ok 2024-03-21T14:08:07.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:08:07.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:08:07.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:07.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[40510]: 7.8 scrub ok 2024-03-21T14:08:07.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:08:07.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:08:07.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:06 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:07.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:07.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:07.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:07.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:07.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:06 smithi118 ceph-mon[33862]: 7.8 scrub ok 2024-03-21T14:08:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:06 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:08:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:06 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:08:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:06 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:07.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:07.549 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:07.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:07.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:07.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:07.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:07.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:07.934 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:07.966 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:07.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:07.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:08.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:08.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:08.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:08.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:08.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:08.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:07 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:08:08.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:08 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:08:08.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:08 smithi003 ceph-mon[29576]: pgmap v217: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.4 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 910 op/s; 1254/67011 objects misplaced (1.871%); 28 MiB/s, 448 objects/s recovering 2024-03-21T14:08:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:07 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:08:08.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:08 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:08:08.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:08 smithi003 ceph-mon[40510]: pgmap v217: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.4 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 910 op/s; 1254/67011 objects misplaced (1.871%); 28 MiB/s, 448 objects/s recovering 2024-03-21T14:08:08.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:08.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:07 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:08:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:07 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:08:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:07 smithi118 ceph-mon[33862]: pgmap v217: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.4 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 910 op/s; 1254/67011 objects misplaced (1.871%); 28 MiB/s, 448 objects/s recovering 2024-03-21T14:08:08.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:08.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:08.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:08.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:08.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:08.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:08.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:08.869 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:08.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:08.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:08.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:08.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:09.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:09.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:09.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:09.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:09.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:09.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:09.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:09 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:08:09.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:09 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:08:09.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:09 smithi118 ceph-mon[33862]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T14:08:09.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:09.385 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:09.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:09.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:09 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:08:09.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:09 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:08:09.469 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:09 smithi003 ceph-mon[29576]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T14:08:09.469 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:09 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:08:09.470 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:09 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:08:09.470 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:09 smithi003 ceph-mon[40510]: osdmap e146: 8 total, 8 up, 8 in 2024-03-21T14:08:09.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:09.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:09.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:09.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:09.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:09.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:09.760 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:09.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:09.885 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:09.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:09.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.012 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:10.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:10.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:10.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:10.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:10.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:10.373 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[40510]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[40510]: pgmap v220: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.4 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 586 op/s; 1254/67011 objects misplaced (1.871%); 18 MiB/s, 286 objects/s recovering 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[29576]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T14:08:10.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[29576]: pgmap v220: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.4 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 586 op/s; 1254/67011 objects misplaced (1.871%); 18 MiB/s, 286 objects/s recovering 2024-03-21T14:08:10.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:08:10.375 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:10 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:08:10.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:10 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:08:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:10 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:08:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:10 smithi118 ceph-mon[33862]: osdmap e147: 8 total, 8 up, 8 in 2024-03-21T14:08:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:10 smithi118 ceph-mon[33862]: pgmap v220: 148 pgs: 4 active+remapped+backfilling, 144 active+clean; 1.4 GiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 586 op/s; 1254/67011 objects misplaced (1.871%); 18 MiB/s, 286 objects/s recovering 2024-03-21T14:08:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:10 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:08:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:10 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:08:10.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:10.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:10.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:08:10.577 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:10.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:10.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:10.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:10.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:10.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:10.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:10.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:11.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:11.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:11.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:11.245 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:11.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:11.380 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:11.414 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:11.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:11 smithi118 ceph-mon[33862]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T14:08:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:11 smithi003 ceph-mon[29576]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T14:08:11.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:11 smithi003 ceph-mon[40510]: osdmap e148: 8 total, 8 up, 8 in 2024-03-21T14:08:11.517 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:11.548 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:11.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:11.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.683 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:11.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:11.781 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:11.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:11.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:11.853 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:11.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:11.944 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:11.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:12.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:12.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:12.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:12.285 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:12.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:08:12.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:08:12.295 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[29576]: pgmap v222: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 964 op/s; 655/62691 objects misplaced (1.045%); 28 MiB/s, 447 objects/s recovering 2024-03-21T14:08:12.296 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:08:12.297 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:12 smithi003 ceph-mon[40510]: pgmap v222: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 964 op/s; 655/62691 objects misplaced (1.045%); 28 MiB/s, 447 objects/s recovering 2024-03-21T14:08:12.324 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:08:12.324 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:08:12.324 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-21T14:08:12.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-21T14:08:12.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:08:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:08:12.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:12 smithi118 ceph-mon[33862]: pgmap v222: 148 pgs: 2 active+remapped+backfilling, 146 active+clean; 1.3 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 964 op/s; 655/62691 objects misplaced (1.045%); 28 MiB/s, 447 objects/s recovering 2024-03-21T14:08:12.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:12.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:12.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:12.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:12.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:12.706 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:08:12.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:12.750 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:08:12.797 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:08:12.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:12.840 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:08:12.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:12.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:12.886 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:08:12.923 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:08:12.967 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:08:12.978 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:13.008 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:08:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.012 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:13.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:08:13.013 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:08:13.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:13.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:13.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.294 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:13.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:13.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1625871322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:13.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:13.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:08:13.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:08:13.446 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1625871322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:08:13.447 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:08:13.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:08:13.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:13.448 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1625871322' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:13.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:13.566 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:13.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:13.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:13.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:13.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:13.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:13.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:13.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:13.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:14.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:14.124 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:14.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:14.258 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:14.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:14.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.399 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:14.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:08:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:08:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:08:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:08:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:08:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:08:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:14 smithi118 ceph-mon[33862]: pgmap v223: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 569 op/s; 12 MiB/s, 196 objects/s recovering 2024-03-21T14:08:14.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:08:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:08:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:08:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[29576]: pgmap v223: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 569 op/s; 12 MiB/s, 196 objects/s recovering 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:08:14.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:08:14.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:08:14.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:08:14.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:14 smithi003 ceph-mon[40510]: pgmap v223: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 569 op/s; 12 MiB/s, 196 objects/s recovering 2024-03-21T14:08:14.545 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:14.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:14.681 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:14.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:14.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:14.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:14.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:14.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:14.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:15.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:15.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:15.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:15.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:15.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:15.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:15.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:15.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:15 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:08:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:15 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:08:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:15 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:08:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:15 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:08:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:08:15.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:08:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:15 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:08:15.509 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:15.541 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:15.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:15.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:15.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:15.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:15.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:15.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:15.920 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:15.945 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:15.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:15.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:15.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:16.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:16.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:16.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:16.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:08:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:08:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: pgmap v224: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 703 B/s rd, 0 B/s wr, 470 op/s; 10 MiB/s, 162 objects/s recovering 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]: dispatch 2024-03-21T14:08:16.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]: dispatch 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]: dispatch 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]: dispatch 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:08:16.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: pgmap v224: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 703 B/s rd, 0 B/s wr, 470 op/s; 10 MiB/s, 162 objects/s recovering 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]: dispatch 2024-03-21T14:08:16.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]: dispatch 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]: dispatch 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]: dispatch 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:16.258 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:16.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:16.386 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:08:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:08:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:08:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:08:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: pgmap v224: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 703 B/s rd, 0 B/s wr, 470 op/s; 10 MiB/s, 162 objects/s recovering 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]: dispatch 2024-03-21T14:08:16.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]: dispatch 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]: dispatch 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]: dispatch 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:16.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:16.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:16.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:16.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:16.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:08:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:08:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:08:16.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.797 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:16.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:16.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:16.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:08:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:08:16.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:16.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:17.040 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:17.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:17.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:17.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:17.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:17.347 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 3.1a scrub starts 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 3.1a scrub ok 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:08:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]': finished 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T14:08:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:17.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:17.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 3.1a scrub starts 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 3.1a scrub ok 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]': finished 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]': finished 2024-03-21T14:08:17.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]': finished 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]': finished 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]': finished 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 3.1a scrub starts 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 3.1a scrub ok 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:08:17.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 0]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [1, 0]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1e", "id": [1, 2]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.0", "id": [3, 6]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.b", "id": [3, 0]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.d", "id": [2, 6]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.13", "id": [2, 1]}]': finished 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: osdmap e149: 8 total, 8 up, 8 in 2024-03-21T14:08:17.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:17.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:17.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:17.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:17.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:17.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:17.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:17.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:17.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:17.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:18.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:18.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:18.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: pgmap v227: 148 pgs: 7 peering, 141 active+clean; 805 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 10 MiB/s, 162 objects/s recovering 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:08:18.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: pgmap v227: 148 pgs: 7 peering, 141 active+clean; 805 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 10 MiB/s, 162 objects/s recovering 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:08:18.240 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:18 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:08:18.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:18.295 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:18.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: osdmap e150: 8 total, 8 up, 8 in 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: pgmap v227: 148 pgs: 7 peering, 141 active+clean; 805 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.28k op/s; 10 MiB/s, 162 objects/s recovering 2024-03-21T14:08:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:08:18.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:18 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:08:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:18.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:18.458 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:08:18.458 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:08:18.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:08:18.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:18.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:18.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:18.709 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:18.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:18.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:18.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:18.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.005 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:19.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:19.106 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:19.140 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:19.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:19.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:19.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:19.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:19.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:19.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:08:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:19 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:08:19.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:08:19.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:08:19.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:08:19.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:08:19.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:08:19.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:19 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:08:19.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:19.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:19.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:19.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:19.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:19.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:19.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:19.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:20.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:20.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:20.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:20.144 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:20.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:20.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:20.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:20.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: pgmap v228: 148 pgs: 7 peering, 141 active+clean; 805 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 978 op/s; 4.8 MiB/s, 77 objects/s recovering 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1439858118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:08:20.336 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:08:20.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: pgmap v228: 148 pgs: 7 peering, 141 active+clean; 805 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 978 op/s; 4.8 MiB/s, 77 objects/s recovering 2024-03-21T14:08:20.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:08:20.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:08:20.337 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1439858118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:20.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: pgmap v228: 148 pgs: 7 peering, 141 active+clean; 805 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 978 op/s; 4.8 MiB/s, 77 objects/s recovering 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:08:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1439858118' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:08:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:20.553 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:20.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:20.715 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:20.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:20.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:20.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:20.996 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:20.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:21.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:21.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:21.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:21.262 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:21.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:21.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:21.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:21.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:08:21.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:08:21.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:08:21.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1439858118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:08:21.424 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: 5.1e deep-scrub starts 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[29576]: 5.1e deep-scrub ok 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1439858118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: 5.1e deep-scrub starts 2024-03-21T14:08:21.425 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:21 smithi003 ceph-mon[40510]: 5.1e deep-scrub ok 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1439858118' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: osdmap e151: 8 total, 8 up, 8 in 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: 5.1e deep-scrub starts 2024-03-21T14:08:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:21 smithi118 ceph-mon[33862]: 5.1e deep-scrub ok 2024-03-21T14:08:21.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:21.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:21.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:21.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:21.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:21.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:21.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:21.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:21.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:21.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:21.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:22.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:22.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:22.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:22.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:22.246 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:22.348 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:22.384 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:22.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:22.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:22 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:08:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:22 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:08:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:22 smithi118 ceph-mon[33862]: pgmap v230: 148 pgs: 4 peering, 144 active+clean; 650 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s; 36 B/s, 1 objects/s recovering 2024-03-21T14:08:22.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:22 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:08:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:22 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:08:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:22 smithi003 ceph-mon[29576]: pgmap v230: 148 pgs: 4 peering, 144 active+clean; 650 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s; 36 B/s, 1 objects/s recovering 2024-03-21T14:08:22.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:22 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:08:22.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:22 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:08:22.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:22 smithi003 ceph-mon[40510]: pgmap v230: 148 pgs: 4 peering, 144 active+clean; 650 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.39k op/s; 36 B/s, 1 objects/s recovering 2024-03-21T14:08:22.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:22.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:22.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:22.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:22.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:08:22.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:22.782 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:22.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:22.910 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:22.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:22.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:23.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:23.072 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:23.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:23.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:23.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:23.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:23.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:08:23.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[29576]: 4.a deep-scrub starts 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[29576]: 4.a deep-scrub ok 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[40510]: 4.a deep-scrub starts 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[40510]: 4.a deep-scrub ok 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:08:23.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:23 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:08:23.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:23.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-21T14:08:23.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.496 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:23.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:23.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:23.625 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:23.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:23 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:08:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:23 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:08:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:23 smithi118 ceph-mon[33862]: 4.a deep-scrub starts 2024-03-21T14:08:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:23 smithi118 ceph-mon[33862]: 4.a deep-scrub ok 2024-03-21T14:08:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:23 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:08:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:23 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:08:23.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:23.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:23.876 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:23.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:08:23.910 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:23.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:24.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:24.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:24.142 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:24.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:24.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:24.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:24.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:08:24.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:08:24.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[40510]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T14:08:24.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[40510]: pgmap v232: 148 pgs: 148 active+clean; 450 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 913 op/s; 35 B/s, 2 objects/s recovering 2024-03-21T14:08:24.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:24.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:24.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:24.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:24.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:24.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:24.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:24.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:08:24.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:08:24.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[29576]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T14:08:24.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:24 smithi003 ceph-mon[29576]: pgmap v232: 148 pgs: 148 active+clean; 450 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 913 op/s; 35 B/s, 2 objects/s recovering 2024-03-21T14:08:24.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:24 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:24 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:24 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:08:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:24 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:08:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:24 smithi118 ceph-mon[33862]: osdmap e152: 8 total, 8 up, 8 in 2024-03-21T14:08:24.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:24 smithi118 ceph-mon[33862]: pgmap v232: 148 pgs: 148 active+clean; 450 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 822 B/s rd, 0 B/s wr, 913 op/s; 35 B/s, 2 objects/s recovering 2024-03-21T14:08:24.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:24.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:24.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:24.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:24.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:25.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:25.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:25.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:25.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:25.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 5.18 scrub starts 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 5.18 scrub ok 2024-03-21T14:08:25.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 5.18 scrub starts 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 5.18 scrub ok 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:08:25.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:08:25.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:08:25.248 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:25 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:08:25.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.375 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:25.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:25.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:25.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:25.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:25.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:25.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 5.18 scrub starts 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 5.18 scrub ok 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:08:25.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:25 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:08:25.754 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:25.786 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:25.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:25.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:25.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:25.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:26.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:26.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:26.078 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:26.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:26.195 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:26.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:08:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:08:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:08:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:08:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: 4.2 deep-scrub starts 2024-03-21T14:08:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: 4.2 deep-scrub ok 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[29576]: pgmap v233: 148 pgs: 148 active+clean; 450 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 710 op/s; 27 B/s, 1 objects/s recovering 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: 4.2 deep-scrub starts 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: 4.2 deep-scrub ok 2024-03-21T14:08:26.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-mon[40510]: pgmap v233: 148 pgs: 148 active+clean; 450 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 710 op/s; 27 B/s, 1 objects/s recovering 2024-03-21T14:08:26.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:26.326 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:26.428 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:26.462 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: 4.2 deep-scrub starts 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: 4.2 deep-scrub ok 2024-03-21T14:08:26.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-mon[33862]: pgmap v233: 148 pgs: 148 active+clean; 450 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 710 op/s; 27 B/s, 1 objects/s recovering 2024-03-21T14:08:26.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:26.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:26.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:26.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:26.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:26.752 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:08:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:08:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:08:26.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:26.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:26.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:26.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:08:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:08:26.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:27.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:27.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:27.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:27.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:27.144 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:27.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:27.263 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:27.263 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:27.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[29576]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T14:08:27.264 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:27.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:08:27.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:27.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:27.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[40510]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T14:08:27.264 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:27.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:27.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:27.308 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:08:27.347 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:08:27.390 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:27.401 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:08:27.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:27.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:27.440 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:08:27.478 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:08:27.522 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:08:27.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:27.569 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:08:27.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:27.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:27.586 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:08:27.586 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:08:27.586 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:08:27.586 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:08:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:08:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:08:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:08:27.587 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:08:27.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:27.680 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 121014 objects 2024-03-21T14:08:27.680 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :136.872 2024-03-21T14:08:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:27 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:27 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:27 smithi118 ceph-mon[33862]: osdmap e153: 8 total, 8 up, 8 in 2024-03-21T14:08:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:27.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:27.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:27.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:27.925 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:27.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:27.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:28.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:28.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:28.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:28.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:28.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.14158.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:00:14.324848+0000", 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 493.54352753500001, 2024-03-21T14:08:28.323 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.62369602999999996, 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.14158", 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/384540368", 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 1 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324848+0000", 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.324 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324848+0000", 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324849+0000", 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0249999999999999e-06 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324861+0000", 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1895e-05 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.325 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324871+0000", 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9229999999999997e-06 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324880+0000", 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.533e-06 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324948+0000", 2024-03-21T14:08:28.326 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7484999999999995e-05 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.324956+0000", 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4090000000000003e-06 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.947207+0000", 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.62225083299999995 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.327 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.947379+0000", 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017184599999999999 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.947561+0000", 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018233299999999999 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.948289+0000", 2024-03-21T14:08:28.328 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00072761099999999997 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.948339+0000", 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9914999999999999e-05 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.948476+0000", 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013688200000000001 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.948510+0000", 2024-03-21T14:08:28.329 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4746000000000002e-05 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:00:14.948544+0000", 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3593999999999997e-05 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:79678 7.b 7:de2b972b:::benchmark_data_smithi003_80362_object79677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:29.258490+0000", 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 178.60988570399999, 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.074769474000000002, 2024-03-21T14:08:28.330 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 79678 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.258490+0000", 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.331 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.258490+0000", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.258491+0000", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4410000000000001e-06 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.258513+0000", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2209999999999999e-05 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.258524+0000", 2024-03-21T14:08:28.332 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0822000000000001e-05 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.258532+0000", 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6459999999999993e-06 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.263119+0000", 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045875170000000002 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.263211+0000", 2024-03-21T14:08:28.333 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1433000000000004e-05 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.263306+0000", 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5635000000000002e-05 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.267370+0000", 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040636719999999999 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.267958+0000", 2024-03-21T14:08:28.334 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00058803600000000001 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.333144+0000", 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.065185921999999993 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.333211+0000", 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6409999999999996e-05 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.335 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:29.333259+0000", 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8730000000000003e-05 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:105840 7.d e109/89 7:bd4bae29:::benchmark_data_smithi003_80362_object105839:head v 109'6500, mlcod=109'6498)", 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.474120+0000", 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.394255456, 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.092444702000000004, 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.336 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.474120+0000", 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.474120+0000", 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.474121+0000", 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1769999999999999e-06 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.337 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.474194+0000", 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2934000000000003e-05 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.474206+0000", 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1649e-05 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.474214+0000", 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.32e-06 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.338 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.475134+0000", 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091940900000000005 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.475157+0000", 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3034e-05 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.566516+0000", 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091359770000000007 2024-03-21T14:08:28.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.566565+0000", 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8408999999999997e-05 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:105841 7.d e109/89 7:ba2e495a:::benchmark_data_smithi003_80362_object105840:head v 109'6501, mlcod=109'6498)", 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.476937+0000", 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.391438314, 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.092584423999999999, 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.340 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.476937+0000", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.476937+0000", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.476938+0000", 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7199999999999998e-07 2024-03-21T14:08:28.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.477050+0000", 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112191 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.477062+0000", 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1969e-05 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.477069+0000", 2024-03-21T14:08:28.342 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0119999999999999e-06 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.479453+0000", 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023836600000000001 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.479476+0000", 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3096e-05 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.569474+0000", 2024-03-21T14:08:28.343 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.089997938999999999 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.569522+0000", 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7784999999999997e-05 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:105846 7.b 7:d2f4c0a9:::benchmark_data_smithi003_80362_object105845:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.480069+0000", 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.388306271, 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093239978000000001, 2024-03-21T14:08:28.344 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105846 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480069+0000", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480069+0000", 2024-03-21T14:08:28.345 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480071+0000", 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.821e-06 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480096+0000", 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5062000000000002e-05 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.346 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480109+0000", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2798e-05 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480118+0000", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6619999999999994e-06 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563643+0000", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083525164999999998 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563836+0000", 2024-03-21T14:08:28.347 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000192981 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.564006+0000", 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169918 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.572404+0000", 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0083979610000000007 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.348 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.572453+0000", 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9709999999999997e-05 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.572465+0000", 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1259e-05 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573103+0000", 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063848099999999999 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.349 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573309+0000", 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020615999999999999 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:105847 7.2 e109/89 7:4a69db23:::benchmark_data_smithi003_80362_object105846:head v 109'6686, mlcod=109'6684)", 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.480297+0000", 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.38807857099999, 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083359550000000004, 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.350 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480297+0000", 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480297+0000", 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480298+0000", 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.33e-07 2024-03-21T14:08:28.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480321+0000", 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3555000000000001e-05 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480331+0000", 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9830000000000008e-06 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480338+0000", 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6760000000000001e-06 2024-03-21T14:08:28.352 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480558+0000", 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021968000000000001 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480669+0000", 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111408 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563611+0000", 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.082941816000000002 2024-03-21T14:08:28.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563656+0000", 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5498999999999999e-05 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:105845 7.d e109/89 7:bf90065f:::benchmark_data_smithi003_80362_object105844:head v 109'6502, mlcod=109'6498)", 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.480435+0000", 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.38794007000001, 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091802649, 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480435+0000", 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480435+0000", 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.355 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480436+0000", 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5899999999999996e-07 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480481+0000", 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4481000000000003e-05 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480489+0000", 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4509999999999992e-06 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.356 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.480496+0000", 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5470000000000002e-06 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563620+0000", 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.083124940999999994 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563644+0000", 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3173999999999999e-05 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.357 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.572196+0000", 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085518759999999999 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.572238+0000", 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2519999999999999e-05 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:105849 7.4 e109/89 7:2a74527d:::benchmark_data_smithi003_80362_object105848:head v 109'6733, mlcod=109'6730)", 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.481507+0000", 2024-03-21T14:08:28.358 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.38686848699999, 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088298188, 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481507+0000", 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481507+0000", 2024-03-21T14:08:28.359 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481508+0000", 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.114e-06 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481531+0000", 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2405999999999999e-05 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481541+0000", 2024-03-21T14:08:28.360 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0973e-05 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481549+0000", 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4159999999999998e-06 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481629+0000", 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0520999999999997e-05 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.481650+0000", 2024-03-21T14:08:28.361 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0088000000000001e-05 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.569751+0000", 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.088101344999999998 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.569805+0000", 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4324999999999999e-05 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.362 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:105851 7.b 7:d5f8101c:::benchmark_data_smithi003_80362_object105850:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.482035+0000", 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.386340582, 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091664171000000003, 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 105851 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.363 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.482035+0000", 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.482035+0000", 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.482037+0000", 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7e-06 2024-03-21T14:08:28.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.482060+0000", 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3540000000000002e-05 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563578+0000", 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.081517453000000004 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563595+0000", 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6991999999999999e-05 2024-03-21T14:08:28.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.566449+0000", 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028542020000000001 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.566606+0000", 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015676399999999999 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.566785+0000", 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000179116 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573361+0000", 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065767619999999999 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573535+0000", 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017312299999999999 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573604+0000", 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9061999999999997e-05 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573660+0000", 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6294e-05 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.573699+0000", 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9162999999999998e-05 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.368 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:105859 7.0 e109/89 7:0469f6b7:::benchmark_data_smithi003_80362_object105858:head v 109'6595, mlcod=109'6593)", 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:55.488008+0000", 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.38036760099999, 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.080341553999999996, 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.488008+0000", 2024-03-21T14:08:28.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.488008+0000", 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.488016+0000", 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3680000000000002e-06 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.488064+0000", 2024-03-21T14:08:28.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7413000000000002e-05 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563586+0000", 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.075521928000000002 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563597+0000", 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.187e-05 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563699+0000", 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101751 2024-03-21T14:08:28.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.563718+0000", 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9238000000000001e-05 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.568296+0000", 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004577299 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:55.568349+0000", 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3687000000000003e-05 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:134364 7.0 e118/89 7:023058ab:::benchmark_data_smithi003_80362_object13347:head v 118'8340, mlcod=118'8338)", 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:24.351589+0000", 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.516786032, 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.089506453999999999, 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351589+0000", 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351589+0000", 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351591+0000", 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1459999999999999e-06 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351596+0000", 2024-03-21T14:08:28.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5639999999999996e-06 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351606+0000", 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0001e-05 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351613+0000", 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9419999999999998e-06 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351675+0000", 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2247000000000002e-05 2024-03-21T14:08:28.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.351695+0000", 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9293e-05 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.441053+0000", 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.089357877000000002 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.441096+0000", 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3383999999999997e-05 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:134367 7.0 e118/89 7:0e4e9baf:::benchmark_data_smithi003_80362_object13350:head v 118'8341, mlcod=118'8338)", 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:24.354899+0000", 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.513476889, 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.096384453999999994, 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.354899+0000", 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.354899+0000", 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.354900+0000", 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8100000000000001e-07 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.354907+0000", 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5390000000000002e-06 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.423956+0000", 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.069048641999999993 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.423981+0000", 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5171e-05 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.428814+0000", 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00483293 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.428840+0000", 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5707999999999998e-05 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.451195+0000", 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022355114999999998 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.451283+0000", 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8368000000000005e-05 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:134374 7.d e118/89 7:b758ccef:::benchmark_data_smithi003_80362_object13357:head v 118'8284, mlcod=118'8282)", 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:24.362251+0000", 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.506124063, 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.079402847999999998, 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.362251+0000", 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.362251+0000", 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.362252+0000", 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.074e-06 2024-03-21T14:08:28.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.362262+0000", 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4530000000000008e-06 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.424011+0000", 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.061748997 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.424018+0000", 2024-03-21T14:08:28.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1019999999999999e-06 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.424268+0000", 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024959299999999999 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.424297+0000", 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.9215999999999999e-05 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.441609+0000", 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01731253 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:24.441654+0000", 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4882999999999997e-05 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:196357 7.b 7:d501eafe:::benchmark_data_smithi003_80362_object75340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:38.063153+0000", 2024-03-21T14:08:28.385 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.805222628000003, 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.081708049000000005, 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196357 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063153+0000", 2024-03-21T14:08:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063153+0000", 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063155+0000", 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8330000000000001e-06 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.387 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063161+0000", 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4930000000000003e-06 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063173+0000", 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1826e-05 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063181+0000", 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0180000000000001e-06 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.388 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063244+0000", 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3406999999999998e-05 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063356+0000", 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111993 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.063472+0000", 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115077 2024-03-21T14:08:28.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.068432+0000", 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049602420000000001 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.068449+0000", 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7377999999999999e-05 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.144730+0000", 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076280411000000006 2024-03-21T14:08:28.390 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.144814+0000", 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4552e-05 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.144861+0000", 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6819e-05 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.391 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:196359 7.9 7:98caa353:::benchmark_data_smithi003_80362_object75342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:38.065566+0000", 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.802809388999997, 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095281696999999999, 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196359 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065566+0000", 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065566+0000", 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065568+0000", 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5239999999999999e-06 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065574+0000", 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4799999999999998e-06 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065591+0000", 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7309e-05 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065601+0000", 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8649999999999996e-06 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065753+0000", 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015189399999999999 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065989+0000", 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023558100000000001 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.066133+0000", 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014420399999999999 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.068696+0000", 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025626659999999999 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.070214+0000", 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0015185559999999999 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.070582+0000", 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036826200000000001 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.160667+0000", 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.090084485000000006 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.160731+0000", 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4379000000000001e-05 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.160848+0000", 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116492 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:196360 7.b 7:d375e271:::benchmark_data_smithi003_80362_object75343:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:38.065627+0000", 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.802748389000001, 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.079370606999999996, 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196360 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065627+0000", 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065627+0000", 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065628+0000", 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6300000000000005e-07 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065631+0000", 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3160000000000001e-06 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065646+0000", 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5139e-05 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065654+0000", 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0760000000000003e-06 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065721+0000", 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6593000000000004e-05 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065846+0000", 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012557000000000001 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,6", 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.065976+0000", 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129961 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.068454+0000", 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024777610000000002 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.068493+0000", 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8729000000000001e-05 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.144904+0000", 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076411661000000006 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.144959+0000", 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4710999999999998e-05 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.144998+0000", 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8526999999999999e-05 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:196622 7.9 7:97795475:::benchmark_data_smithi003_80362_object75605:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T14:08:28.403 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:38.419801+0000", 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.448574702999998, 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076388506999999994, 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 196622 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419801+0000", 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419801+0000", 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419802+0000", 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.415e-06 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419807+0000", 2024-03-21T14:08:28.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8239999999999999e-06 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419816+0000", 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4460000000000007e-06 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419822+0000", 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5210000000000002e-06 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.419879+0000", 2024-03-21T14:08:28.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7122e-05 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.420010+0000", 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000130913 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.420136+0000", 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012567399999999999 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.489031+0000", 2024-03-21T14:08:28.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.068894940000000002 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.489086+0000", 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5250000000000001e-05 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.489260+0000", 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017442900000000001 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.494980+0000", 2024-03-21T14:08:28.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057193779999999998 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.495474+0000", 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049439600000000003 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.496189+0000", 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00071519900000000002 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.409 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(7.1 137/130 [PushOp(7:892b7010:::benchmark_data_smithi003_80362_object114322:head, version: 112'7136, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:892b7010:::benchmark_data_smithi003_80362_object114322:head@112'7136, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:38.421807+0000", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 49.446568282999998, 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091073915000000005, 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.421807+0000", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.421807+0000", 2024-03-21T14:08:28.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.421808+0000", 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2300000000000001e-07 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.421893+0000", 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5449000000000001e-05 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.482730+0000", 2024-03-21T14:08:28.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.060836620000000001 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.482783+0000", 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3368000000000001e-05 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.510327+0000", 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.027544173000000002 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.510370+0000", 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2531999999999999e-05 2024-03-21T14:08:28.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:38.512881+0000", 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025111500000000002 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:219490 7.5 e145/130 7:a460da89:::benchmark_data_smithi003_80362_object98473:head v 145'13488, mlcod=145'13485)", 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:05.918379+0000", 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 21.949996473999999, 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076775789999999997, 2024-03-21T14:08:28.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918379+0000", 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918379+0000", 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918380+0000", 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7100000000000001e-07 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918389+0000", 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2380000000000003e-06 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918399+0000", 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5899999999999997e-06 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918406+0000", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1620000000000002e-06 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918608+0000", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020241500000000001 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.918629+0000", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0829999999999999e-05 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.993387+0000", 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.074757658000000005 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:05.995155+0000", 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017681260000000001 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221749 7.9 7:96145550:::benchmark_data_smithi003_80362_object100732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.246771+0000", 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 18.621604185999999, 2024-03-21T14:08:28.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87310833899999996, 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221749 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246771+0000", 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246771+0000", 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246772+0000", 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0699999999999996e-07 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246777+0000", 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0880000000000002e-06 2024-03-21T14:08:28.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246786+0000", 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0659999999999993e-06 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246794+0000", 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8939999999999993e-06 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246855+0000", 2024-03-21T14:08:28.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0977000000000002e-05 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246863+0000", 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1019999999999997e-06 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.116362+0000", 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86949848699999999 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.116503+0000", 2024-03-21T14:08:28.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014111199999999999 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.116650+0000", 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014681899999999999 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119736+0000", 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030864619999999999 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119754+0000", 2024-03-21T14:08:28.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8315999999999999e-05 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119786+0000", 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1615000000000001e-05 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119840+0000", 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4014000000000002e-05 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119880+0000", 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9579999999999997e-05 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:28.424 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:28.424 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:08:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:28.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:08:28.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1115407373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[29576]: pgmap v235: 148 pgs: 148 active+clean; 80 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.68k op/s; 31 B/s, 1 objects/s recovering 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1115407373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:28.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:28 smithi003 ceph-mon[40510]: pgmap v235: 148 pgs: 148 active+clean; 80 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.68k op/s; 31 B/s, 1 objects/s recovering 2024-03-21T14:08:28.526 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:28.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:28.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:28.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:28.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1115407373' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:28.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:28 smithi118 ceph-mon[33862]: pgmap v235: 148 pgs: 148 active+clean; 80 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.68k op/s; 31 B/s, 1 objects/s recovering 2024-03-21T14:08:28.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:28.803 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:28.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:28.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:28.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:28.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:29.033 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:29.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:29.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:29.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:29.415 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:29.442 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:29.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:29.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:29.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:29.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:29.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:29.826 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:29.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:29.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:29.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:29.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:30.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:30.085 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:30.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:30.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:30.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:30.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:30.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:30.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:08:30.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:30.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:30.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:30 smithi003 ceph-mon[29576]: pgmap v236: 148 pgs: 148 active+clean; 80 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:08:30.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:30 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:08:30.652 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:30 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:08:30.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:30 smithi003 ceph-mon[40510]: pgmap v236: 148 pgs: 148 active+clean; 80 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:08:30.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:30 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:08:30.652 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:30 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:08:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:30 smithi118 ceph-mon[33862]: pgmap v236: 148 pgs: 148 active+clean; 80 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.14k op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:08:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:30 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:08:30.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:30 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:08:30.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:30.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:30.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:30.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:30.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:30.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:30.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:30.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:31.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:31.100 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:31.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:31.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:31.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:31.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:31.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:31.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:31.477 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:31.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:31.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:31.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:31.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:31.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:31.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:31.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:31.857 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:31.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:31.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:08:31.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:08:31.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: 4.13 deep-scrub starts 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: 4.13 deep-scrub ok 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: 4.13 deep-scrub starts 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: 4.13 deep-scrub ok 2024-03-21T14:08:31.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:08:31.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:31 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:08:31.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:31.989 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:31.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:32.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:32.116 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:32.157 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: 4.13 deep-scrub starts 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: 4.13 deep-scrub ok 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:08:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:31 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:08:32.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:32.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:32.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:32.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:32.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:32.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:32.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:32.560 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-21T14:08:32.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-21T14:08:32.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:32.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:32.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:32.755 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:32.857 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:32.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:08:32.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:08:32.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: pgmap v237: 148 pgs: 148 active+clean; 39 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s 2024-03-21T14:08:32.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: 6.2 scrub starts 2024-03-21T14:08:32.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: 6.2 scrub ok 2024-03-21T14:08:32.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: 5.2 deep-scrub starts 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: 5.2 deep-scrub ok 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: pgmap v237: 148 pgs: 148 active+clean; 39 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: 6.2 scrub starts 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: 6.2 scrub ok 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: 5.2 deep-scrub starts 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: 5.2 deep-scrub ok 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:32.873 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:32 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:32.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:32.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:32.986 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:33.013 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:33.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:33.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:33.143 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:33.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: pgmap v237: 148 pgs: 148 active+clean; 39 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 806 op/s 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: 6.2 scrub starts 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: 6.2 scrub ok 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: 5.2 deep-scrub starts 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: 5.2 deep-scrub ok 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:32 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:33.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:33.272 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:33.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:33.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:33.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:33.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:33.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:33.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:33.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:33.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:33.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:33.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:33.880 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:33.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:33.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:34.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:34.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 7.b scrub starts 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 7.b scrub ok 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 7.6 scrub starts 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 7.6 scrub ok 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:08:34.068 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 7.b scrub starts 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 7.b scrub ok 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 7.6 scrub starts 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 7.6 scrub ok 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:08:34.069 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:08:34.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:08:34.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:34.070 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:33 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:34.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:34.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 7.b scrub starts 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 7.b scrub ok 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 7.6 scrub starts 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 7.6 scrub ok 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: osdmap e154: 8 total, 8 up, 8 in 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:08:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:08:34.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:34.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:33 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:34.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:34.287 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:34.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:34.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:34.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:34.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:34.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:34.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:34.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-21T14:08:34.639 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:34.667 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:34.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:34.769 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:34.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:34.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:08:34.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:34.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:34.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:34.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: pgmap v239: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s 2024-03-21T14:08:34.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:08:34.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 7.e scrub starts 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 7.e scrub ok 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3644789622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: pgmap v239: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:08:34.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 7.e scrub starts 2024-03-21T14:08:34.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 7.e scrub ok 2024-03-21T14:08:34.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T14:08:34.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:08:34.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:08:34.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3644789622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:08:35.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:35.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:35.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:35.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:35.176 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: pgmap v239: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 900 op/s 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 7.e scrub starts 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 7.e scrub ok 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: osdmap e155: 8 total, 8 up, 8 in 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:08:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3644789622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:08:35.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:35.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:35.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:35.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:35.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:35.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:35.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:35.689 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:35.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:35.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:35.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:35.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:35.883 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3644789622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[40510]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[40510]: 7.d scrub starts 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3644789622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[29576]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T14:08:35.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:35 smithi003 ceph-mon[29576]: 7.d scrub starts 2024-03-21T14:08:35.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:35.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:35.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:36.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:36.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:36.131 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:36.150 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:36.156 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:08:36.156 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:08:36.156 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-03-21T14:08:36.156 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:08:36.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:35 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:35 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:35 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3644789622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:35 smithi118 ceph-mon[33862]: osdmap e156: 8 total, 8 up, 8 in 2024-03-21T14:08:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:35 smithi118 ceph-mon[33862]: 7.d scrub starts 2024-03-21T14:08:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.196 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:36.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:36.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:36.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:36.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:36.454 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:36.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:36.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:36.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:36.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:36.708 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:08:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:08:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:08:36.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:36 smithi118 ceph-mon[33862]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 213 op/s 2024-03-21T14:08:36.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:08:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:08:36.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:36.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:36.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:37.063 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:37.090 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:37.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:37.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:37.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:36 smithi003 ceph-mon[29576]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 213 op/s 2024-03-21T14:08:37.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:36 smithi003 ceph-mon[40510]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 213 op/s 2024-03-21T14:08:37.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:37.346 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:37.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:37.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:37.476 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:37.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:37.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:37.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:37.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:37.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:37.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:37.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:37.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:37.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:08:37.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:37.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:37.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:38.085 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:38.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:38.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:38.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:38.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:38.336 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:38.365 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:38.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:38.494 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:38.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T14:08:38.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:38.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:38.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:38.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:38.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:38.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:38.906 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:38.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:39.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:39.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:39.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:39.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:38 smithi118 ceph-mon[33862]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 102 op/s 2024-03-21T14:08:39.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:39.166 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:38 smithi003 ceph-mon[29576]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 102 op/s 2024-03-21T14:08:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:38 smithi003 ceph-mon[40510]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 102 op/s 2024-03-21T14:08:39.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:39.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:39.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:39.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:39.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:39.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:39.539 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:08:39.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:39.560 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:08:39.580 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:08:39.602 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:08:39.623 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:08:39.644 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:08:39.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:39.665 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:08:39.677 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:39.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:39.686 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:08:39.688 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:08:39.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:39.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:39.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:39.935 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:39.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:39.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:40.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:40.064 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:40.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:40.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:39 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:39 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2884568067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:40.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:40.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:39 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:39 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2884568067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:39 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:39 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2884568067' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:40.291 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:40.318 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:40.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:40.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:40.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:40.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:40.545 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:40.572 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:40.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:40.697 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:40.701 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:40.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:40.829 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:40.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:40.958 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:40.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:40.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[29576]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T14:08:40.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[29576]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T14:08:40.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:40.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:40.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[40510]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T14:08:40.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[40510]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T14:08:40.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:40.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:40 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:40.980 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:41.059 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:41.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:40 smithi118 ceph-mon[33862]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 780 B/s rd, 0 op/s 2024-03-21T14:08:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:40 smithi118 ceph-mon[33862]: osdmap e157: 8 total, 8 up, 8 in 2024-03-21T14:08:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:40 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:40 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:41.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:41.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.211 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:41.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:41.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:41.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:41.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:41.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:41.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:41.589 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:41.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:41.691 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:41.719 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:41.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:41.820 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:41.850 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:41.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:41.854 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_0' removed 2024-03-21T14:08:41.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:41.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:41.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:42.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:42.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:42.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:41 smithi118 ceph-mon[33862]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T14:08:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2939173792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:08:42.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.247 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:42.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:41 smithi003 ceph-mon[29576]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T14:08:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2939173792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:08:42.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:41 smithi003 ceph-mon[40510]: osdmap e158: 8 total, 8 up, 8 in 2024-03-21T14:08:42.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2939173792' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:08:42.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:42.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:42.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:42.497 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:42.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:42.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:08:42.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:42.623 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:42.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:42.724 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:42.748 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:42.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:42.873 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:42.975 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:42.999 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:42.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:43.100 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:43.123 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:42 smithi118 ceph-mon[33862]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T14:08:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:42 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2939173792' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:08:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:42 smithi118 ceph-mon[33862]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T14:08:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:42 smithi003 ceph-mon[29576]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T14:08:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:42 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2939173792' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:08:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:42 smithi003 ceph-mon[29576]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T14:08:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:42 smithi003 ceph-mon[40510]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-21T14:08:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:42 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2939173792' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:08:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:42 smithi003 ceph-mon[40510]: osdmap e159: 8 total, 8 up, 8 in 2024-03-21T14:08:43.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:43.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:43.253 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T14:08:43.254 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T14:08:43.254 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-21T14:08:43.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-21T14:08:43.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:43.377 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:43.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:43.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:43.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:43.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:43.727 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:43.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:43.729 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:43.752 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:43.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:43.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:43.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:43.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:43.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:43.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:44.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:44.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:44.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:44.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:44.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:44.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:44.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:44.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:44.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:44.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:44.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:44.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:44.635 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:44.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:44.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:44.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:44.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:44.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:44.891 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:44.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:44.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:44 smithi003 ceph-mon[29576]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:08:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:44 smithi003 ceph-mon[40510]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:08:45.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:45.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:45.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:45.149 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:44 smithi118 ceph-mon[33862]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:08:45.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:45.278 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:45.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:45.408 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:45.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:45.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:45.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:45.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:45.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:45.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:45.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:45.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:45.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:45.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:45.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:46.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:46.044 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:46.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:46.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:46.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:46.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:46.300 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:46.383 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:08:46.383 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:08:46.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:46.428 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:46.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.558 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:46.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:46.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:46.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:46.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:08:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:08:46.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:46.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:46.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:46.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:46.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:08:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:08:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:46 smithi118 ceph-mon[33862]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:08:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:46.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:46.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:46.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:47.038 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:47.066 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:47.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:47.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:47.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:47.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-mon[29576]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:08:47.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:47.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:47.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-mon[40510]: pgmap v250: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:08:47.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:08:47.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:08:47.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:47.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:47.418 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:47.446 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:47.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:47.574 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:47.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:47.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T14:08:47.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:47.702 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:47.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:47.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.829 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:47.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:47.930 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:47.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:47.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:47.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:47.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:48.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:48.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:48.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:48.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:48.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:48.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:48.337 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:48.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:48.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:48.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:48.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:48.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:48.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.721 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:48.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:48.773 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:48.816 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:08:48.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:48.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:48.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:48.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:48.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:48.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:48 smithi003 ceph-mon[29576]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3764691139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:48 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:48 smithi003 ceph-mon[40510]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3764691139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:48 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:49.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.108 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:49.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:48 smithi118 ceph-mon[33862]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3764691139' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:48 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:08:49.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:49.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:49.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:49.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:49.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:49.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:49.490 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:49.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:49.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:49.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:49.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:49.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:49.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:49.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:49.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:49.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:50.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:50.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:49 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:50.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:49 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:08:50.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:49 smithi003 ceph-mon[29576]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T14:08:50.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:49 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:50.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:49 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:08:50.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:49 smithi003 ceph-mon[40510]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T14:08:50.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:50.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:49 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:49 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:08:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:49 smithi118 ceph-mon[33862]: osdmap e160: 8 total, 8 up, 8 in 2024-03-21T14:08:50.209 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:08:50.229 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:08:50.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:50.248 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:08:50.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:50.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:50.269 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:08:50.289 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:08:50.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:08:50.331 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:08:50.351 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:08:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:08:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:08:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:08:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:08:50.352 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:08:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:08:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:08:50.353 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:08:50.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:50.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:50.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:50.514 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:50.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:50.643 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:50.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:50.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:50.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:50.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:50.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:50.998 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:51.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:51.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:51.155 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:50 smithi118 ceph-mon[33862]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T14:08:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/199495295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:51.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:50 smithi003 ceph-mon[40510]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T14:08:51.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/199495295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:51.218 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:50 smithi003 ceph-mon[29576]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-21T14:08:51.218 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/199495295' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:08:51.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:51.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:51.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:51.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:51.412 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:51.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:51.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:51.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:51.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:51.667 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:51.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:51.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:51.797 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:51.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:51.892 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:51.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:51.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:51.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:52.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:52.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:52.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:52.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:51 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:08:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:51 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:08:52.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:52.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:51 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:08:52.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:51 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:08:52.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:51 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:08:52.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:51 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:08:52.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.3 reset_purged_snaps_last 2024-03-21T14:08:52.283 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:52.310 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:52.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:52.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:52.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:52.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:52.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:52.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:52.695 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:52.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:52.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:52.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:52.922 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:52.949 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:52.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:53.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:53.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:53.124 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:52 smithi003 ceph-mon[29576]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:08:53.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:52 smithi003 ceph-mon[29576]: osdmap e161: 8 total, 8 up, 8 in 2024-03-21T14:08:53.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:52 smithi003 ceph-mon[40510]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:08:53.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:52 smithi003 ceph-mon[40510]: osdmap e161: 8 total, 8 up, 8 in 2024-03-21T14:08:53.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:52 smithi118 ceph-mon[33862]: pgmap v254: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:08:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:52 smithi118 ceph-mon[33862]: osdmap e161: 8 total, 8 up, 8 in 2024-03-21T14:08:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:53.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:53.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:08:53.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:53.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:53.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:53.457 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:53.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:53.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:140727 7.e 7:7c6116bf:::benchmark_data_smithi003_80362_object19710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:30.549679+0000", 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 142.873283378, 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.140970928, 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.619 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140727 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549679+0000", 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549679+0000", 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549681+0000", 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.576e-06 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549688+0000", 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1790000000000002e-06 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549710+0000", 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2541e-05 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549722+0000", 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1779e-05 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549778+0000", 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5767999999999999e-05 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549894+0000", 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011581 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.550015+0000", 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121532 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.553472+0000", 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003456359 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.555122+0000", 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016506820000000001 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.690582+0000", 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13545950400000001 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.690618+0000", 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5834000000000003e-05 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.690650+0000", 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2363999999999997e-05 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:141678 7.e 7:7e298d8b:::benchmark_data_smithi003_80362_object20661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:31.587643+0000", 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 141.835319736, 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.132807325, 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141678 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587643+0000", 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587643+0000", 2024-03-21T14:08:53.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587644+0000", 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.587e-06 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587651+0000", 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4439999999999996e-06 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587667+0000", 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6446000000000001e-05 2024-03-21T14:08:53.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587676+0000", 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2539999999999998e-06 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.597199+0000", 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095225520000000001 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.597318+0000", 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118934 2024-03-21T14:08:53.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.597436+0000", 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118468 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.602852+0000", 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054152159999999996 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.603071+0000", 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000219435 2024-03-21T14:08:53.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.720379+0000", 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.117308329 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.720418+0000", 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9060999999999998e-05 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.720450+0000", 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1599e-05 2024-03-21T14:08:53.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151001 7.e 7:71a6f342:::benchmark_data_smithi003_80362_object29984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.792444+0000", 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.630518086, 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.150560518, 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151001 2024-03-21T14:08:53.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792444+0000", 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792444+0000", 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792446+0000", 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6220000000000001e-06 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792452+0000", 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.212e-06 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792464+0000", 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2109e-05 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792473+0000", 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.619e-06 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792566+0000", 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3263999999999993e-05 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792699+0000", 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013310000000000001 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792812+0000", 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112281 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.800854+0000", 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080425960000000008 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.801137+0000", 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000282422 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.942687+0000", 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14155068200000001 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.942852+0000", 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016438800000000001 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943005+0000", 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015322299999999999 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151003 7.e 7:783de3cd:::benchmark_data_smithi003_80362_object29986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.792496+0000", 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.630466349, 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15094943299999999, 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151003 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792496+0000", 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792496+0000", 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792497+0000", 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9699999999999996e-07 2024-03-21T14:08:53.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792500+0000", 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1159999999999999e-06 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792507+0000", 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1709999999999996e-06 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792513+0000", 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6810000000000003e-06 2024-03-21T14:08:53.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.796583+0000", 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040704900000000004 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.796702+0000", 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011917999999999999 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.796826+0000", 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123504 2024-03-21T14:08:53.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.801070+0000", 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042447459999999998 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.801160+0000", 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9766999999999996e-05 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943163+0000", 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.142002613 2024-03-21T14:08:53.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943313+0000", 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000150588 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943446+0000", 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013208 2024-03-21T14:08:53.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151012 7.e 7:79355c2f:::benchmark_data_smithi003_80362_object29995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.807247+0000", 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.615715177, 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.136874252, 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151012 2024-03-21T14:08:53.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807247+0000", 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807247+0000", 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807249+0000", 2024-03-21T14:08:53.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1229999999999998e-06 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807257+0000", 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5020000000000003e-06 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807277+0000", 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9769999999999999e-05 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807286+0000", 2024-03-21T14:08:53.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5189999999999998e-06 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807489+0000", 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000203301 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.808025+0000", 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053579400000000003 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.808323+0000", 2024-03-21T14:08:53.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029820500000000001 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825027+0000", 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016703273000000001 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.834701+0000", 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096738910000000004 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943838+0000", 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109137516 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.944000+0000", 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016142000000000001 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.944122+0000", 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121938 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151016 7.e 7:7ed0d353:::benchmark_data_smithi003_80362_object29999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.813183+0000", 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.609779663, 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13364878899999999, 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151016 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813183+0000", 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813183+0000", 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813185+0000", 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.221e-06 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813191+0000", 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3080000000000004e-06 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813208+0000", 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6954999999999999e-05 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813217+0000", 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1470000000000007e-06 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.820048+0000", 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068310599999999999 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.820197+0000", 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014832800000000001 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.820330+0000", 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133487 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825236+0000", 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004905385 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.834728+0000", 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094926720000000006 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.946306+0000", 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11157792 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.946617+0000", 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031110000000000003 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.946832+0000", 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021420599999999999 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.654 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151020 7.e 7:7885fc55:::benchmark_data_smithi003_80362_object30003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.824394+0000", 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.598568363, 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.123109119, 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.655 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151020 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824394+0000", 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824394+0000", 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824396+0000", 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.728e-06 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824401+0000", 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5899999999999998e-06 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824418+0000", 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6558999999999998e-05 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824427+0000", 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8149999999999995e-06 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825402+0000", 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097474200000000001 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825755+0000", 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000353767 2024-03-21T14:08:53.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.826716+0000", 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00096025099999999996 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.834886+0000", 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081701729999999993 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.838069+0000", 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031830299999999999 2024-03-21T14:08:53.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.947044+0000", 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10897557400000001 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.947337+0000", 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029230899999999998 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.947503+0000", 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016658100000000001 2024-03-21T14:08:53.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151160 7.e 7:7eca1350:::benchmark_data_smithi003_80362_object30143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:46.141625+0000", 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.28133696099999, 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.152556267, 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151160 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141625+0000", 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141625+0000", 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141627+0000", 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6169999999999999e-06 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141633+0000", 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6799999999999998e-06 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141655+0000", 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2582000000000002e-05 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141664+0000", 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4780000000000008e-06 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.142204+0000", 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054035599999999997 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.142502+0000", 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029758299999999997 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.142874+0000", 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037233600000000002 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.147284+0000", 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044097900000000002 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.147380+0000", 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6098e-05 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294083+0000", 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14670290699999999 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294137+0000", 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4388999999999997e-05 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294182+0000", 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4450999999999998e-05 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151179 7.e 7:7b27c9b6:::benchmark_data_smithi003_80362_object30162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:46.173411+0000", 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 127.249551541, 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120895954, 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151179 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173411+0000", 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173411+0000", 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173413+0000", 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0229999999999999e-06 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173420+0000", 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173436+0000", 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6217000000000001e-05 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173448+0000", 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1602e-05 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173637+0000", 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018898400000000001 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.174000+0000", 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036340999999999999 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.174394+0000", 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000393419 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.178269+0000", 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038754890000000002 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.178308+0000", 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8575999999999998e-05 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294221+0000", 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115913008 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294271+0000", 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0358000000000001e-05 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294307+0000", 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5528000000000003e-05 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152526 7.e 7:72655171:::benchmark_data_smithi003_80362_object31509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.324698+0000", 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.098264329, 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.134498811, 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152526 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.674 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324698+0000", 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324698+0000", 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324699+0000", 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.302e-06 2024-03-21T14:08:53.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324705+0000", 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3739999999999997e-06 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324717+0000", 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.253e-05 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324725+0000", 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1750000000000005e-06 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.325984+0000", 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012589019999999999 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.326314+0000", 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000329535 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.326428+0000", 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113921 2024-03-21T14:08:53.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335437+0000", 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0090096570000000008 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335452+0000", 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4775e-05 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459046+0000", 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12359379199999999 2024-03-21T14:08:53.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459144+0000", 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7990000000000002e-05 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459197+0000", 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2858000000000001e-05 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.679 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152530 7.e 7:79192372:::benchmark_data_smithi003_80362_object31513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.329026+0000", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.09393604100001, 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13036171099999999, 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152530 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329026+0000", 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329026+0000", 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329028+0000", 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5629999999999999e-06 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329034+0000", 2024-03-21T14:08:53.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.835e-06 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329051+0000", 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.685e-05 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329064+0000", 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3219e-05 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.330419+0000", 2024-03-21T14:08:53.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013551469999999999 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.330719+0000", 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030028800000000001 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.331228+0000", 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050899500000000004 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335489+0000", 2024-03-21T14:08:53.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042610110000000003 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335500+0000", 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0553e-05 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459255+0000", 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.123755485 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459345+0000", 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9462000000000005e-05 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459388+0000", 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3303000000000002e-05 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152536 7.e 7:74894159:::benchmark_data_smithi003_80362_object31519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.334871+0000", 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.088091569, 2024-03-21T14:08:53.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.124694136, 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152536 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334871+0000", 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334871+0000", 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334873+0000", 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.784e-06 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334877+0000", 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1439999999999996e-06 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334888+0000", 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1365e-05 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334893+0000", 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2800000000000003e-06 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335627+0000", 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00073397600000000001 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335925+0000", 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029732000000000001 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.336298+0000", 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037340700000000002 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.343414+0000", 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071161480000000001 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.343433+0000", 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9108999999999999e-05 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459435+0000", 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.116001458 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459523+0000", 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8332999999999996e-05 2024-03-21T14:08:53.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459565+0000", 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1811999999999999e-05 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152538 7.e 7:77c3ed5e:::benchmark_data_smithi003_80362_object31521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.338885+0000", 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 125.084077193, 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120848841, 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.691 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152538 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338885+0000", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338885+0000", 2024-03-21T14:08:53.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338886+0000", 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7100000000000001e-07 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338890+0000", 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.794e-06 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338902+0000", 2024-03-21T14:08:53.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2140000000000001e-05 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338908+0000", 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1929999999999998e-06 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.339836+0000", 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092747499999999998 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.340014+0000", 2024-03-21T14:08:53.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017830100000000001 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.340117+0000", 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102717 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.344340+0000", 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042231459999999997 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.344355+0000", 2024-03-21T14:08:53.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5058000000000001e-05 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459610+0000", 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115255682 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459694+0000", 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3944999999999997e-05 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459734+0000", 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9718999999999997e-05 2024-03-21T14:08:53.696 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153387 7.e 7:7b4816c7:::benchmark_data_smithi003_80362_object32370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.627694+0000", 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.795267963, 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13285661700000001, 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153387 2024-03-21T14:08:53.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627694+0000", 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627694+0000", 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627696+0000", 2024-03-21T14:08:53.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.683e-06 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627704+0000", 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1329999999999999e-06 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627719+0000", 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4761e-05 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627728+0000", 2024-03-21T14:08:53.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1719999999999996e-06 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627810+0000", 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2275999999999996e-05 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627946+0000", 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135752 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.628070+0000", 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123917 2024-03-21T14:08:53.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.632122+0000", 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040513629999999997 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.633010+0000", 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088858399999999999 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.760389+0000", 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12737904899999999 2024-03-21T14:08:53.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.760499+0000", 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010972999999999999 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.760551+0000", 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2197000000000002e-05 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.702 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153393 7.e 7:7f50553e:::benchmark_data_smithi003_80362_object32376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.640759+0000", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.78220382000001, 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.126200215, 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153393 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640759+0000", 2024-03-21T14:08:53.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640759+0000", 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640760+0000", 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5099999999999999e-06 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640766+0000", 2024-03-21T14:08:53.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.631e-06 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640776+0000", 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.029e-05 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640783+0000", 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6590000000000001e-06 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.641111+0000", 2024-03-21T14:08:53.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032799499999999998 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.641231+0000", 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012035 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.641346+0000", 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011500100000000001 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.646310+0000", 2024-03-21T14:08:53.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049644310000000001 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.648887+0000", 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025767889999999999 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.766776+0000", 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.117888702 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.766908+0000", 2024-03-21T14:08:53.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131662 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.766959+0000", 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1195e-05 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153700 7.e 7:767bfc88:::benchmark_data_smithi003_80362_object32683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.113168+0000", 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 123.309794868, 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15167361400000001, 2024-03-21T14:08:53.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153700 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113168+0000", 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113168+0000", 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113169+0000", 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.562e-06 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113174+0000", 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8590000000000004e-06 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113192+0000", 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7941999999999999e-05 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113200+0000", 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6260000000000003e-06 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.114412+0000", 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001212791 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.114526+0000", 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113539 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.114645+0000", 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119078 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.118486+0000", 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038406989999999999 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.118575+0000", 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.954e-05 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.264731+0000", 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.146156233 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.264794+0000", 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2465999999999995e-05 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.264841+0000", 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7278999999999999e-05 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154042 7.e 7:7e3a347b:::benchmark_data_smithi003_80362_object33025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.643263+0000", 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 122.779699741, 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16783620799999999, 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.714 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154042 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643263+0000", 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643263+0000", 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643264+0000", 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.463e-06 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643269+0000", 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2900000000000002e-06 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643289+0000", 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9221000000000001e-05 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643297+0000", 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.909e-06 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643350+0000", 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3458999999999998e-05 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643459+0000", 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108498 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.717 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643569+0000", 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110876 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.648285+0000", 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047160789999999998 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.648366+0000", 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0090000000000001e-05 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.810900+0000", 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.162534278 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811043+0000", 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014269 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811099+0000", 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6354999999999998e-05 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154058 7.e 7:733d882b:::benchmark_data_smithi003_80362_object33041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 122.75793277699999, 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.146220038, 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154058 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.23e-07 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665034+0000", 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.109e-06 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665043+0000", 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4710000000000007e-06 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665049+0000", 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5480000000000001e-06 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665097+0000", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8627e-05 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665183+0000", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6013000000000005e-05 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665254+0000", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0959999999999998e-05 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.667703+0000", 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024487559999999998 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.668694+0000", 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00099065200000000007 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811155+0000", 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14246165799999999 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811207+0000", 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2009999999999998e-05 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811250+0000", 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2511e-05 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154760 7.e 7:7bbbc488:::benchmark_data_smithi003_80362_object33743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:51.850395+0000", 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 121.572567451, 2024-03-21T14:08:53.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11783648200000001, 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154760 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850395+0000", 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850395+0000", 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850397+0000", 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7129999999999999e-06 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850403+0000", 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4300000000000003e-06 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850419+0000", 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5654999999999998e-05 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850428+0000", 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1800000000000002e-06 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850493+0000", 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4937999999999994e-05 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850622+0000", 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128801 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850703+0000", 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1545999999999995e-05 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.854287+0000", 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035834679999999998 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.858521+0000", 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042342430000000004 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.967804+0000", 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109283279 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.968057+0000", 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025259799999999998 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.968231+0000", 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017463099999999999 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:176341 7.e 7:72b7b1c7:::benchmark_data_smithi003_80362_object55324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:13.608919+0000", 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.814043526000006, 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.118495692, 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:08:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176341 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608919+0000", 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608919+0000", 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:08:53.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608921+0000", 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608933+0000", 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.225e-05 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608947+0000", 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3934000000000001e-05 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608956+0000", 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8210000000000008e-06 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.626731+0000", 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017775076000000001 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.626844+0000", 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112972 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.626962+0000", 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0001187 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.677437+0000", 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050474473999999998 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.681328+0000", 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038916240000000002 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.727293+0000", 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045964694 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.727368+0000", 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4567999999999995e-05 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.727415+0000", 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6968999999999997e-05 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:08:53.737 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:08:53.737 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:08:53.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:53.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:53.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:08:53.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:53.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:53.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:54.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:54.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:54.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:54.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:54.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:54.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:54.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T14:08:54.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:54.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:54.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:54.557 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:54.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:54.687 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:54.791 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:54.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:54.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:54.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:54.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:54.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:54 smithi003 ceph-mon[29576]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:55.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:54 smithi003 ceph-mon[40510]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:55.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:55.081 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:55.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:54 smithi118 ceph-mon[33862]: pgmap v256: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:55.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:55.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:55.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:55.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:55.469 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:55.499 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:55.500 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:55.505 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:08:55.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:55.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:55.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:55.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:55.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:55.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:55.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:55.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:55.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:55.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:55.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:55.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:56.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:56.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:56.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:56.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:56.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:56.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:56.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:56.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:56.476 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:56.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:56.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:56.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:56.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:56.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:56.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:08:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:08:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:08:56.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:56.859 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:56.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:56 smithi118 ceph-mon[33862]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:08:56.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:08:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:08:56.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:56.988 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:56.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:57.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:57.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:56 smithi003 ceph-mon[29576]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:08:57.106 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:56 smithi003 ceph-mon[40510]: pgmap v257: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:08:57.117 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:57.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:57.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:57.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:57.371 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:57.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:57.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:57.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:57.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:57.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:57.626 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:57.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:08:57.754 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:08:57.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:57.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:57.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:57.981 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:58.009 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:58.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:08:58.112 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:58.112 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:58.137 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:08:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:08:58.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:58.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:58.366 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:58.394 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:58.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:58.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:58.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:58.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:58.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:58.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:58.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:58.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:58.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:58.902 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:58.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:59.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:08:59.029 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:59.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:08:59.093 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:08:58 smithi003 ceph-mon[29576]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:59.093 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:08:58 smithi003 ceph-mon[40510]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:59.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:59.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:08:58 smithi118 ceph-mon[33862]: pgmap v258: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:08:59.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:59.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:08:59.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:08:59.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:08:59.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:08:59.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:08:59.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:08:59.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:59.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:08:59.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:08:59.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:08:59.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:08:59.887 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:08:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:08:59.916 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:08:59.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:00.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:00.041 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:00.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:00.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:00.268 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:00.295 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:00.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:00.423 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:00.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:00.448 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:09:00.448 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_1' created 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[29576]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3946765960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2742582321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[40510]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3946765960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T14:09:00.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T14:09:00.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2742582321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:00.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:00 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:00.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:00.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:00.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:00.676 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:00.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:00.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:00.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:00.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:00.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:00.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:00.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:00 smithi118 ceph-mon[33862]: pgmap v259: 132 pgs: 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3946765960' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T14:09:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:00 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-21T14:09:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2742582321' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:00 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:01.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:01.059 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:01.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:01.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-21T14:09:01.161 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:01.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:01.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:01.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:01.318 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:01.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:01.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:01.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:01.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:01.577 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:01.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:01.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:01.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-21T14:09:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:09:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[29576]: osdmap e162: 8 total, 8 up, 8 in 2024-03-21T14:09:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-21T14:09:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:09:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[40510]: osdmap e162: 8 total, 8 up, 8 in 2024-03-21T14:09:01.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:01.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:01.831 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:01.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:01.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-21T14:09:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:01 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:01 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-21T14:09:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:01 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:09:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:01 smithi118 ceph-mon[33862]: osdmap e162: 8 total, 8 up, 8 in 2024-03-21T14:09:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:01.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:01.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:01.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:01.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:02.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:02.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:02.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:02.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:02.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:02.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:02.338 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:02.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:02.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:02.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:02 smithi003 ceph-mon[29576]: pgmap v261: 148 pgs: 7 creating+peering, 9 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:09:02.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:02 smithi003 ceph-mon[29576]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:09:02.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:02 smithi003 ceph-mon[29576]: osdmap e163: 8 total, 8 up, 8 in 2024-03-21T14:09:02.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:02 smithi003 ceph-mon[40510]: pgmap v261: 148 pgs: 7 creating+peering, 9 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:09:02.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:02 smithi003 ceph-mon[40510]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:09:02.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:02 smithi003 ceph-mon[40510]: osdmap e163: 8 total, 8 up, 8 in 2024-03-21T14:09:02.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:02.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:02.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:02.722 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:02.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:02.749 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:02.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:02.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:02 smithi118 ceph-mon[33862]: pgmap v261: 148 pgs: 7 creating+peering, 9 unknown, 132 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-21T14:09:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:02 smithi118 ceph-mon[33862]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:09:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:02 smithi118 ceph-mon[33862]: osdmap e163: 8 total, 8 up, 8 in 2024-03-21T14:09:02.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:02.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:02.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:03.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:03.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:03.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:03.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:03.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:03.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:03.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:03.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:03.481 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:09:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:09:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[29576]: osdmap e164: 8 total, 8 up, 8 in 2024-03-21T14:09:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:09:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:09:03.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:09:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:09:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[40510]: osdmap e164: 8 total, 8 up, 8 in 2024-03-21T14:09:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:09:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:03 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:09:03.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:03.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:03.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:03.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:03.734 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:03.836 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:03.861 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:03.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:03 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:09:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:03 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:09:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:03 smithi118 ceph-mon[33862]: osdmap e164: 8 total, 8 up, 8 in 2024-03-21T14:09:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:03 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:09:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:03 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:09:03.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:03.987 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:03.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:03.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:04.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:04.113 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:04.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:04.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.240 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:04.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:04.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:04.365 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:04.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:04.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.491 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:04.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:04.563 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:09:04.582 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:09:04.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:09:04.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:04.603 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:09:04.625 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:09:04.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:04.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:04.645 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:09:04.668 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:09:04.689 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:09:04.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:09:04.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:09:04.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:09:04.689 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: pgmap v264: 148 pgs: 2 creating+activating, 7 creating+peering, 139 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:09:04.690 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: pgmap v264: 148 pgs: 2 creating+activating, 7 creating+peering, 139 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:09:04.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:04 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:09:04.709 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:09:04.711 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:09:04.729 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:04.752 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:04.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:04.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:04.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: pgmap v264: 148 pgs: 2 creating+activating, 7 creating+peering, 139 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:09:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:09:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:09:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:04 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:09:04.980 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:05.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.005 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:05.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:05.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:05.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:05.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:05.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:05.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:05.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:05.380 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:05.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:05.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:05.506 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:05.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:05.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:05.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:05.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:09:05.721 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1181844777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:09:05.722 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1181844777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:09:05.723 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:05 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:09:05.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:05.748 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:05.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:05.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:05.758 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:05.760 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:09:05.760 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:09:05.767 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:05.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:05.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:05.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1181844777' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:09:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:09:05.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:09:05.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:09:05.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:05 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:09:05.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:06.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:06.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:06.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:06.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:06.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:06.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:06.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:06.385 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:06.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:06.509 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:06.611 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:06.634 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:06.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:09:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:09:06.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-mon[40510]: pgmap v265: 148 pgs: 2 creating+activating, 7 creating+peering, 139 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:09:06.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:09:06.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:09:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-mon[29576]: pgmap v265: 148 pgs: 2 creating+activating, 7 creating+peering, 139 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:09:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:09:06.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:06 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:09:06.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:06.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:06.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:06.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:06.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:06.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:06 smithi118 ceph-mon[33862]: pgmap v265: 148 pgs: 2 creating+activating, 7 creating+peering, 139 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:09:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:06 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:09:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:06 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:09:06.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:09:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:09:06.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:07.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:07.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:07.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:07.141 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:07.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:07.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:07.269 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:07.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:07.395 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:07.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:09:07.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:07.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:07.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:07.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:07.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:07 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:09:07.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:07 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:09:07.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:07 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:09:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:07 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:09:07.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:07.778 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.805 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:07.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:07.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:07.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T14:09:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:07 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:09:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:07 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:09:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:07.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:07.932 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:07.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:08.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:08.057 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:08.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:08.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:08.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:08.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:08.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:08.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:08.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:08.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:08.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:08.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:08.695 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:08.708 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[29576]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[40510]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:09:08.709 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:08 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:09:08.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:08.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:08.926 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:08.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:08 smithi118 ceph-mon[33862]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:09:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:08 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:09:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:08 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:09:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:08 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:09:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:08 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:09:08.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:08.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:09.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:09.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:09.185 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:09.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:09.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:09.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:09.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:09.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:09.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:09.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:09.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:09.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:09.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:09.726 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:09.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:09.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:09 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:09:09.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:09 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:09:09.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:09 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:09:09.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:09 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:09:09.828 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:09.832 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:09 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:09:09.832 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:09 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:09:09.855 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:09.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:09.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:09.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:09.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:10.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:10.107 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:10.209 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:10.237 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:10.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:10.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:10.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:10.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:10.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:10.492 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-21T14:09:10.544 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: pgmap v267: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:09:10.545 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3151150126' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: pgmap v267: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:09:10.546 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3151150126' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:09:10.547 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:10 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:09:10.592 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:10.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:10.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:10.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:10.876 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:10.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:09:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:09:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:09:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:09:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: pgmap v267: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:09:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3151150126' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:09:10.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:10 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:09:10.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:11.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:11.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:11.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:11.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:11.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:11.355 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:11.384 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:11.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:11.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:11.514 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:11.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:11.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:11.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:11.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: osdmap e165: 8 total, 8 up, 8 in 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:09:11.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:09:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: osdmap e165: 8 total, 8 up, 8 in 2024-03-21T14:09:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:09:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:09:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:09:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:11 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:09:11.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:11.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:11.819 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:11.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:11.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: osdmap e165: 8 total, 8 up, 8 in 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:09:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:11 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:09:11.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:12.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:12.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:12.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:12.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-21T14:09:12.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:12.274 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:12.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:12.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:12.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:12.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:12.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:12.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:12.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:12.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:12.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:09:12.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:09:12.746 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:09:12.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:09:12.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:09:12.747 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:12 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:09:12.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:12.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:12.827 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-21T14:09:12.829 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T14:09:12.829 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T14:09:12.880 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-03-21T14:09:12.880 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T14:09:12.880 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_110242 2024-03-21T14:09:12.880 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:09:12.881 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-03-21T14:09:12.881 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:12.915 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:12.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:12.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:09:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:12 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:09:13.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:13.054 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:13.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:13.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:13.194 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:13.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:13.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:13.331 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:13.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:13.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:13.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:13.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:13.609 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:13.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:13.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:13.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:13.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:13.881 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3406 3390 211.841 211.875 0.0060725 0.00469415 2024-03-21T14:09:13.887 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:13.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:13.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:09:13.983 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:09:13.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:13.984 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:13.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:09:13.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:09:13.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:13.984 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:13 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:13.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:14.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:14.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:14.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:14.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.174 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:14.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:13 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:09:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:13 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:09:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:13 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:13 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:14.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:14.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:14.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:14.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:14.449 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:14.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:14.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:14.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:14.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:14.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:14.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:14.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:14.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:14.880 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 6597 6581 205.631 199.438 0.00403602 0.00485291 2024-03-21T14:09:14.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:15.011 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:15.113 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:15.146 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:15.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:09:15.162 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:15.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:09:15.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:09:15.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:15.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:09:15.163 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:14 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:09:15.167 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:15.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:15.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:15.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: pgmap v270: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:09:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:14 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:09:15.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:15.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:15.538 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.576 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:15.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:15.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:15.713 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:15.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:15.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:15.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:15.880 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 15 9445 9430 196.436 178.062 0.00387279 0.0050821 2024-03-21T14:09:15.954 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:09:15.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:16.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:16.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:16.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:16.144 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:16.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:16.245 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2890633269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2890633269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:16.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:16.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:16.380 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:16.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:16.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:15 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:09:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:15 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:09:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:15 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:09:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:15 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:09:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:15 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2890633269' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:16.520 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:16.555 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:16.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:16.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:16.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:16.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:09:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:09:16.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:16.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:16.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:16.845 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:09:16.845 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:09:16.881 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 12196 12180 190.287 171.875 0.00570074 0.00524308 2024-03-21T14:09:16.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:09:16.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:16.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:16.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:17.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:17.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2890633269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: osdmap e166: 8 total, 8 up, 8 in 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:09:17.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2890633269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: osdmap e166: 8 total, 8 up, 8 in 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:09:17.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:09:17.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:16 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:09:17.109 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:09:17.114 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:17.153 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:09:17.202 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:09:17.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:17.245 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:09:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:17.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:17.285 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:09:17.327 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:09:17.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:17.368 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:09:17.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:17.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:17.407 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:09:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:09:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:09:17.411 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:09:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:09:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:09:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:09:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:09:17.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: pgmap v271: 148 pgs: 148 active+clean; 582 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2890633269' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: osdmap e166: 8 total, 8 up, 8 in 2024-03-21T14:09:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:09:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:09:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:09:17.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:16 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:09:17.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:17.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:17.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:17.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:17.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:17.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:17.816 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:17.881 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14974 14958 186.952 173.625 0.00382442 0.00534231 2024-03-21T14:09:17.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:17.952 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:17.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:18.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:09:18.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:09:18.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:18.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/298435573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:18.013 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[29576]: 6.17 scrub starts 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[29576]: 6.17 scrub ok 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/298435573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[40510]: 6.17 scrub starts 2024-03-21T14:09:18.014 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:17 smithi003 ceph-mon[40510]: 6.17 scrub ok 2024-03-21T14:09:18.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:18.090 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:18.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:18.192 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:18.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.218 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:18.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:18.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:18.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:17 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:17 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:17 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/298435573' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:17 smithi118 ceph-mon[33862]: 6.17 scrub starts 2024-03-21T14:09:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:17 smithi118 ceph-mon[33862]: 6.17 scrub ok 2024-03-21T14:09:18.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:18.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:18.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:18.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:18.634 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:18.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:18.771 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:18.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:18.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:18.881 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 17684 17668 184.02 169.375 0.00465665 0.00542829 2024-03-21T14:09:18.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:18.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:19.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:19.043 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:19.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.180 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:19.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:19.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:09:19.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:09:19.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[29576]: pgmap v273: 148 pgs: 148 active+clean; 700 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-21T14:09:19.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[29576]: osdmap e167: 8 total, 8 up, 8 in 2024-03-21T14:09:19.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:09:19.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:09:19.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:09:19.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:09:19.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[40510]: pgmap v273: 148 pgs: 148 active+clean; 700 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-21T14:09:19.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[40510]: osdmap e167: 8 total, 8 up, 8 in 2024-03-21T14:09:19.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:09:19.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:18 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:09:19.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:19.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:19.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:18 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:18 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:18 smithi118 ceph-mon[33862]: pgmap v273: 148 pgs: 148 active+clean; 700 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-21T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:18 smithi118 ceph-mon[33862]: osdmap e167: 8 total, 8 up, 8 in 2024-03-21T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:18 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:09:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:18 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:09:19.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:19.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:19.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:19.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:19.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:19.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:19.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:19.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:19.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:09:19.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:19.881 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 20352 20336 181.551 166.75 0.00945131 0.00550152 2024-03-21T14:09:19.894 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:19.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:19.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:19.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:20.029 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:20.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:20.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:09:20.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:09:20.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:09:20.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:09:20.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[29576]: pgmap v275: 148 pgs: 148 active+clean; 700 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-21T14:09:20.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:09:20.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:09:20.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:09:20.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:09:20.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:19 smithi003 ceph-mon[40510]: pgmap v275: 148 pgs: 148 active+clean; 700 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-21T14:09:20.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:20.165 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:20.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.166 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:20.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T14:09:20.267 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:20.301 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:20.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:19 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:09:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:19 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:09:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:19 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:09:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:19 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:09:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:19 smithi118 ceph-mon[33862]: pgmap v275: 148 pgs: 148 active+clean; 700 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-03-21T14:09:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:20.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:20.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:20.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:20.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:20.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:20.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:20.871 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:20.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:20.882 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 22726 22710 177.376 148.375 0.00994963 0.00562989 2024-03-21T14:09:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:20.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:21.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:21.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:21.119 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:21.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:20 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:09:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:20 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:09:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:20 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:09:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:20 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:09:21.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:21.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:21.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:21.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:21.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:21.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:20 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:09:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:20 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:09:21.524 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.557 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:21.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:21.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:09:21.658 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:21.687 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:21.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:21.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:21.821 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:21.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 24541 24525 170.268 113.438 0.0165883 0.00586587 2024-03-21T14:09:21.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:21.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:21.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:22.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:22.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:22.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.247 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:22.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:21 smithi003 ceph-mon[29576]: pgmap v276: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-03-21T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:09:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:21 smithi003 ceph-mon[40510]: pgmap v276: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-03-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:09:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:09:22.348 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:22.382 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:21 smithi118 ceph-mon[33862]: pgmap v276: 148 pgs: 148 active+clean; 1.0 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.12k op/s 2024-03-21T14:09:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:21 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:09:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:21 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:09:22.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:22.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:22.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:22.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.658 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:22.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:22.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:22.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:22.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 26408 26392 164.91 116.688 0.00689378 0.00605743 2024-03-21T14:09:22.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:22.934 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:22.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:23.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:23.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:23.131 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:23.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:23.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:23.208 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[29576]: 4.c deep-scrub starts 2024-03-21T14:09:23.209 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[29576]: 4.c deep-scrub ok 2024-03-21T14:09:23.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[40510]: 4.c deep-scrub starts 2024-03-21T14:09:23.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:22 smithi003 ceph-mon[40510]: 4.c deep-scrub ok 2024-03-21T14:09:23.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:23.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:23.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:22 smithi118 ceph-mon[33862]: 4.c deep-scrub starts 2024-03-21T14:09:23.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:22 smithi118 ceph-mon[33862]: 4.c deep-scrub ok 2024-03-21T14:09:23.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:23.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:23.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:23.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:23.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:23.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:23.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-21T14:09:23.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:23.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 28615 28599 162.457 137.938 0.00451163 0.00614904 2024-03-21T14:09:23.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:23.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:23.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:24.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:23 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:09:24.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:23 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:09:24.022 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:24.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:24.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:24.160 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:24.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:24.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:24.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:24 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:09:24.345 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:24 smithi003 ceph-mon[29576]: pgmap v277: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 3.00k op/s 2024-03-21T14:09:24.345 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:24 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:09:24.345 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:24 smithi003 ceph-mon[40510]: pgmap v277: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 3.00k op/s 2024-03-21T14:09:24.407 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:23 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:23 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:09:24.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:23 smithi118 ceph-mon[33862]: pgmap v277: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 187 MiB/s wr, 3.00k op/s 2024-03-21T14:09:24.440 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:24.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:24.576 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:24.678 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:24.702 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:24.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:24.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:24.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:24.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 30468 30452 158.569 115.812 0.00174683 0.00629073 2024-03-21T14:09:24.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:24.970 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:24.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:25.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:25.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:25.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:25 smithi003 ceph-mon[29576]: 3.4 deep-scrub starts 2024-03-21T14:09:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:25 smithi003 ceph-mon[29576]: 3.4 deep-scrub ok 2024-03-21T14:09:25.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:25 smithi003 ceph-mon[40510]: 3.4 deep-scrub starts 2024-03-21T14:09:25.311 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:25 smithi003 ceph-mon[40510]: 3.4 deep-scrub ok 2024-03-21T14:09:25.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:25.388 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:25.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:25 smithi118 ceph-mon[33862]: 3.4 deep-scrub starts 2024-03-21T14:09:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:25 smithi118 ceph-mon[33862]: 3.4 deep-scrub ok 2024-03-21T14:09:25.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:25.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:25.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:25.636 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:25.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:25.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:25.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:25.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:25.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 32319 32303 155.27 115.688 0.00531418 0.00643544 2024-03-21T14:09:25.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:25.932 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:25.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:25.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:26.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:26.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:26.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:26.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:26.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:26.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:26.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:26.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:26 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:09:26.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:26 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:09:26.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:26 smithi118 ceph-mon[33862]: 6.1c deep-scrub starts 2024-03-21T14:09:26.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:26 smithi118 ceph-mon[33862]: 6.1c deep-scrub ok 2024-03-21T14:09:26.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:26 smithi118 ceph-mon[33862]: pgmap v278: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-21T14:09:26.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:26.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:09:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[29576]: 6.1c deep-scrub starts 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[29576]: 6.1c deep-scrub ok 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[29576]: pgmap v278: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[40510]: 6.1c deep-scrub starts 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[40510]: 6.1c deep-scrub ok 2024-03-21T14:09:26.493 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-mon[40510]: pgmap v278: 148 pgs: 148 active+clean; 1.5 GiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 159 MiB/s wr, 2.55k op/s 2024-03-21T14:09:26.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:26.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:26.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:26.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:09:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:09:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:09:26.761 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:26.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:26.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 34336 34320 153.179 126.062 0.00933929 0.00652241 2024-03-21T14:09:26.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:26.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:26.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:09:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:09:27.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:27.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:27.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:27.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:27.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:09:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:09:27.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:09:27.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:27.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:09:27.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:27.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:27.470 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:27.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:27.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:27.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:27.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:27.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:27.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:27.883 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 36324 36308 151.25 124.25 0.00804709 0.00660624 2024-03-21T14:09:27.898 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:27.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:27.997 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:28.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:28.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:28.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:28.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:28.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:09:28.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 1.0 deep-scrub starts 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 1.0 deep-scrub ok 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[29576]: pgmap v279: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 1.0 deep-scrub starts 2024-03-21T14:09:28.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 1.0 deep-scrub ok 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:09:28.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:28 smithi003 ceph-mon[40510]: pgmap v279: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-21T14:09:28.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:28.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 1.0 deep-scrub starts 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 1.0 deep-scrub ok 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:09:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:09:28.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:09:28.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:09:28.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:09:28.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:28 smithi118 ceph-mon[33862]: pgmap v279: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-21T14:09:28.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:28.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:28.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:28.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:28.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:28.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:28.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:28.884 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 15 38036 38021 148.483 107.062 0.0066095 0.00672954 2024-03-21T14:09:28.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:28.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:28.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:29.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:29.106 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:09:29.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:29.196 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: 5.8 deep-scrub starts 2024-03-21T14:09:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: 5.8 deep-scrub ok 2024-03-21T14:09:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:09:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2033665968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:09:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: 3.0 scrub starts 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[29576]: 3.0 scrub ok 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: 5.8 deep-scrub starts 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: 5.8 deep-scrub ok 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2033665968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: 3.0 scrub starts 2024-03-21T14:09:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:29 smithi003 ceph-mon[40510]: 3.0 scrub ok 2024-03-21T14:09:29.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:29.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:29.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: 5.8 deep-scrub starts 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: 5.8 deep-scrub ok 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2033665968' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: 3.0 scrub starts 2024-03-21T14:09:29.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:29 smithi118 ceph-mon[33862]: 3.0 scrub ok 2024-03-21T14:09:29.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:29.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:29.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:29.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:29.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:29.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:29.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:29.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:29.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:29.884 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 40014 39998 147.017 123.562 0.0101147 0.00679685 2024-03-21T14:09:29.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:29.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:29.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:30.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:30.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:30.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:30.171 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:30.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:30.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.312 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:30.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:30.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:30.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:30 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:30.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:30 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2033665968' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:09:30.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:30 smithi118 ceph-mon[33862]: osdmap e168: 8 total, 8 up, 8 in 2024-03-21T14:09:30.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:30 smithi118 ceph-mon[33862]: pgmap v281: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-21T14:09:30.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:30.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:30.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:30.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2033665968' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:09:30.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[29576]: osdmap e168: 8 total, 8 up, 8 in 2024-03-21T14:09:30.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[29576]: pgmap v281: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-21T14:09:30.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:30.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2033665968' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:09:30.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[40510]: osdmap e168: 8 total, 8 up, 8 in 2024-03-21T14:09:30.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:30 smithi003 ceph-mon[40510]: pgmap v281: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-21T14:09:30.555 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:30.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:30.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:30.737 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:30.839 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:30.873 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:30.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:30.884 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 41974 41958 145.655 122.5 0.00964963 0.0068606 2024-03-21T14:09:30.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:31.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:31.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:31.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:31.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.296 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:31.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:31.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:31.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:31.434 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:31.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:31.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:31.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:31.536 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:31.572 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:31.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:31.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:31.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:31.849 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:31.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:31.884 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 43858 43842 144.186 117.75 0.0072709 0.00693064 2024-03-21T14:09:31.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:31.985 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:31.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:32.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:32.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:32.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:09:32.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:32.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:32 smithi003 ceph-mon[29576]: pgmap v282: 148 pgs: 148 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-21T14:09:32.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:32 smithi003 ceph-mon[40510]: pgmap v282: 148 pgs: 148 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-21T14:09:32.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:32.393 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:09:32.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:32.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:32.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:32 smithi118 ceph-mon[33862]: pgmap v282: 148 pgs: 148 active+clean; 2.3 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s 2024-03-21T14:09:32.453 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:09:32.489 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:09:32.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:32.532 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:09:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.539 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:32.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:32.575 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:09:32.625 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:09:32.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:32.665 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:09:32.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:32.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:32.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:09:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:09:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:09:32.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:32.817 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:32.884 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:09:32.885829+0000 min lat: 0.00121156 max lat: 0.127508 avg lat: 0.00705567 2024-03-21T14:09:32.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:09:32.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 45341 45325 141.611 92.6875 0.0135405 0.00705567 2024-03-21T14:09:32.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:32.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:32.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:33.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:33.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:33.096 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:33.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:33.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:33.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:33.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:33.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:33 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/786343203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:33 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/786343203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:33.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:33.372 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:33.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:33.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:33 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/786343203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:33.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:33.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:33.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:33.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:33.647 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:33.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:33.746 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:09:33.746 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:09:33.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:87195 7.a 7:59999b48:::benchmark_data_smithi003_80362_object87194:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.657703+0000", 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.61481525299999, 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11201855400000001, 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 87195 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657703+0000", 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657703+0000", 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657704+0000", 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4330000000000001e-06 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657735+0000", 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0791999999999999e-05 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657748+0000", 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2621e-05 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657756+0000", 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3569999999999998e-06 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657849+0000", 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2990000000000002e-05 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.657994+0000", 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000144914 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.658146+0000", 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015230899999999999 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.660966+0000", 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028199750000000002 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668207+0000", 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072408459999999996 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.769498+0000", 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.101290772 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.769669+0000", 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171359 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.769721+0000", 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2185999999999998e-05 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.752 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:87210 7.2 7:4fe49e08:::benchmark_data_smithi003_80362_object87209:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:05:36.668338+0000", 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 235.604180152, 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.112905648, 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 87210 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668338+0000", 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668338+0000", 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668339+0000", 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4169999999999999e-06 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668372+0000", 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2941999999999997e-05 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668382+0000", 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0273e-05 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668389+0000", 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9550000000000003e-06 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.755 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668425+0000", 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.595e-05 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.668521+0000", 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6049000000000001e-05 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.753667+0000", 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.085145891000000001 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.759992+0000", 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063248009999999997 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.769556+0000", 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095643250000000003 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.780974+0000", 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011417729 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.781123+0000", 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000148813 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:05:36.781243+0000", 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012050299999999999 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:216280 7.f 7:f89a58b1:::benchmark_data_smithi003_80362_object95263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:09:33.758 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:02.224319+0000", 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 90.048198494999994, 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113031408, 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216280 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224319+0000", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224319+0000", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224320+0000", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7599999999999997e-07 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224323+0000", 2024-03-21T14:09:33.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.19e-06 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224330+0000", 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.922e-06 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224336+0000", 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2910000000000004e-06 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.249153+0000", 2024-03-21T14:09:33.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024816110999999998 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.249276+0000", 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123471 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.249406+0000", 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013040999999999999 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253123+0000", 2024-03-21T14:09:33.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003716254 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253754+0000", 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063169099999999998 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327175+0000", 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073420507999999995 2024-03-21T14:09:33.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337107+0000", 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0099322019999999994 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337209+0000", 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101482 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337351+0000", 2024-03-21T14:09:33.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014229999999999999 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:216283 7.f 7:f841127a:::benchmark_data_smithi003_80362_object95266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:02.224403+0000", 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 90.048114490000003, 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113449484, 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216283 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224403+0000", 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224403+0000", 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224404+0000", 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-07 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224406+0000", 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3029999999999998e-06 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224412+0000", 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3879999999999999e-06 2024-03-21T14:09:33.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224416+0000", 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8969999999999999e-06 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253807+0000", 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029390899000000002 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253936+0000", 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128765 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.254105+0000", 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016864199999999999 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327220+0000", 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073115515000000006 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327259+0000", 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8886e-05 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327298+0000", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8504000000000003e-05 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337363+0000", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010065780999999999 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337522+0000", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015878800000000001 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337853+0000", 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000330716 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221713 7.f 7:fc6b6dba:::benchmark_data_smithi003_80362_object100696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.074539823999999, 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93373331800000003, 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.774 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221713 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197980+0000", 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.705e-06 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197988+0000", 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2139999999999996e-06 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198007+0000", 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9247999999999999e-05 2024-03-21T14:09:33.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198018+0000", 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0268000000000001e-05 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198077+0000", 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9280999999999997e-05 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198085+0000", 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7959999999999996e-06 2024-03-21T14:09:33.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119714+0000", 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92162893099999998 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119872+0000", 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158045 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.120011+0000", 2024-03-21T14:09:33.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013935900000000001 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131462+0000", 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011450777000000001 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131476+0000", 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4005999999999999e-05 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131627+0000", 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000150791 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131673+0000", 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6919000000000003e-05 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131711+0000", 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7978000000000001e-05 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.780 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221725 7.f 7:fe480980:::benchmark_data_smithi003_80362_object100708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.046724822000002, 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90603860999999997, 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221725 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.781 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:09:33.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225795+0000", 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0959999999999999e-06 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225804+0000", 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8510000000000005e-06 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.783 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225818+0000", 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3557000000000001e-05 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225828+0000", 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0644e-05 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225939+0000", 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110461 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.784 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225950+0000", 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1712999999999999e-05 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123657+0000", 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.897706589 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123780+0000", 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012321000000000001 2024-03-21T14:09:33.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123890+0000", 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110142 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131508+0000", 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076180889999999998 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131594+0000", 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5677e-05 2024-03-21T14:09:33.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131747+0000", 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015268700000000001 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131788+0000", 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1032999999999998e-05 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131832+0000", 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3860999999999999e-05 2024-03-21T14:09:33.787 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221726 7.f 7:fbed2f09:::benchmark_data_smithi003_80362_object100709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.046668566999998, 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91368115900000002, 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221726 2024-03-21T14:09:33.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225850+0000", 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7000000000000005e-07 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225854+0000", 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5920000000000001e-06 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225861+0000", 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4329999999999998e-06 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225867+0000", 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2999999999999998e-06 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225979+0000", 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111437 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225986+0000", 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9e-06 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127686+0000", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90170028599999996 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127815+0000", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012929099999999999 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127949+0000", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013351500000000001 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139334+0000", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011385072 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139378+0000", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3952999999999997e-05 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139425+0000", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6764999999999997e-05 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139479+0000", 2024-03-21T14:09:33.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4245e-05 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139531+0000", 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999999e-05 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221737 7.f 7:f1651cbf:::benchmark_data_smithi003_80362_object100720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.035003115999999, 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90934260099999997, 2024-03-21T14:09:33.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221737 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237516+0000", 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0100000000000001e-06 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237521+0000", 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5430000000000001e-06 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237538+0000", 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7096e-05 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237547+0000", 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9870000000000005e-06 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237607+0000", 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9367000000000003e-05 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237616+0000", 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5820000000000008e-06 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131857+0000", 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89424055899999999 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131984+0000", 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012695299999999999 2024-03-21T14:09:33.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.132098+0000", 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113804 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139390+0000", 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072926889999999998 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139610+0000", 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000219626 2024-03-21T14:09:33.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146690+0000", 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070798809999999997 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146801+0000", 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110599 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146857+0000", 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6904999999999998e-05 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.800 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221741 7.f 7:fa1cb433:::benchmark_data_smithi003_80362_object100724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.03108426, 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90562954600000001, 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221741 2024-03-21T14:09:33.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241435+0000", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4470000000000001e-06 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241442+0000", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2640000000000002e-06 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241460+0000", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7186000000000001e-05 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241469+0000", 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9719999999999998e-06 2024-03-21T14:09:33.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241541+0000", 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2210000000000002e-05 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241551+0000", 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9399999999999997e-06 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135494+0000", 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89394319300000002 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135628+0000", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013400099999999999 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135788+0000", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016006200000000001 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139568+0000", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037803009999999998 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139659+0000", 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0662999999999996e-05 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146934+0000", 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072749190000000004 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.147017+0000", 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2694000000000005e-05 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.147063+0000", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6693999999999997e-05 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221746 7.f 7:fec8ebdb:::benchmark_data_smithi003_80362_object100729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.026526275999998, 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90374667900000005, 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.807 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221746 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245993+0000", 2024-03-21T14:09:33.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.432e-06 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245999+0000", 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.088e-06 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246013+0000", 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4158999999999999e-05 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246022+0000", 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7050000000000005e-06 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246089+0000", 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6668999999999995e-05 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246098+0000", 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6609999999999995e-06 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139691+0000", 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89359230599999995 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139805+0000", 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114026 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139913+0000", 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108721 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146871+0000", 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069575490000000004 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149464+0000", 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025927440000000001 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149513+0000", 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8949000000000002e-05 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149686+0000", 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000173308 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149738+0000", 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2361999999999998e-05 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221753 7.f 7:f88a5a14:::benchmark_data_smithi003_80362_object100736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:09:33.813 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.020453505999996, 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89806965699999997, 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221753 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252065+0000", 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0200000000000002e-07 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252069+0000", 2024-03-21T14:09:33.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7479999999999999e-06 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252076+0000", 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252083+0000", 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5230000000000001e-06 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252137+0000", 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4317999999999997e-05 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252147+0000", 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0257e-05 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143316+0000", 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89116910800000004 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143431+0000", 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011446300000000001 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143540+0000", 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108909 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149748+0000", 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062084599999999998 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149786+0000", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7997000000000002e-05 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149826+0000", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9373999999999999e-05 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150000+0000", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017394 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150134+0000", 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013446700000000001 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221755 7.f 7:f52429fb:::benchmark_data_smithi003_80362_object100738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 83.015795752000002, 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90123331200000001, 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.820 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221755 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256723+0000", 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.223e-06 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256729+0000", 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7960000000000001e-06 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256742+0000", 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2394e-05 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256750+0000", 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9270000000000005e-06 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256802+0000", 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2578999999999999e-05 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256811+0000", 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1349999999999998e-06 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150166+0000", 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89335426299999998 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150275+0000", 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109422 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150383+0000", 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108297 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157790+0000", 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074066669999999996 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157802+0000", 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2245e-05 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157842+0000", 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0120000000000002e-05 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157908+0000", 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6168999999999996e-05 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157956+0000", 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7074999999999999e-05 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221760 7.f 7:f2fb3067:::benchmark_data_smithi003_80362_object100743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:09:33.826 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 82.988336732999997, 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89272659300000001, 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221760 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284183+0000", 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.711e-06 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284190+0000", 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5209999999999997e-06 2024-03-21T14:09:33.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284207+0000", 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6325e-05 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284215+0000", 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4319999999999999e-06 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284280+0000", 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5042000000000005e-05 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284292+0000", 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2099000000000001e-05 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154224+0000", 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86993135600000004 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154329+0000", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010535300000000001 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154439+0000", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011024 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.166586+0000", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012147120000000001 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.166631+0000", 2024-03-21T14:09:33.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4297e-05 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176731+0000", 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010100074000000001 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176826+0000", 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5345000000000003e-05 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176908+0000", 2024-03-21T14:09:33.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1678e-05 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30436 8.f 8:f8a7c51e:::benchmark_data_smithi003_110242_object30435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.4176354809999996, 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12503477399999999, 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:33.833 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30436 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.834 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854883+0000", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854905+0000", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1386000000000002e-05 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854915+0000", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0138e-05 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854922+0000", 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2810000000000003e-06 2024-03-21T14:09:33.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860219+0000", 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052964709999999996 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860345+0000", 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012642600000000001 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860468+0000", 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122581 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866120+0000", 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056519700000000001 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866164+0000", 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4041000000000001e-05 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866321+0000", 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015716800000000001 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866370+0000", 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8786999999999999e-05 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979917+0000", 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11354760999999999 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.838 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30438 8.f 8:fe80adc5:::benchmark_data_smithi003_110242_object30437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.4174014709999998, 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12806651599999999, 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30438 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855117+0000", 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8999999999999995e-07 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855163+0000", 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6081000000000001e-05 2024-03-21T14:09:33.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855173+0000", 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6180000000000001e-06 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855180+0000", 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.81e-06 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863014+0000", 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078345389999999997 2024-03-21T14:09:33.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863134+0000", 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119196 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863267+0000", 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133738 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866211+0000", 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002943373 2024-03-21T14:09:33.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979935+0000", 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113724297 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.982232+0000", 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002297112 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.982341+0000", 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010846800000000001 2024-03-21T14:09:33.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983183+0000", 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084249399999999995 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30445 8.f 8:f69cc62f:::benchmark_data_smithi003_110242_object30444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.4142505209999996, 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12520763700000001, 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.844 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30445 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858269+0000", 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.344e-06 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858324+0000", 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4917000000000003e-05 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858335+0000", 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1732999999999999e-05 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858344+0000", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2150000000000001e-06 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864522+0000", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061783519999999998 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864635+0000", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113094 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:33.847 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864765+0000", 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129839 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979940+0000", 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115175427 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980642+0000", 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070184399999999997 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983236+0000", 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025941480000000001 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983433+0000", 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019648300000000001 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983475+0000", 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2240999999999998e-05 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.849 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30459 8.0 8:00d9c66a:::benchmark_data_smithi003_110242_object30458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.862891+0000", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.409626898, 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.121083247, 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30459 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862891+0000", 2024-03-21T14:09:33.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862891+0000", 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862892+0000", 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2099999999999996e-07 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862911+0000", 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8842e-05 2024-03-21T14:09:33.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862921+0000", 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0312999999999999e-05 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862938+0000", 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7220000000000001e-05 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863494+0000", 2024-03-21T14:09:33.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055609799999999999 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863653+0000", 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158682 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863788+0000", 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135127 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865255+0000", 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001466619 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865972+0000", 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00071715400000000001 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983870+0000", 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.117898002 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983934+0000", 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3995e-05 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983974+0000", 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0473999999999999e-05 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30464 8.0 8:02e6bb23:::benchmark_data_smithi003_110242_object30463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.865950+0000", 2024-03-21T14:09:33.855 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.4065683189999998, 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120651603, 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30464 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865950+0000", 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865950+0000", 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865951+0000", 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.248e-06 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865970+0000", 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8947999999999999e-05 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865979+0000", 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.431e-06 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865986+0000", 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3810000000000004e-06 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866019+0000", 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3278000000000002e-05 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866113+0000", 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4518000000000005e-05 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866253+0000", 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013929099999999999 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981357+0000", 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11510411 2024-03-21T14:09:33.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.984021+0000", 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026637779999999999 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.986471+0000", 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024504570000000001 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.986560+0000", 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9351999999999994e-05 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.986601+0000", 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0810999999999999e-05 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15132.0:30445 8.f e167/162 ondisk, result = 0)", 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.4060371500000004, 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114169123, 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8700000000000005e-07 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866486+0000", 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7090000000000001e-06 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979873+0000", 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113386367 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979886+0000", 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3909e-05 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980628+0000", 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074143099999999997 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980636+0000", 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3909999999999998e-06 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980650+0000", 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3628999999999999e-05 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15132.0:30464 8.0 e167/162 ondisk, result = 0)", 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 7.4051643479999996, 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114010399, 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7700000000000004e-07 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867358+0000", 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5760000000000002e-06 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979959+0000", 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11260105500000001 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979969+0000", 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0672999999999999e-05 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981344+0000", 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013744600000000001 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981351+0000", 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0090000000000001e-06 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981364+0000", 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3049e-05 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:09:33.868 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:09:33.869 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:33.884 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 47218 47202 140.453 117.312 0.00873296 0.00711503 2024-03-21T14:09:33.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:33.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:33.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:34.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:34.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:34.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:34.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:34.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.321 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:34.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:34.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:34 smithi118 ceph-mon[33862]: pgmap v283: 148 pgs: 148 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-21T14:09:34.451 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:34 smithi003 ceph-mon[29576]: pgmap v283: 148 pgs: 148 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-21T14:09:34.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:34 smithi003 ceph-mon[40510]: pgmap v283: 148 pgs: 148 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-21T14:09:34.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:34.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:34.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:34.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:34.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:34.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:34.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:34.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 48918 48902 138.898 106.25 0.00837119 0.0071944 2024-03-21T14:09:34.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:35.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:35.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:35.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:35.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:35.245 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:35.278 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:35.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:35.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:35.414 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:35.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:09:35.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:35.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:35.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:35.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:35.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:35.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:35.848 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:35.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 50670 50654 137.62 109.5 0.00937957 0.00726122 2024-03-21T14:09:35.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:35.972 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:35.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:35.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:36.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:36.107 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:36.209 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:36.240 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:36.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:36.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:36.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:36.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:36.514 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:36.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:36.624 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:36 smithi118 ceph-mon[33862]: pgmap v284: 148 pgs: 148 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-21T14:09:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:36.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:36.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:36 smithi003 ceph-mon[29576]: pgmap v284: 148 pgs: 148 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-21T14:09:36.719 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:09:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:09:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:09:36.719 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:36 smithi003 ceph-mon[40510]: pgmap v284: 148 pgs: 148 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-21T14:09:36.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:36.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:36.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 52309 52293 136.154 102.438 0.0154052 0.00733883 2024-03-21T14:09:36.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:36.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:09:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:09:36.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:36.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:37.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:37.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:37.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:37.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:37.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:37.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:37.374 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:37.476 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:37.505 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:37.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:37.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:37.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:37.667 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:37.748 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:37.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:37.776 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:37.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:37.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 53710 53694 134.21 87.5625 0.0487982 0.00743062 2024-03-21T14:09:37.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:37.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:38.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.044 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:38.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:38.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:38.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:38.283 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:38.298 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-21T14:09:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.319 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:38.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:38.402 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:09:38.403 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:09:38.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:38.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:38.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:38 smithi003 ceph-mon[29576]: pgmap v285: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-21T14:09:38.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:38 smithi003 ceph-mon[40510]: pgmap v285: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-21T14:09:38.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:09:38.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:38.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:38.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:38.668 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:38 smithi118 ceph-mon[33862]: pgmap v285: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.82k op/s 2024-03-21T14:09:38.710 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:38.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:38.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:38.876 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:38.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:38.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 15 55135 55120 132.475 89.125 0.0646583 0.00753337 2024-03-21T14:09:38.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:39.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:39.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:39.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:39.257 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:39.296 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:39.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:09:39.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:39.404 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:39.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:39.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:39.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:39.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:39.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:39.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:39.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:39.795 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:39.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:39.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 56565 56549 130.877 89.3125 0.0102153 0.00763556 2024-03-21T14:09:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:39.912 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:39.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:40.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:40.035 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:40.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:40.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:40.180 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:40.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:40.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:40.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:40.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:40.423 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:40.452 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:40.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:40.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:40.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:40.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:40.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:40 smithi003 ceph-mon[29576]: pgmap v286: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 997 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T14:09:40.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:40 smithi003 ceph-mon[40510]: pgmap v286: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 997 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T14:09:40.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:40.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:40.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:40.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:40.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 15 58079 58064 129.584 94.6875 0.00742847 0.00771228 2024-03-21T14:09:40.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:40 smithi118 ceph-mon[33862]: pgmap v286: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 997 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-21T14:09:40.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:41.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:41.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:41.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:41.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:41.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:41.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:41.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:41.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:41.504 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:09:41.522 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4045897595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:41.523 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:41 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:41.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4045897595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:41.524 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:41 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:41.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:41.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4045897595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:41.553 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:41 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:09:41.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:41.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:41.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:41.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:41.838 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:41.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 59537 59521 128.255 91.0625 0.0199346 0.00779182 2024-03-21T14:09:41.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:41.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:41.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:42.020 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:09:42.020 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:09:42.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:42.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:42.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:42.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:42.253 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:42.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:42.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:42.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:42.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:42.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:42.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:42.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[29576]: pgmap v287: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T14:09:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:09:42.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[29576]: osdmap e169: 8 total, 8 up, 8 in 2024-03-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[40510]: pgmap v287: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:09:42.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:42 smithi003 ceph-mon[40510]: osdmap e169: 8 total, 8 up, 8 in 2024-03-21T14:09:42.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:42.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 61104 61088 127.245 97.9375 0.00644541 0.00785413 2024-03-21T14:09:42.915 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:42 smithi118 ceph-mon[33862]: pgmap v287: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T14:09:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:42 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:09:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:42 smithi118 ceph-mon[33862]: osdmap e169: 8 total, 8 up, 8 in 2024-03-21T14:09:42.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:42.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:42.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:43.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:09:43.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:43.090 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:43.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:43.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:43.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:43.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:43.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:43.398 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:43.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.513 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:43.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:43.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:43.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:43.748 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:43.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:43.887 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 62537 62521 126.029 89.5625 0.00541948 0.00792988 2024-03-21T14:09:43.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:43.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:44.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:44.055 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:44.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:44.192 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:44.294 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:44.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:44.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:44.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:44.470 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:44.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:44.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:44.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:44.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:09:44.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:44.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:44.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:44 smithi003 ceph-mon[29576]: pgmap v289: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:09:44.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:44 smithi003 ceph-mon[40510]: pgmap v289: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:09:44.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:44.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:44.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:44.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:44.885 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 63939 63923 124.829 87.625 0.0116704 0.00800542 2024-03-21T14:09:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:44 smithi118 ceph-mon[33862]: pgmap v289: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:09:44.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:45.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:45.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:45.164 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:45.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:45.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:45.294 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:45.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:45.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:45.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:45.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:45.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:45.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:45.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:45.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:45.755 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:09:45.793 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:09:45.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:45.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:09:45.872 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:09:45.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:45.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:45.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:45.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 65309 65293 123.641 85.625 0.0048103 0.00808211 2024-03-21T14:09:45.910 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:09:45.958 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:09:45.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:46.010 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:09:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:09:46.041 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:09:46.042 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:09:46.042 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:09:46.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:46.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:46.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:46.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:46.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:46.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:46.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193115 7.4 7:23e77ff3:::benchmark_data_smithi003_80362_object72098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.718159+0000", 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.69317294300001, 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0401704999999999, 2024-03-21T14:09:46.416 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193115 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718159+0000", 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718159+0000", 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718160+0000", 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909999999999999e-06 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718165+0000", 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0649999999999998e-06 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718177+0000", 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1471000000000001e-05 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718183+0000", 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2339999999999999e-06 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718232+0000", 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8414000000000001e-05 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718324+0000", 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880999999999994e-05 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,6", 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718401+0000", 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6329999999999996e-05 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.720669+0000", 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00226822 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.720714+0000", 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4978000000000002e-05 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.726514+0000", 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057999740000000003 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.726526+0000", 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2184999999999999e-05 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758285+0000", 2024-03-21T14:09:46.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.031759222 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758330+0000", 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4335000000000002e-05 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193119 7.4 7:2d9b1e7b:::benchmark_data_smithi003_80362_object72102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.730256+0000", 2024-03-21T14:09:46.423 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.68107573500001, 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0340747850000001, 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193119 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730256+0000", 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730256+0000", 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730258+0000", 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.203e-06 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730262+0000", 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5430000000000004e-06 2024-03-21T14:09:46.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730272+0000", 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0074e-05 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730278+0000", 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0299999999999999e-06 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730319+0000", 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0463000000000002e-05 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730325+0000", 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6710000000000002e-06 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758690+0000", 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.028364201 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758787+0000", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7291000000000001e-05 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758888+0000", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101508 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764227+0000", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053386040000000003 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764265+0000", 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7851000000000001e-05 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764275+0000", 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0709e-05 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764308+0000", 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2449000000000001e-05 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764331+0000", 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3187999999999999e-05 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193125 7.4 7:2ef2efbc:::benchmark_data_smithi003_80362_object72108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.749172+0000", 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.66215963499999, 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.019614807, 2024-03-21T14:09:46.430 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193125 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749172+0000", 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749172+0000", 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749174+0000", 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4559999999999999e-06 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749180+0000", 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4030000000000004e-06 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749194+0000", 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3648e-05 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749202+0000", 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4919999999999993e-06 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749263+0000", 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0850000000000002e-05 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749272+0000", 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1689999999999998e-06 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.761037+0000", 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117644429999999 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.761135+0000", 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7755000000000002e-05 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.761212+0000", 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6964000000000004e-05 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764339+0000", 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031274359999999999 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768583+0000", 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042440500000000001 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768677+0000", 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3974999999999994e-05 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768734+0000", 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6718000000000001e-05 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768787+0000", 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3448000000000001e-05 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193129 7.4 7:2937aaf4:::benchmark_data_smithi003_80362_object72112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.754621+0000", 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.65671144000001, 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.014328082, 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193129 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754621+0000", 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754621+0000", 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754622+0000", 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754626+0000", 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9940000000000002e-06 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754636+0000", 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0409e-05 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754644+0000", 2024-03-21T14:09:46.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5530000000000004e-06 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754691+0000", 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7607e-05 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754700+0000", 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1990000000000006e-06 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.762687+0000", 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.007987382 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.762750+0000", 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3196999999999998e-05 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.762817+0000", 2024-03-21T14:09:46.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7040999999999997e-05 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768592+0000", 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057748820000000003 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768629+0000", 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6761000000000003e-05 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768843+0000", 2024-03-21T14:09:46.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021430599999999999 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768899+0000", 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5597999999999999e-05 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768949+0000", 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9920000000000003e-05 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.444 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193143 7.4 7:20ba6c65:::benchmark_data_smithi003_80362_object72126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.769319+0000", 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.64201354100001, 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99980547399999997, 2024-03-21T14:09:46.446 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193143 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769319+0000", 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.447 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769319+0000", 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769320+0000", 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.274e-06 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769326+0000", 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7520000000000002e-06 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769340+0000", 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4537e-05 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769349+0000", 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6819999999999992e-06 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769402+0000", 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3217999999999998e-05 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769408+0000", 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7400000000000001e-06 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764364+0000", 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99495648800000003 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764429+0000", 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4486999999999994e-05 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764494+0000", 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5752000000000003e-05 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768800+0000", 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043050409999999999 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768984+0000", 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018443600000000001 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769021+0000", 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7500999999999999e-05 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769073+0000", 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1650000000000002e-05 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769124+0000", 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0915999999999998e-05 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193164 7.4 7:2c7218ff:::benchmark_data_smithi003_80362_object72147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.797383+0000", 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.613949211, 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97718391999999998, 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193164 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.453 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797383+0000", 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797383+0000", 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797384+0000", 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8700000000000004e-07 2024-03-21T14:09:46.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797388+0000", 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8770000000000003e-06 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797395+0000", 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9859999999999997e-06 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.455 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797400+0000", 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9830000000000004e-06 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797445+0000", 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5787000000000001e-05 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797451+0000", 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9320000000000001e-06 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.766211+0000", 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96876004699999996 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.766316+0000", 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105072 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.766435+0000", 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118355 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768991+0000", 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025563259999999998 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769158+0000", 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016738300000000001 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774462+0000", 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053030890000000004 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774518+0000", 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6388000000000003e-05 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774567+0000", 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8807999999999998e-05 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193165 7.4 7:27395a1d:::benchmark_data_smithi003_80362_object72148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.459 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.797413+0000", 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.61391898700001, 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.981967598, 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193165 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.460 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797413+0000", 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797413+0000", 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797414+0000", 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2599999999999998e-07 2024-03-21T14:09:46.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797416+0000", 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2340000000000001e-06 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797421+0000", 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5910000000000003e-06 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.462 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797427+0000", 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.344e-06 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797466+0000", 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9601000000000003e-05 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797470+0000", 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3450000000000002e-06 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.463 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769193+0000", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97172361500000004 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769308+0000", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011507500000000001 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769435+0000", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126489 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774623+0000", 2024-03-21T14:09:46.464 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051881729999999999 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774637+0000", 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4122e-05 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779266+0000", 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046288290000000001 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779336+0000", 2024-03-21T14:09:46.465 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9722999999999996e-05 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779381+0000", 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5031000000000003e-05 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193181 7.4 7:203760eb:::benchmark_data_smithi003_80362_object72164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.826429+0000", 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.584902809, 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95319384900000004, 2024-03-21T14:09:46.466 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193181 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826429+0000", 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.467 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826429+0000", 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826431+0000", 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.48e-06 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826437+0000", 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8490000000000002e-06 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.468 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826449+0000", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2614e-05 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826457+0000", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6760000000000007e-06 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826506+0000", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9452e-05 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826515+0000", 2024-03-21T14:09:46.469 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1409999999999994e-06 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.772040+0000", 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94552436299999998 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.772148+0000", 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108558 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.772267+0000", 2024-03-21T14:09:46.470 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118244 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779443+0000", 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071765889999999997 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779458+0000", 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4436e-05 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779564+0000", 2024-03-21T14:09:46.471 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105952 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779596+0000", 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2041e-05 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779623+0000", 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7453999999999998e-05 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.472 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193188 7.4 7:2a95fdc0:::benchmark_data_smithi003_80362_object72171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.833625+0000", 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.57770660599999, 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95566202300000003, 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193188 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.473 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833625+0000", 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833625+0000", 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833627+0000", 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3060000000000001e-06 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.474 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833632+0000", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9060000000000001e-06 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833645+0000", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3298999999999999e-05 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833653+0000", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2109999999999998e-06 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.475 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833709+0000", 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5512e-05 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833718+0000", 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3109999999999995e-06 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774673+0000", 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94095470599999997 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.476 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774792+0000", 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119088 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774925+0000", 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013321599999999999 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779462+0000", 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045366859999999998 2024-03-21T14:09:46.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779512+0000", 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9988000000000003e-05 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789180+0000", 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096685169999999997 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789237+0000", 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6954999999999999e-05 2024-03-21T14:09:46.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789287+0000", 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0321999999999997e-05 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193233 7.4 7:2c84c41f:::benchmark_data_smithi003_80362_object72216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.896923+0000", 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.51440929399999, 2024-03-21T14:09:46.479 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89245795400000005, 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193233 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.896923+0000", 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.480 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.896923+0000", 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.896924+0000", 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5969999999999999e-06 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.896930+0000", 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0449999999999997e-06 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.481 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.896945+0000", 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4858000000000001e-05 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.896952+0000", 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1189999999999999e-06 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.482 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.897021+0000", 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8626000000000003e-05 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.897029+0000", 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8189999999999992e-06 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.776940+0000", 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87991138800000002 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.483 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.777053+0000", 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011263 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.777178+0000", 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012491600000000001 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779632+0000", 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024543669999999998 2024-03-21T14:09:46.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779682+0000", 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9602999999999999e-05 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789320+0000", 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096384399999999999 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.485 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789353+0000", 2024-03-21T14:09:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:46.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3181e-05 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789381+0000", 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7365e-05 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193288 7.4 7:2e613ec8:::benchmark_data_smithi003_80362_object72271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.984391+0000", 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.42694082700001, 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81684215699999996, 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193288 2024-03-21T14:09:46.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984391+0000", 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984391+0000", 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.490 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984392+0000", 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1739999999999999e-06 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984397+0000", 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7079999999999996e-06 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984417+0000", 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0305e-05 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.491 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984424+0000", 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2539999999999998e-06 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984480+0000", 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6022e-05 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.984485+0000", 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.716e-06 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.492 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779718+0000", 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.79523280900000004 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779788+0000", 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9331000000000004e-05 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779893+0000", 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105251 2024-03-21T14:09:46.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789023+0000", 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091298149999999995 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789034+0000", 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1787000000000001e-05 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.494 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801113+0000", 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01207854 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801178+0000", 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5007999999999998e-05 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801233+0000", 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5436999999999998e-05 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193329 7.4 7:22c4b8e9:::benchmark_data_smithi003_80362_object72312:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:34.039870+0000", 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.37146218000001, 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.76159102199999995, 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193329 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.496 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039870+0000", 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039870+0000", 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039871+0000", 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4360000000000001e-06 2024-03-21T14:09:46.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039877+0000", 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0809999999999999e-06 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039891+0000", 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3677e-05 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039899+0000", 2024-03-21T14:09:46.498 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6939999999999995e-06 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039968+0000", 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9344999999999994e-05 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.039974+0000", 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7660000000000004e-06 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.499 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.782107+0000", 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74213330799999999 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.782196+0000", 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8558000000000002e-05 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.782264+0000", 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8619000000000001e-05 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.500 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789038+0000", 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067738809999999998 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789086+0000", 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8016999999999998e-05 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801310+0000", 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012223968999999999 2024-03-21T14:09:46.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801415+0000", 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104695 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801461+0000", 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5976000000000002e-05 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193332 7.4 7:258618fe:::benchmark_data_smithi003_80362_object72315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:34.040533+0000", 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.37079894300001, 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.76809493799999995, 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193332 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040533+0000", 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040533+0000", 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040534+0000", 2024-03-21T14:09:46.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040538+0000", 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7189999999999999e-06 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040548+0000", 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0562e-05 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040554+0000", 2024-03-21T14:09:46.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3789999999999997e-06 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040599+0000", 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5435999999999997e-05 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.040606+0000", 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0310000000000001e-06 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.783898+0000", 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.74329215000000004 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.783996+0000", 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8116999999999996e-05 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.784117+0000", 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120711 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.507 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789117+0000", 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005000422 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789131+0000", 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3740999999999999e-05 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808515+0000", 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019384321999999999 2024-03-21T14:09:46.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808583+0000", 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7892999999999995e-05 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808628+0000", 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4719000000000003e-05 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193336 7.4 7:2e460f6c:::benchmark_data_smithi003_80362_object72319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:34.047327+0000", 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.364005408, 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.76141203599999996, 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193336 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047327+0000", 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047327+0000", 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047328+0000", 2024-03-21T14:09:46.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2100000000000001e-06 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047333+0000", 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2100000000000001e-06 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047346+0000", 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2588e-05 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047352+0000", 2024-03-21T14:09:46.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2099999999999998e-06 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047403+0000", 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1159999999999998e-05 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.047409+0000", 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9349999999999999e-06 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.786361+0000", 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.73895222999999999 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.786477+0000", 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115703 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.786604+0000", 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012691499999999999 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789138+0000", 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002534194 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789438+0000", 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000300115 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808664+0000", 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.019225507999999999 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808706+0000", 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2583999999999998e-05 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808739+0000", 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2474000000000001e-05 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193362 7.4 7:2aa80363:::benchmark_data_smithi003_80362_object72345:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.516 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:34.104305+0000", 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.307026831, 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.707612876, 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193362 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104305+0000", 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104305+0000", 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104307+0000", 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104313+0000", 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5660000000000003e-06 2024-03-21T14:09:46.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104326+0000", 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2259000000000001e-05 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104334+0000", 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4039999999999995e-06 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104394+0000", 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9865999999999999e-05 2024-03-21T14:09:46.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.104403+0000", 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7220000000000005e-06 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789484+0000", 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.68508131100000003 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789578+0000", 2024-03-21T14:09:46.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4049999999999996e-05 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789668+0000", 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0362e-05 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801590+0000", 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011922021 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.808990+0000", 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073997159999999998 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.811810+0000", 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028197439999999999 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.811859+0000", 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8918000000000002e-05 2024-03-21T14:09:46.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.811918+0000", 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9422000000000001e-05 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193363 7.4 7:2cc21177:::benchmark_data_smithi003_80362_object72346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:34.105621+0000", 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 132.305710619, 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.70752038500000003, 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.523 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193363 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105621+0000", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105621+0000", 2024-03-21T14:09:46.524 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105622+0000", 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7300000000000005e-07 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105626+0000", 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9720000000000003e-06 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.525 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105636+0000", 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2569999999999996e-06 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105641+0000", 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2989999999999996e-06 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105681+0000", 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0330999999999997e-05 2024-03-21T14:09:46.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.105686+0000", 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0780000000000003e-06 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.794014+0000", 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.68832804000000003 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.794128+0000", 2024-03-21T14:09:46.527 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011381 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.794239+0000", 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000111316 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.801636+0000", 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073965309999999996 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.811930+0000", 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010293623999999999 2024-03-21T14:09:46.528 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.813006+0000", 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001076844 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.813098+0000", 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1772000000000002e-05 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.529 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.813142+0000", 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3637999999999998e-05 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:196351 7.4 7:2581001f:::benchmark_data_smithi003_80362_object75334:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:38.059724+0000", 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 128.351608175, 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.107393117, 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.530 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 196351 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.059724+0000", 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.059724+0000", 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.531 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.059725+0000", 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4309999999999999e-06 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.059731+0000", 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7470000000000002e-06 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.149113+0000", 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.089381879999999997 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.532 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.149126+0000", 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2641e-05 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.149374+0000", 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024879299999999997 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.149495+0000", 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012104600000000001 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.533 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.149611+0000", 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115319 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.157946+0000", 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0083349610000000001 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.157990+0000", 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4393e-05 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.534 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.167002+0000", 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090114759999999992 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.167059+0000", 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7017e-05 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:38.167117+0000", 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8412999999999998e-05 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.535 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:30162 8.b 8:d0c8b97d:::benchmark_data_smithi003_110242_object30161:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:24.647238+0000", 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.764093975000002, 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.096807878, 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30162 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.536 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647238+0000", 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647238+0000", 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647239+0000", 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3379999999999999e-06 2024-03-21T14:09:46.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647268+0000", 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8941999999999998e-05 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647279+0000", 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0254999999999999e-05 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647287+0000", 2024-03-21T14:09:46.538 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0639999999999994e-06 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647341+0000", 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4344e-05 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647463+0000", 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121954 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.647598+0000", 2024-03-21T14:09:46.539 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013496600000000001 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.648895+0000", 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012970340000000001 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650456+0000", 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015606330000000001 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.743943+0000", 2024-03-21T14:09:46.540 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.093487818 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.743991+0000", 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.791e-05 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.744046+0000", 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4620000000000002e-05 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:30169 8.b 8:d7d0c1a8:::benchmark_data_smithi003_110242_object30168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:24.650240+0000", 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 21.761092278, 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.098406342999999993, 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.542 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30169 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650240+0000", 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.543 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650240+0000", 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650241+0000", 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2800000000000005e-07 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650266+0000", 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5647999999999999e-05 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.544 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650277+0000", 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0681e-05 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650283+0000", 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2920000000000001e-06 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650494+0000", 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021094000000000001 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.545 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650640+0000", 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014619899999999999 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.650772+0000", 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013146800000000001 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.651705+0000", 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00093339099999999999 2024-03-21T14:09:46.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.652229+0000", 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000523425 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.748564+0000", 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.096335407999999997 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.748600+0000", 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6174000000000003e-05 2024-03-21T14:09:46.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.748646+0000", 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5788999999999998e-05 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:31922 8.b 8:de47c025:::benchmark_data_smithi003_110242_object31921:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:25.626732+0000", 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 20.784600464, 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.098165679000000006, 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:46.548 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 31922 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626732+0000", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626732+0000", 2024-03-21T14:09:46.549 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626733+0000", 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.387e-06 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626829+0000", 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6415000000000004e-05 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626842+0000", 2024-03-21T14:09:46.550 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2994e-05 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626850+0000", 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8990000000000001e-06 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.626904+0000", 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3329999999999999e-05 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.627070+0000", 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016657600000000001 2024-03-21T14:09:46.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.627210+0000", 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013990899999999999 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.628499+0000", 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012888229999999999 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.630638+0000", 2024-03-21T14:09:46.552 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021389349999999998 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.724800+0000", 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.094162207999999997 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.724854+0000", 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3647999999999999e-05 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:25.724897+0000", 2024-03-21T14:09:46.553 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3554999999999999e-05 2024-03-21T14:09:46.554 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.554 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.554 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.554 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:46.554 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:46.554 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:09:46.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:46.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:46 smithi118 ceph-mon[33862]: pgmap v290: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:09:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:46 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4164490203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:46.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:46.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:09:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:09:46.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-mon[29576]: pgmap v290: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:09:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4164490203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:46.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:09:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:09:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-mon[40510]: pgmap v290: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:09:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4164490203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:09:46.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:46.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:46.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 66671 66655 122.508 85.125 0.0139679 0.00815737 2024-03-21T14:09:46.898 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:46.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:47.000 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:47.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:47.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:47.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:47.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:47.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:47.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:47.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.318 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:47.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:47.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:47.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:47.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:47.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:47.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:47.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:47.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:09:47.752 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:47.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:47.788 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:47 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:09:47.788 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:47 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:09:47.788 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:47.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:47.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 68016 68000 121.409 84.0625 0.00429236 0.00823147 2024-03-21T14:09:47.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:47.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:47.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:47 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:09:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:47 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:09:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:48.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:47 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:09:48.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:47 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:09:48.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:48.126 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:48.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:48.175 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:48.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:48.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:48.315 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:48.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:48.452 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:48.554 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:48.578 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:48.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:48.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:48.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:48.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:48.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:48.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:48.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 69236 69220 120.155 76.25 0.00471551 0.00830773 2024-03-21T14:09:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:48 smithi118 ceph-mon[33862]: pgmap v291: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T14:09:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:48 smithi118 ceph-mon[33862]: osdmap e170: 8 total, 8 up, 8 in 2024-03-21T14:09:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:48 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:09:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:48 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:09:48.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:48.999 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[29576]: pgmap v291: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[29576]: osdmap e170: 8 total, 8 up, 8 in 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[40510]: pgmap v291: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[40510]: osdmap e170: 8 total, 8 up, 8 in 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:09:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:48 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:09:49.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:09:49.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:49.133 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:49.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:49.266 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:49.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:49.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:49.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:49.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:49.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:49.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:49.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:49.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:49.727 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:49 smithi118 ceph-mon[33862]: osdmap e171: 8 total, 8 up, 8 in 2024-03-21T14:09:49.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:49.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:49 smithi003 ceph-mon[40510]: osdmap e171: 8 total, 8 up, 8 in 2024-03-21T14:09:49.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:49.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:49.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:49 smithi003 ceph-mon[29576]: osdmap e171: 8 total, 8 up, 8 in 2024-03-21T14:09:49.834 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:49.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 70428 70412 118.92 74.5 0.00907476 0.00840237 2024-03-21T14:09:49.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:49.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:49.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:50.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:50.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:50.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:50.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:50.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:50.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:50.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:50.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.518 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:50.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:50.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.653 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:50.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:50.754 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:50.789 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:50.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:50.883 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:50 smithi118 ceph-mon[33862]: pgmap v294: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 649 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-21T14:09:50.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 71775 71759 118.006 84.1875 0.00686271 0.00846882 2024-03-21T14:09:50.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:50.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:50.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:50 smithi003 ceph-mon[29576]: pgmap v294: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 649 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-21T14:09:51.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:50 smithi003 ceph-mon[40510]: pgmap v294: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 649 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-21T14:09:51.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:51.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:51.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:51.219 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:51.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:51.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:51.392 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:09:51.392 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:09:51.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.392 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:72442 8.9 e172/162 8:9b4aeb11:::benchmark_data_smithi003_110242_object72441:head v 172'4551, mlcod=172'4548)", 2024-03-21T14:09:51.392 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:51.384560+0000", 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0069620200000000002, 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0069914010000000004, 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384560+0000", 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384560+0000", 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384560+0000", 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8700000000000005e-07 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384619+0000", 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8879999999999999e-05 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384633+0000", 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3611000000000001e-05 2024-03-21T14:09:51.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384640+0000", 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.52e-06 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384859+0000", 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021910500000000001 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:51.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.384879+0000", 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9398e-05 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:72448 8.9 e172/162 8:9fc4626a:::benchmark_data_smithi003_110242_object72447:head v 172'4552, mlcod=172'4548)", 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:51.387684+0000", 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0038379170000000002, 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039329500000000002, 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:51.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.387684+0000", 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.387684+0000", 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.387684+0000", 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1099999999999995e-07 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:51.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.387709+0000", 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4420999999999999e-05 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.387719+0000", 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.747e-06 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.387726+0000", 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.498e-06 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389302+0000", 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015758689999999999 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389322+0000", 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9956999999999999e-05 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:72454 8.8 8:17cc4e28:::benchmark_data_smithi003_110242_object72453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-21T14:09:51.399 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:51.389359+0000", 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0021625849999999999, 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002324221, 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 72454 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389359+0000", 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389359+0000", 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389360+0000", 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.522e-06 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389390+0000", 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9181999999999999e-05 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389398+0000", 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2479999999999996e-06 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389405+0000", 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1269999999999997e-06 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389469+0000", 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3768000000000003e-05 2024-03-21T14:09:51.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389600+0000", 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000130844 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389740+0000", 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140352 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:72455 8.8 8.db427ef8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-21T14:09:51.403 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:51.389429+0000", 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.002092118, 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023244170000000001, 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 72455 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389429+0000", 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389429+0000", 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389430+0000", 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2500000000000005e-07 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389468+0000", 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7824999999999998e-05 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389474+0000", 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6919999999999999e-06 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.389479+0000", 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5679999999999999e-06 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.406 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:72458 8.2 e172/162)", 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:51.390604+0000", 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00091744600000000004, 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012011789999999999, 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.390604+0000", 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:09:51.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.390604+0000", 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.390605+0000", 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0800000000000004e-07 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.390627+0000", 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2665999999999999e-05 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:09:51.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.390640+0000", 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2621e-05 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:51.390647+0000", 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7739999999999999e-06 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 5 2024-03-21T14:09:51.409 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:09:51.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:51.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:51.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:51.656 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:51 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:09:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:51 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:09:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:51 smithi003 ceph-mon[29576]: osdmap e172: 8 total, 8 up, 8 in 2024-03-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:51 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:51 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:09:51.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:51 smithi003 ceph-mon[40510]: osdmap e172: 8 total, 8 up, 8 in 2024-03-21T14:09:51.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:51.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:51.886 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 73136 73120 117.161 85.0625 0.0094754 0.00853035 2024-03-21T14:09:51.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:51.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:51.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:51.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:51 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:09:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:51 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:09:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:51 smithi118 ceph-mon[33862]: osdmap e172: 8 total, 8 up, 8 in 2024-03-21T14:09:52.001 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:52.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:52.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:52.099 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:09:52.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:52.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:52.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:52.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:52.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:52.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:52.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:52.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:52.506 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:52.592 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:52 smithi003 ceph-mon[29576]: pgmap v296: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T14:09:52.593 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:52 smithi003 ceph-mon[40510]: pgmap v296: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T14:09:52.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:52.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.628 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:52.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:52.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:52.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:52.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:52.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:52.887 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:09:52.888239+0000 min lat: 0.00121156 max lat: 0.128614 avg lat: 0.00860875 2024-03-21T14:09:52.887 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:09:52.887 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 74323 74307 116.085 74.1875 0.0134057 0.00860875 2024-03-21T14:09:52.904 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:52 smithi118 ceph-mon[33862]: pgmap v296: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-21T14:09:52.916 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:52.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:52.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:53.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:53.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:53.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:53.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:53.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:53.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:53.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:53.610 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:53.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:53.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:53.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:53.887 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 75433 75417 114.945 69.375 0.00809769 0.00869465 2024-03-21T14:09:53.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:53.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:53.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:53.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:54.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:54.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:54.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:54.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:54.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:54.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:54.397 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:09:54.398 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:09:54.398 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:09:54.398 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:09:54.398 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:09:54.408 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:54.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:54.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:54.600 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:09:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:09:54.642 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:09:54.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:54.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:09:54.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:54.721 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:09:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:54 smithi003 ceph-mon[29576]: pgmap v297: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-21T14:09:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:54 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3122248625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:54 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4180490870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:54 smithi003 ceph-mon[40510]: pgmap v297: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-21T14:09:54.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:54 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3122248625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:54.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:54 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4180490870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:54.760 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:09:54.799 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:09:54.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:54.837 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:54.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:54.844 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:09:54.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:09:54.887 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 76604 76588 113.951 73.1875 0.0100408 0.00877089 2024-03-21T14:09:54.888 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:09:54.888 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:09:54.888 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:09:54.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:09:54.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:09:54.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:09:54.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:09:54.889 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:09:54.894 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:09:54.894 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:09:54.895 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-21T14:09:54.895 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-21T14:09:54.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:09:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:54 smithi118 ceph-mon[33862]: pgmap v297: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-21T14:09:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:54 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3122248625' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:09:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:54 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4180490870' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:09:54.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:54.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:54.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:55.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:55.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:55.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:55.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:55.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:55.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:55.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:55.543 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:55.645 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:55.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:55.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:55.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3122248625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: osdmap e173: 8 total, 8 up, 8 in 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:09:55.851 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3122248625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: osdmap e173: 8 total, 8 up, 8 in 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:09:55.852 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:55 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:09:55.888 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 77784 77768 113.016 73.75 0.00687206 0.0088438 2024-03-21T14:09:55.900 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:55.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3122248625' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: osdmap e173: 8 total, 8 up, 8 in 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:09:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:55 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:09:55.964 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:55.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:56.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.103 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:56.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:56.170 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:09:56.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:56.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:56.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:56.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:56.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:56.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:56.570 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:56.610 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:09:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:09:56.672 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:56.716 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:56.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:56.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:56.867 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: pgmap v299: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 747 B/s rd, 59 MiB/s wr, 947 op/s 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:09:56.867 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:09:56.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:56 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:09:56.888 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 79092 79076 112.305 81.75 0.0117931 0.00889898 2024-03-21T14:09:56.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:09:56.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:09:56.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: pgmap v299: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 747 B/s rd, 59 MiB/s wr, 947 op/s 2024-03-21T14:09:56.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:09:56.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:09:56.900 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: pgmap v299: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 747 B/s rd, 59 MiB/s wr, 947 op/s 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:09:56.901 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:09:56.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:09:56.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:09:56.902 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:56 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:09:56.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:56.994 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:56.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:57.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:57.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:57.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:57.271 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:57.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:57.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:57.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:57.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:57.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:57.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:57.791 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:57.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:57.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:57.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: osdmap e174: 8 total, 8 up, 8 in 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:09:57.886 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: osdmap e174: 8 total, 8 up, 8 in 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:09:57.887 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:57 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:09:57.888 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 80449 80433 111.692 84.8125 0.010782 0.00894807 2024-03-21T14:09:57.926 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: osdmap e174: 8 total, 8 up, 8 in 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:09:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:57 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:09:57.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:57.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:57.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:09:58.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:09:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:58.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:09:58.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:09:58.238 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:58.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:09:58.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:58.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:09:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:58.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:58.426 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:09:58.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:58.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:58.664 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:58.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:58.753 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:58 smithi003 ceph-mon[29576]: pgmap v301: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-21T14:09:58.753 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:58 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:09:58.753 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:58 smithi003 ceph-mon[29576]: osdmap e175: 8 total, 8 up, 8 in 2024-03-21T14:09:58.753 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:58 smithi003 ceph-mon[40510]: pgmap v301: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-21T14:09:58.753 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:58 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:09:58.753 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:58 smithi003 ceph-mon[40510]: osdmap e175: 8 total, 8 up, 8 in 2024-03-21T14:09:58.770 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:58 smithi118 ceph-mon[33862]: pgmap v301: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.5 KiB/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-21T14:09:58.770 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:58 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:09:58.771 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:58 smithi118 ceph-mon[33862]: osdmap e175: 8 total, 8 up, 8 in 2024-03-21T14:09:58.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:58.807 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:09:58.817 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:09:58.847 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:09:58.847 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:09:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:58.890 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 81510 81494 110.705 66.3125 0.0129675 0.00902802 2024-03-21T14:09:58.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:58.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:58.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:58.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:59.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:09:59.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:09:59.230 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:59.264 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:09:59.366 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:09:59.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:09:59.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:59.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:09:59.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:59.623 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-03-21T14:09:57.629659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5156116904,"num_objects":78865,"num_object_clones":0,"num_object_copies":236595,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78865,"num_whiteouts":0,"num_read":1367,"num_read_kb":1213,"num_write":79282,"num_write_kb":5037149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":2363178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77026,"ondisk_log_size":77026,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16629780,"kb_used_data":14967112,"kb_used_omap":12,"kb_used_meta":1662579,"kb_avail":733331436,"statfs":{"total":767960285184,"available":750931390464,"internally_reserved":0,"allocated":15326322688,"data_stored":15322997333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1702481456},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,5,1,1,3,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":630390784,"num_objects":9619,"num_object_clones":0,"num_object_copies":28857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9619,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9619,"num_write_kb":615616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.258188"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":240,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:55.330055+0000","last_change":"2024-03-21T14:09:55.330094+0000","last_active":"2024-03-21T14:09:55.330055+0000","last_peered":"2024-03-21T14:09:55.330055+0000","last_clean":"2024-03-21T14:09:55.330055+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:09:55.330055+0000","last_undegraded":"2024-03-21T14:09:55.330055+0000","last_fullsized":"2024-03-21T14:09:55.330055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.329965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.329965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.949341+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":255,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611740+0000","last_change":"2024-03-21T14:09:02.129524+0000","last_active":"2024-03-21T14:09:56.611740+0000","last_peered":"2024-03-21T14:09:56.611740+0000","last_clean":"2024-03-21T14:09:56.611740+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:09:56.611740+0000","last_undegraded":"2024-03-21T14:09:56.611740+0000","last_fullsized":"2024-03-21T14:09:56.611740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:02.129441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.129441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.443888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":106,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598698+0000","last_change":"2024-03-21T14:08:26.447065+0000","last_active":"2024-03-21T14:09:54.598698+0000","last_peered":"2024-03-21T14:09:54.598698+0000","last_clean":"2024-03-21T14:09:54.598698+0000","last_became_active":"2024-03-21T14:07:23.349536+0000","last_became_peered":"2024-03-21T14:07:23.349536+0000","last_unstale":"2024-03-21T14:09:54.598698+0000","last_undegraded":"2024-03-21T14:09:54.598698+0000","last_fullsized":"2024-03-21T14:09:54.598698+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:26.446938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.446938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":265,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602382+0000","last_change":"2024-03-21T14:09:15.098201+0000","last_active":"2024-03-21T14:09:54.602382+0000","last_peered":"2024-03-21T14:09:54.602382+0000","last_clean":"2024-03-21T14:09:54.602382+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:09:54.602382+0000","last_undegraded":"2024-03-21T14:09:54.602382+0000","last_fullsized":"2024-03-21T14:09:54.602382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:15.098048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.098048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:43.207181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":107,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600786+0000","last_change":"2024-03-21T14:07:25.072258+0000","last_active":"2024-03-21T14:09:54.600786+0000","last_peered":"2024-03-21T14:09:54.600786+0000","last_clean":"2024-03-21T14:09:54.600786+0000","last_became_active":"2024-03-21T14:07:23.355776+0000","last_became_peered":"2024-03-21T14:07:23.355776+0000","last_unstale":"2024-03-21T14:09:54.600786+0000","last_undegraded":"2024-03-21T14:09:54.600786+0000","last_fullsized":"2024-03-21T14:09:54.600786+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:07:25.072003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:07:25.072003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602429+0000","last_change":"2024-03-21T14:08:23.927313+0000","last_active":"2024-03-21T14:09:54.602429+0000","last_peered":"2024-03-21T14:09:54.602429+0000","last_clean":"2024-03-21T14:09:54.602429+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:09:54.602429+0000","last_undegraded":"2024-03-21T14:09:54.602429+0000","last_fullsized":"2024-03-21T14:09:54.602429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.927116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:32.256757+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.927116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.619862+0000","last_change":"2024-03-21T14:08:31.020727+0000","last_active":"2024-03-21T14:09:56.619862+0000","last_peered":"2024-03-21T14:09:56.619862+0000","last_clean":"2024-03-21T14:09:56.619862+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:09:56.619862+0000","last_undegraded":"2024-03-21T14:09:56.619862+0000","last_fullsized":"2024-03-21T14:09:56.619862+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:31.020641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.020641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":284,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620226+0000","last_change":"2024-03-21T14:08:16.858074+0000","last_active":"2024-03-21T14:09:56.620226+0000","last_peered":"2024-03-21T14:09:56.620226+0000","last_clean":"2024-03-21T14:09:56.620226+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:09:56.620226+0000","last_undegraded":"2024-03-21T14:09:56.620226+0000","last_fullsized":"2024-03-21T14:09:56.620226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:08:16.857912+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.857912+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":108,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.614503+0000","last_change":"2024-03-21T14:08:34.165641+0000","last_active":"2024-03-21T14:09:56.614503+0000","last_peered":"2024-03-21T14:09:56.614503+0000","last_clean":"2024-03-21T14:09:56.614503+0000","last_became_active":"2024-03-21T14:07:23.360834+0000","last_became_peered":"2024-03-21T14:07:23.360834+0000","last_unstale":"2024-03-21T14:09:56.614503+0000","last_undegraded":"2024-03-21T14:09:56.614503+0000","last_fullsized":"2024-03-21T14:09:56.614503+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:34.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:34.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":244,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600091+0000","last_change":"2024-03-21T14:09:10.043497+0000","last_active":"2024-03-21T14:09:54.600091+0000","last_peered":"2024-03-21T14:09:54.600091+0000","last_clean":"2024-03-21T14:09:54.600091+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:09:54.600091+0000","last_undegraded":"2024-03-21T14:09:54.600091+0000","last_fullsized":"2024-03-21T14:09:54.600091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.043396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.043396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.977610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":252,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598620+0000","last_change":"2024-03-21T14:08:23.436895+0000","last_active":"2024-03-21T14:09:54.598620+0000","last_peered":"2024-03-21T14:09:54.598620+0000","last_clean":"2024-03-21T14:09:54.598620+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:09:54.598620+0000","last_undegraded":"2024-03-21T14:09:54.598620+0000","last_fullsized":"2024-03-21T14:09:54.598620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.436768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.436768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":256,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602678+0000","last_change":"2024-03-21T14:08:12.828629+0000","last_active":"2024-03-21T14:09:54.602678+0000","last_peered":"2024-03-21T14:09:54.602678+0000","last_clean":"2024-03-21T14:09:54.602678+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:09:54.602678+0000","last_undegraded":"2024-03-21T14:09:54.602678+0000","last_fullsized":"2024-03-21T14:09:54.602678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:08:12.828494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.828494+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226185+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":246,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598078+0000","last_change":"2024-03-21T14:08:10.366349+0000","last_active":"2024-03-21T14:09:54.598078+0000","last_peered":"2024-03-21T14:09:54.598078+0000","last_clean":"2024-03-21T14:09:54.598078+0000","last_became_active":"2024-03-21T14:07:23.350761+0000","last_became_peered":"2024-03-21T14:07:23.350761+0000","last_unstale":"2024-03-21T14:09:54.598078+0000","last_undegraded":"2024-03-21T14:09:54.598078+0000","last_fullsized":"2024-03-21T14:09:54.598078+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.366216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.366216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600628+0000","last_change":"2024-03-21T14:09:25.498525+0000","last_active":"2024-03-21T14:09:54.600628+0000","last_peered":"2024-03-21T14:09:54.600628+0000","last_clean":"2024-03-21T14:09:54.600628+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:09:54.600628+0000","last_undegraded":"2024-03-21T14:09:54.600628+0000","last_fullsized":"2024-03-21T14:09:54.600628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.498388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.498388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:43.494502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":252,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598698+0000","last_change":"2024-03-21T14:07:13.402391+0000","last_active":"2024-03-21T14:09:54.598698+0000","last_peered":"2024-03-21T14:09:54.598698+0000","last_clean":"2024-03-21T14:09:54.598698+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:09:54.598698+0000","last_undegraded":"2024-03-21T14:09:54.598698+0000","last_fullsized":"2024-03-21T14:09:54.598698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:07:13.402269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:07:13.402269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":258,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603017+0000","last_change":"2024-03-21T14:09:26.168563+0000","last_active":"2024-03-21T14:09:54.603017+0000","last_peered":"2024-03-21T14:09:54.603017+0000","last_clean":"2024-03-21T14:09:54.603017+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:09:54.603017+0000","last_undegraded":"2024-03-21T14:09:54.603017+0000","last_fullsized":"2024-03-21T14:09:54.603017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:09:26.168400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.168400+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.775561+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":233,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603388+0000","last_change":"2024-03-21T14:08:18.854622+0000","last_active":"2024-03-21T14:09:54.603388+0000","last_peered":"2024-03-21T14:09:54.603388+0000","last_clean":"2024-03-21T14:09:54.603388+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:09:54.603388+0000","last_undegraded":"2024-03-21T14:09:54.603388+0000","last_fullsized":"2024-03-21T14:09:54.603388+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:18.854505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:18.854505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":246,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611046+0000","last_change":"2024-03-21T14:08:18.152818+0000","last_active":"2024-03-21T14:09:56.611046+0000","last_peered":"2024-03-21T14:09:56.611046+0000","last_clean":"2024-03-21T14:09:56.611046+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:09:56.611046+0000","last_undegraded":"2024-03-21T14:09:56.611046+0000","last_fullsized":"2024-03-21T14:09:56.611046+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:18.152711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:18.152711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755443+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":106,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602962+0000","last_change":"2024-03-21T14:09:19.179886+0000","last_active":"2024-03-21T14:09:54.602962+0000","last_peered":"2024-03-21T14:09:54.602962+0000","last_clean":"2024-03-21T14:09:54.602962+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:09:54.602962+0000","last_undegraded":"2024-03-21T14:09:54.602962+0000","last_fullsized":"2024-03-21T14:09:54.602962+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.179692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.179692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:38.891081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":292,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.605410+0000","last_change":"2024-03-21T14:09:26.954314+0000","last_active":"2024-03-21T14:09:56.605410+0000","last_peered":"2024-03-21T14:09:56.605410+0000","last_clean":"2024-03-21T14:09:56.605410+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:09:56.605410+0000","last_undegraded":"2024-03-21T14:09:56.605410+0000","last_fullsized":"2024-03-21T14:09:56.605410+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:09:26.954184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.954184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.810324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.622284+0000","last_change":"2024-03-21T14:09:04.145571+0000","last_active":"2024-03-21T14:09:56.622284+0000","last_peered":"2024-03-21T14:09:56.622284+0000","last_clean":"2024-03-21T14:09:56.622284+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:09:56.622284+0000","last_undegraded":"2024-03-21T14:09:56.622284+0000","last_fullsized":"2024-03-21T14:09:56.622284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.145471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.145471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.785029+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":236,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603107+0000","last_change":"2024-03-21T14:08:10.874464+0000","last_active":"2024-03-21T14:09:54.603107+0000","last_peered":"2024-03-21T14:09:54.603107+0000","last_clean":"2024-03-21T14:09:54.603107+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:09:54.603107+0000","last_undegraded":"2024-03-21T14:09:54.603107+0000","last_fullsized":"2024-03-21T14:09:54.603107+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.874240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.874240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226125+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":252,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602791+0000","last_change":"2024-03-21T14:08:15.817968+0000","last_active":"2024-03-21T14:09:54.602791+0000","last_peered":"2024-03-21T14:09:54.602791+0000","last_clean":"2024-03-21T14:09:54.602791+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:09:54.602791+0000","last_undegraded":"2024-03-21T14:09:54.602791+0000","last_fullsized":"2024-03-21T14:09:54.602791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:15.817858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.817858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":274,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620909+0000","last_change":"2024-03-21T14:09:27.347989+0000","last_active":"2024-03-21T14:09:56.620909+0000","last_peered":"2024-03-21T14:09:56.620909+0000","last_clean":"2024-03-21T14:09:56.620909+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:09:56.620909+0000","last_undegraded":"2024-03-21T14:09:56.620909+0000","last_fullsized":"2024-03-21T14:09:56.620909+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:27.347835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:27.347835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.996152+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":251,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600981+0000","last_change":"2024-03-21T14:09:24.494651+0000","last_active":"2024-03-21T14:09:54.600981+0000","last_peered":"2024-03-21T14:09:54.600981+0000","last_clean":"2024-03-21T14:09:54.600981+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:09:54.600981+0000","last_undegraded":"2024-03-21T14:09:54.600981+0000","last_fullsized":"2024-03-21T14:09:54.600981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.494516+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:32.193912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":348,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.592755+0000","last_change":"2024-03-21T14:08:16.957033+0000","last_active":"2024-03-21T14:09:54.592755+0000","last_peered":"2024-03-21T14:09:54.592755+0000","last_clean":"2024-03-21T14:09:54.592755+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:09:54.592755+0000","last_undegraded":"2024-03-21T14:09:54.592755+0000","last_fullsized":"2024-03-21T14:09:54.592755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:08:16.956896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.956896+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":272,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611529+0000","last_change":"2024-03-21T14:09:09.213981+0000","last_active":"2024-03-21T14:09:56.611529+0000","last_peered":"2024-03-21T14:09:56.611529+0000","last_clean":"2024-03-21T14:09:56.611529+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:09:56.611529+0000","last_undegraded":"2024-03-21T14:09:56.611529+0000","last_fullsized":"2024-03-21T14:09:56.611529+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:09:09.213878+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.213878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.594067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":311,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.619073+0000","last_change":"2024-03-21T14:09:02.781223+0000","last_active":"2024-03-21T14:09:56.619073+0000","last_peered":"2024-03-21T14:09:56.619073+0000","last_clean":"2024-03-21T14:09:56.619073+0000","last_became_active":"2024-03-21T14:08:17.144672+0000","last_became_peered":"2024-03-21T14:08:17.144672+0000","last_unstale":"2024-03-21T14:09:56.619073+0000","last_undegraded":"2024-03-21T14:09:56.619073+0000","last_fullsized":"2024-03-21T14:09:56.619073+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:09:02.781039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.781039+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:13.917935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":242,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.610791+0000","last_change":"2024-03-21T14:08:09.054661+0000","last_active":"2024-03-21T14:09:56.610791+0000","last_peered":"2024-03-21T14:09:56.610791+0000","last_clean":"2024-03-21T14:09:56.610791+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:09:56.610791+0000","last_undegraded":"2024-03-21T14:09:56.610791+0000","last_fullsized":"2024-03-21T14:09:56.610791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:09.054580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:09.054580+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":248,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.619330+0000","last_change":"2024-03-21T14:08:20.617200+0000","last_active":"2024-03-21T14:09:56.619330+0000","last_peered":"2024-03-21T14:09:56.619330+0000","last_clean":"2024-03-21T14:09:56.619330+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:09:56.619330+0000","last_undegraded":"2024-03-21T14:09:56.619330+0000","last_fullsized":"2024-03-21T14:09:56.619330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:08:20.617078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":262,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.625171+0000","last_change":"2024-03-21T14:08:24.891285+0000","last_active":"2024-03-21T14:09:56.625171+0000","last_peered":"2024-03-21T14:09:56.625171+0000","last_clean":"2024-03-21T14:09:56.625171+0000","last_became_active":"2024-03-21T14:07:23.364680+0000","last_became_peered":"2024-03-21T14:07:23.364680+0000","last_unstale":"2024-03-21T14:09:56.625171+0000","last_undegraded":"2024-03-21T14:09:56.625171+0000","last_fullsized":"2024-03-21T14:09:56.625171+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:24.891079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:08:24.891079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"127'16","reported_seq":281,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.592657+0000","last_change":"2024-03-21T14:09:05.048112+0000","last_active":"2024-03-21T14:09:54.592657+0000","last_peered":"2024-03-21T14:09:54.592657+0000","last_clean":"2024-03-21T14:09:54.592657+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:09:54.592657+0000","last_undegraded":"2024-03-21T14:09:54.592657+0000","last_fullsized":"2024-03-21T14:09:54.592657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'16","last_scrub_stamp":"2024-03-21T14:09:05.047983+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.047983+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.538539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600509+0000","last_change":"2024-03-21T14:08:11.713805+0000","last_active":"2024-03-21T14:09:54.600509+0000","last_peered":"2024-03-21T14:09:54.600509+0000","last_clean":"2024-03-21T14:09:54.600509+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:09:54.600509+0000","last_undegraded":"2024-03-21T14:09:54.600509+0000","last_fullsized":"2024-03-21T14:09:54.600509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:11.713692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:11.713692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.591245+0000","last_change":"2024-03-21T14:08:18.005058+0000","last_active":"2024-03-21T14:09:54.591245+0000","last_peered":"2024-03-21T14:09:54.591245+0000","last_clean":"2024-03-21T14:09:54.591245+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:09:54.591245+0000","last_undegraded":"2024-03-21T14:09:54.591245+0000","last_fullsized":"2024-03-21T14:09:54.591245+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:18.004981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:18.004981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.618950+0000","last_change":"2024-03-21T14:08:33.573417+0000","last_active":"2024-03-21T14:09:56.618950+0000","last_peered":"2024-03-21T14:09:56.618950+0000","last_clean":"2024-03-21T14:09:56.618950+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:09:56.618950+0000","last_undegraded":"2024-03-21T14:09:56.618950+0000","last_fullsized":"2024-03-21T14:09:56.618950+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:33.573334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.573334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.673670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.625267+0000","last_change":"2024-03-21T14:08:33.005924+0000","last_active":"2024-03-21T14:09:56.625267+0000","last_peered":"2024-03-21T14:09:56.625267+0000","last_clean":"2024-03-21T14:09:56.625267+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:09:56.625267+0000","last_undegraded":"2024-03-21T14:09:56.625267+0000","last_fullsized":"2024-03-21T14:09:56.625267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:08:33.005817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.005817+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620699+0000","last_change":"2024-03-21T14:09:08.089755+0000","last_active":"2024-03-21T14:09:56.620699+0000","last_peered":"2024-03-21T14:09:56.620699+0000","last_clean":"2024-03-21T14:09:56.620699+0000","last_became_active":"2024-03-21T14:08:17.194651+0000","last_became_peered":"2024-03-21T14:08:17.194651+0000","last_unstale":"2024-03-21T14:09:56.620699+0000","last_undegraded":"2024-03-21T14:09:56.620699+0000","last_fullsized":"2024-03-21T14:09:56.620699+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:08.089654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:08.089654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:11.989970+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":254,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600729+0000","last_change":"2024-03-21T14:09:09.475589+0000","last_active":"2024-03-21T14:09:54.600729+0000","last_peered":"2024-03-21T14:09:54.600729+0000","last_clean":"2024-03-21T14:09:54.600729+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:09:54.600729+0000","last_undegraded":"2024-03-21T14:09:54.600729+0000","last_fullsized":"2024-03-21T14:09:54.600729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.475453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.475453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.336499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":267,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:55.427665+0000","last_change":"2024-03-21T14:09:55.427714+0000","last_active":"2024-03-21T14:09:55.427665+0000","last_peered":"2024-03-21T14:09:55.427665+0000","last_clean":"2024-03-21T14:09:55.427665+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:09:55.427665+0000","last_undegraded":"2024-03-21T14:09:55.427665+0000","last_fullsized":"2024-03-21T14:09:55.427665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:55.427563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.427563+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.673618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":279,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620091+0000","last_change":"2024-03-21T14:09:56.592384+0000","last_active":"2024-03-21T14:09:56.620091+0000","last_peered":"2024-03-21T14:09:56.620091+0000","last_clean":"2024-03-21T14:09:56.620091+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:09:56.620091+0000","last_undegraded":"2024-03-21T14:09:56.620091+0000","last_fullsized":"2024-03-21T14:09:56.620091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:56.592242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.592242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.889308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.617777+0000","last_change":"2024-03-21T14:09:12.819080+0000","last_active":"2024-03-21T14:09:56.617777+0000","last_peered":"2024-03-21T14:09:56.617777+0000","last_clean":"2024-03-21T14:09:56.617777+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:09:56.617777+0000","last_undegraded":"2024-03-21T14:09:56.617777+0000","last_fullsized":"2024-03-21T14:09:56.617777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:12.818982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:12.818982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.361591+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":248,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620621+0000","last_change":"2024-03-21T14:09:10.079022+0000","last_active":"2024-03-21T14:09:56.620621+0000","last_peered":"2024-03-21T14:09:56.620621+0000","last_clean":"2024-03-21T14:09:56.620621+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:09:56.620621+0000","last_undegraded":"2024-03-21T14:09:56.620621+0000","last_fullsized":"2024-03-21T14:09:56.620621+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.078939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.078939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.610355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":254,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600998+0000","last_change":"2024-03-21T14:08:14.668920+0000","last_active":"2024-03-21T14:09:54.600998+0000","last_peered":"2024-03-21T14:09:54.600998+0000","last_clean":"2024-03-21T14:09:54.600998+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:09:54.600998+0000","last_undegraded":"2024-03-21T14:09:54.600998+0000","last_fullsized":"2024-03-21T14:09:54.600998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:14.668685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:14.668685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":275,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.613867+0000","last_change":"2024-03-21T14:09:03.162696+0000","last_active":"2024-03-21T14:09:56.613867+0000","last_peered":"2024-03-21T14:09:56.613867+0000","last_clean":"2024-03-21T14:09:56.613867+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:09:56.613867+0000","last_undegraded":"2024-03-21T14:09:56.613867+0000","last_fullsized":"2024-03-21T14:09:56.613867+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:03.162584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.162584+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:11.004039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":109,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.617995+0000","last_change":"2024-03-21T14:08:22.602619+0000","last_active":"2024-03-21T14:09:56.617995+0000","last_peered":"2024-03-21T14:09:56.617995+0000","last_clean":"2024-03-21T14:09:56.617995+0000","last_became_active":"2024-03-21T14:07:23.360741+0000","last_became_peered":"2024-03-21T14:07:23.360741+0000","last_unstale":"2024-03-21T14:09:56.617995+0000","last_undegraded":"2024-03-21T14:09:56.617995+0000","last_fullsized":"2024-03-21T14:09:56.617995+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:22.602499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:22.602499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":261,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.613562+0000","last_change":"2024-03-21T14:09:26.436609+0000","last_active":"2024-03-21T14:09:56.613562+0000","last_peered":"2024-03-21T14:09:56.613562+0000","last_clean":"2024-03-21T14:09:56.613562+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:09:56.613562+0000","last_undegraded":"2024-03-21T14:09:56.613562+0000","last_fullsized":"2024-03-21T14:09:56.613562+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:26.436326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.436326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:26.638026+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":261,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602859+0000","last_change":"2024-03-21T14:09:23.112108+0000","last_active":"2024-03-21T14:09:54.602859+0000","last_peered":"2024-03-21T14:09:54.602859+0000","last_clean":"2024-03-21T14:09:54.602859+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:09:54.602859+0000","last_undegraded":"2024-03-21T14:09:54.602859+0000","last_fullsized":"2024-03-21T14:09:54.602859+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.111999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.111999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:42.481221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"153'16","reported_seq":291,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603467+0000","last_change":"2024-03-21T14:08:30.872145+0000","last_active":"2024-03-21T14:09:54.603467+0000","last_peered":"2024-03-21T14:09:54.603467+0000","last_clean":"2024-03-21T14:09:54.603467+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:09:54.603467+0000","last_undegraded":"2024-03-21T14:09:54.603467+0000","last_fullsized":"2024-03-21T14:09:54.603467+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:08:30.871984+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226246+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":107,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601283+0000","last_change":"2024-03-21T14:09:10.489473+0000","last_active":"2024-03-21T14:09:54.601283+0000","last_peered":"2024-03-21T14:09:54.601283+0000","last_clean":"2024-03-21T14:09:54.601283+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:09:54.601283+0000","last_undegraded":"2024-03-21T14:09:54.601283+0000","last_fullsized":"2024-03-21T14:09:54.601283+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.489356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.489356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.103897+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":350,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598694+0000","last_change":"2024-03-21T14:09:15.064168+0000","last_active":"2024-03-21T14:09:54.598694+0000","last_peered":"2024-03-21T14:09:54.598694+0000","last_clean":"2024-03-21T14:09:54.598694+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:09:54.598694+0000","last_undegraded":"2024-03-21T14:09:54.598694+0000","last_fullsized":"2024-03-21T14:09:54.598694+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:15.063993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.063993+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.135176+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":260,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600691+0000","last_change":"2024-03-21T14:08:17.142960+0000","last_active":"2024-03-21T14:09:54.600691+0000","last_peered":"2024-03-21T14:09:54.600691+0000","last_clean":"2024-03-21T14:09:54.600691+0000","last_became_active":"2024-03-21T14:08:17.142668+0000","last_became_peered":"2024-03-21T14:08:17.142668+0000","last_unstale":"2024-03-21T14:09:54.600691+0000","last_undegraded":"2024-03-21T14:09:54.600691+0000","last_fullsized":"2024-03-21T14:09:54.600691+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:12.703435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.703435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.624238+0000","last_change":"2024-03-21T14:08:15.879476+0000","last_active":"2024-03-21T14:09:56.624238+0000","last_peered":"2024-03-21T14:09:56.624238+0000","last_clean":"2024-03-21T14:09:56.624238+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:09:56.624238+0000","last_undegraded":"2024-03-21T14:09:56.624238+0000","last_fullsized":"2024-03-21T14:09:56.624238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:08:15.879333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.879333+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":241,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603117+0000","last_change":"2024-03-21T14:09:17.192133+0000","last_active":"2024-03-21T14:09:54.603117+0000","last_peered":"2024-03-21T14:09:54.603117+0000","last_clean":"2024-03-21T14:09:54.603117+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:09:54.603117+0000","last_undegraded":"2024-03-21T14:09:54.603117+0000","last_fullsized":"2024-03-21T14:09:54.603117+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:17.192018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.192018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.830635+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"130'9","reported_seq":167,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600935+0000","last_change":"2024-03-21T14:08:13.697250+0000","last_active":"2024-03-21T14:09:54.600935+0000","last_peered":"2024-03-21T14:09:54.600935+0000","last_clean":"2024-03-21T14:09:54.600935+0000","last_became_active":"2024-03-21T14:07:23.354945+0000","last_became_peered":"2024-03-21T14:07:23.354945+0000","last_unstale":"2024-03-21T14:09:54.600935+0000","last_undegraded":"2024-03-21T14:09:54.600935+0000","last_fullsized":"2024-03-21T14:09:54.600935+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:08:13.697139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:13.697139+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":270,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.618211+0000","last_change":"2024-03-21T14:09:21.855689+0000","last_active":"2024-03-21T14:09:56.618211+0000","last_peered":"2024-03-21T14:09:56.618211+0000","last_clean":"2024-03-21T14:09:56.618211+0000","last_became_active":"2024-03-21T14:07:23.358188+0000","last_became_peered":"2024-03-21T14:07:23.358188+0000","last_unstale":"2024-03-21T14:09:56.618211+0000","last_undegraded":"2024-03-21T14:09:56.618211+0000","last_fullsized":"2024-03-21T14:09:56.618211+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:21.852936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:21.852936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.459798+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":265,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598870+0000","last_change":"2024-03-21T14:09:11.022485+0000","last_active":"2024-03-21T14:09:54.598870+0000","last_peered":"2024-03-21T14:09:54.598870+0000","last_clean":"2024-03-21T14:09:54.598870+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:09:54.598870+0000","last_undegraded":"2024-03-21T14:09:54.598870+0000","last_fullsized":"2024-03-21T14:09:54.598870+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:11.022384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.022384+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.993211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.605756+0000","last_change":"2024-03-21T14:09:14.024358+0000","last_active":"2024-03-21T14:09:56.605756+0000","last_peered":"2024-03-21T14:09:56.605756+0000","last_clean":"2024-03-21T14:09:56.605756+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:09:56.605756+0000","last_undegraded":"2024-03-21T14:09:56.605756+0000","last_fullsized":"2024-03-21T14:09:56.605756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:14.024237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:14.024237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.900582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":108,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.615957+0000","last_change":"2024-03-21T14:09:11.232306+0000","last_active":"2024-03-21T14:09:56.615957+0000","last_peered":"2024-03-21T14:09:56.615957+0000","last_clean":"2024-03-21T14:09:56.615957+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:09:56.615957+0000","last_undegraded":"2024-03-21T14:09:56.615957+0000","last_fullsized":"2024-03-21T14:09:56.615957+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.232197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.232197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:40.426883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":262,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:55.186732+0000","last_change":"2024-03-21T14:09:55.186771+0000","last_active":"2024-03-21T14:09:55.186732+0000","last_peered":"2024-03-21T14:09:55.186732+0000","last_clean":"2024-03-21T14:09:55.186732+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:09:55.186732+0000","last_undegraded":"2024-03-21T14:09:55.186732+0000","last_fullsized":"2024-03-21T14:09:55.186732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.186641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.186641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:55.374836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.619173+0000","last_change":"2024-03-21T14:09:05.784892+0000","last_active":"2024-03-21T14:09:56.619173+0000","last_peered":"2024-03-21T14:09:56.619173+0000","last_clean":"2024-03-21T14:09:56.619173+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:09:56.619173+0000","last_undegraded":"2024-03-21T14:09:56.619173+0000","last_fullsized":"2024-03-21T14:09:56.619173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:05.784714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.784714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:33.485395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601049+0000","last_change":"2024-03-21T14:08:10.733076+0000","last_active":"2024-03-21T14:09:54.601049+0000","last_peered":"2024-03-21T14:09:54.601049+0000","last_clean":"2024-03-21T14:09:54.601049+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:09:54.601049+0000","last_undegraded":"2024-03-21T14:09:54.601049+0000","last_fullsized":"2024-03-21T14:09:54.601049+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.732971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.732971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.251270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598462+0000","last_change":"2024-03-21T14:08:13.407387+0000","last_active":"2024-03-21T14:09:54.598462+0000","last_peered":"2024-03-21T14:09:54.598462+0000","last_clean":"2024-03-21T14:09:54.598462+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:09:54.598462+0000","last_undegraded":"2024-03-21T14:09:54.598462+0000","last_fullsized":"2024-03-21T14:09:54.598462+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:13.407278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:13.407278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":267,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.590583+0000","last_change":"2024-03-21T14:09:04.020412+0000","last_active":"2024-03-21T14:09:54.590583+0000","last_peered":"2024-03-21T14:09:54.590583+0000","last_clean":"2024-03-21T14:09:54.590583+0000","last_became_active":"2024-03-21T14:07:23.354457+0000","last_became_peered":"2024-03-21T14:07:23.354457+0000","last_unstale":"2024-03-21T14:09:54.590583+0000","last_undegraded":"2024-03-21T14:09:54.590583+0000","last_fullsized":"2024-03-21T14:09:54.590583+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.020323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.020323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.587036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":143,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601327+0000","last_change":"2024-03-21T14:09:15.585161+0000","last_active":"2024-03-21T14:09:54.601327+0000","last_peered":"2024-03-21T14:09:54.601327+0000","last_clean":"2024-03-21T14:09:54.601327+0000","last_became_active":"2024-03-21T14:07:23.365278+0000","last_became_peered":"2024-03-21T14:07:23.365278+0000","last_unstale":"2024-03-21T14:09:54.601327+0000","last_undegraded":"2024-03-21T14:09:54.601327+0000","last_fullsized":"2024-03-21T14:09:54.601327+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:09:15.584995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.584995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.826538+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"173'4907","reported_seq":4939,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.317320+0000","last_change":"2024-03-21T14:09:01.659122+0000","last_active":"2024-03-21T14:09:56.317320+0000","last_peered":"2024-03-21T14:09:56.317320+0000","last_clean":"2024-03-21T14:09:56.317320+0000","last_became_active":"2024-03-21T14:09:01.658623+0000","last_became_peered":"2024-03-21T14:09:01.658623+0000","last_unstale":"2024-03-21T14:09:56.317320+0000","last_undegraded":"2024-03-21T14:09:56.317320+0000","last_fullsized":"2024-03-21T14:09:56.317320+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4907,"log_dups_size":0,"ondisk_log_size":4907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:03.668632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321585152,"num_objects":4907,"num_object_clones":0,"num_object_copies":14721,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4907,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4907,"num_write_kb":314048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":241,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598417+0000","last_change":"2024-03-21T14:08:32.402570+0000","last_active":"2024-03-21T14:09:54.598417+0000","last_peered":"2024-03-21T14:09:54.598417+0000","last_clean":"2024-03-21T14:09:54.598417+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:09:54.598417+0000","last_undegraded":"2024-03-21T14:09:54.598417+0000","last_fullsized":"2024-03-21T14:09:54.598417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:32.402481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.402481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:58.330250+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":278,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601286+0000","last_change":"2024-03-21T14:08:24.629488+0000","last_active":"2024-03-21T14:09:54.601286+0000","last_peered":"2024-03-21T14:09:54.601286+0000","last_clean":"2024-03-21T14:09:54.601286+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:09:54.601286+0000","last_undegraded":"2024-03-21T14:09:54.601286+0000","last_fullsized":"2024-03-21T14:09:54.601286+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:08:24.629266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:24.629266+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":274,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.599597+0000","last_change":"2024-03-21T14:08:15.390480+0000","last_active":"2024-03-21T14:09:54.599597+0000","last_peered":"2024-03-21T14:09:54.599597+0000","last_clean":"2024-03-21T14:09:54.599597+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:09:54.599597+0000","last_undegraded":"2024-03-21T14:09:54.599597+0000","last_fullsized":"2024-03-21T14:09:54.599597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:08:15.390326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.390326+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":343,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600642+0000","last_change":"2024-03-21T14:09:06.480162+0000","last_active":"2024-03-21T14:09:54.600642+0000","last_peered":"2024-03-21T14:09:54.600642+0000","last_clean":"2024-03-21T14:09:54.600642+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:09:54.600642+0000","last_undegraded":"2024-03-21T14:09:54.600642+0000","last_fullsized":"2024-03-21T14:09:54.600642+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:06.480079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:06.480079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.027337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"173'4789","reported_seq":4821,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.323456+0000","last_change":"2024-03-21T14:09:01.656588+0000","last_active":"2024-03-21T14:09:56.323456+0000","last_peered":"2024-03-21T14:09:56.323456+0000","last_clean":"2024-03-21T14:09:56.323456+0000","last_became_active":"2024-03-21T14:09:01.656279+0000","last_became_peered":"2024-03-21T14:09:01.656279+0000","last_unstale":"2024-03-21T14:09:56.323456+0000","last_undegraded":"2024-03-21T14:09:56.323456+0000","last_fullsized":"2024-03-21T14:09:56.323456+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4789,"log_dups_size":0,"ondisk_log_size":4789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:00.621670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313851904,"num_objects":4789,"num_object_clones":0,"num_object_copies":14367,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4789,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4789,"num_write_kb":306496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":59,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.618215+0000","last_change":"2024-03-21T14:08:17.140702+0000","last_active":"2024-03-21T14:09:56.618215+0000","last_peered":"2024-03-21T14:09:56.618215+0000","last_clean":"2024-03-21T14:09:56.618215+0000","last_became_active":"2024-03-21T14:08:17.140381+0000","last_became_peered":"2024-03-21T14:08:17.140381+0000","last_unstale":"2024-03-21T14:09:56.618215+0000","last_undegraded":"2024-03-21T14:09:56.618215+0000","last_fullsized":"2024-03-21T14:09:56.618215+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.963233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.963233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":262,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601334+0000","last_change":"2024-03-21T14:09:07.478887+0000","last_active":"2024-03-21T14:09:54.601334+0000","last_peered":"2024-03-21T14:09:54.601334+0000","last_clean":"2024-03-21T14:09:54.601334+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:09:54.601334+0000","last_undegraded":"2024-03-21T14:09:54.601334+0000","last_fullsized":"2024-03-21T14:09:54.601334+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:07.478803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:07.478803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:07.531242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":290,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602773+0000","last_change":"2024-03-21T14:09:20.135457+0000","last_active":"2024-03-21T14:09:54.602773+0000","last_peered":"2024-03-21T14:09:54.602773+0000","last_clean":"2024-03-21T14:09:54.602773+0000","last_became_active":"2024-03-21T14:07:23.352151+0000","last_became_peered":"2024-03-21T14:07:23.352151+0000","last_unstale":"2024-03-21T14:09:54.602773+0000","last_undegraded":"2024-03-21T14:09:54.602773+0000","last_fullsized":"2024-03-21T14:09:54.602773+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:20.135288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:20.135288+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:38.512648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.605009+0000","last_change":"2024-03-21T14:09:28.007840+0000","last_active":"2024-03-21T14:09:56.605009+0000","last_peered":"2024-03-21T14:09:56.605009+0000","last_clean":"2024-03-21T14:09:56.605009+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:09:56.605009+0000","last_undegraded":"2024-03-21T14:09:56.605009+0000","last_fullsized":"2024-03-21T14:09:56.605009+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.007714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:46.236621+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"174'4868","reported_seq":4901,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.833029+0000","last_change":"2024-03-21T14:09:01.657853+0000","last_active":"2024-03-21T14:09:56.833029+0000","last_peered":"2024-03-21T14:09:56.833029+0000","last_clean":"2024-03-21T14:09:56.833029+0000","last_became_active":"2024-03-21T14:09:01.656858+0000","last_became_peered":"2024-03-21T14:09:01.656858+0000","last_unstale":"2024-03-21T14:09:56.833029+0000","last_undegraded":"2024-03-21T14:09:56.833029+0000","last_fullsized":"2024-03-21T14:09:56.833029+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4868,"log_dups_size":0,"ondisk_log_size":4868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:05.783679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319029248,"num_objects":4868,"num_object_clones":0,"num_object_copies":14604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4868,"num_write_kb":311552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600402+0000","last_change":"2024-03-21T14:08:21.608247+0000","last_active":"2024-03-21T14:09:54.600402+0000","last_peered":"2024-03-21T14:09:54.600402+0000","last_clean":"2024-03-21T14:09:54.600402+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:09:54.600402+0000","last_undegraded":"2024-03-21T14:09:54.600402+0000","last_fullsized":"2024-03-21T14:09:54.600402+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:21.608128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:21.608128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.251218+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.605811+0000","last_change":"2024-03-21T14:09:04.791742+0000","last_active":"2024-03-21T14:09:56.605811+0000","last_peered":"2024-03-21T14:09:56.605811+0000","last_clean":"2024-03-21T14:09:56.605811+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:09:56.605811+0000","last_undegraded":"2024-03-21T14:09:56.605811+0000","last_fullsized":"2024-03-21T14:09:56.605811+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.791634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.791634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.075282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":254,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601180+0000","last_change":"2024-03-21T14:08:33.537783+0000","last_active":"2024-03-21T14:09:54.601180+0000","last_peered":"2024-03-21T14:09:54.601180+0000","last_clean":"2024-03-21T14:09:54.601180+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:09:54.601180+0000","last_undegraded":"2024-03-21T14:09:54.601180+0000","last_fullsized":"2024-03-21T14:09:54.601180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:33.537669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.537669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"151'28","reported_seq":289,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.591206+0000","last_change":"2024-03-21T14:08:22.013814+0000","last_active":"2024-03-21T14:09:54.591206+0000","last_peered":"2024-03-21T14:09:54.591206+0000","last_clean":"2024-03-21T14:09:54.591206+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:09:54.591206+0000","last_undegraded":"2024-03-21T14:09:54.591206+0000","last_fullsized":"2024-03-21T14:09:54.591206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'28","last_scrub_stamp":"2024-03-21T14:08:22.013464+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:08:22.013464+0000","last_clean_scrub_stamp":"2024-03-21T14:08:22.013464+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"174'4912","reported_seq":4946,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.648514+0000","last_change":"2024-03-21T14:09:01.452705+0000","last_active":"2024-03-21T14:09:56.648514+0000","last_peered":"2024-03-21T14:09:56.648514+0000","last_clean":"2024-03-21T14:09:56.648514+0000","last_became_active":"2024-03-21T14:09:01.452393+0000","last_became_peered":"2024-03-21T14:09:01.452393+0000","last_unstale":"2024-03-21T14:09:56.648514+0000","last_undegraded":"2024-03-21T14:09:56.648514+0000","last_fullsized":"2024-03-21T14:09:56.648514+0000","mapping_epoch":162,"log_start":"165'500","ondisk_log_start":"165'500","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4412,"log_dups_size":0,"ondisk_log_size":4412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.417036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321912832,"num_objects":4912,"num_object_clones":0,"num_object_copies":14736,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4912,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4912,"num_write_kb":314368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":107,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600409+0000","last_change":"2024-03-21T14:09:28.472395+0000","last_active":"2024-03-21T14:09:54.600409+0000","last_peered":"2024-03-21T14:09:54.600409+0000","last_clean":"2024-03-21T14:09:54.600409+0000","last_became_active":"2024-03-21T14:07:23.354443+0000","last_became_peered":"2024-03-21T14:07:23.354443+0000","last_unstale":"2024-03-21T14:09:54.600409+0000","last_undegraded":"2024-03-21T14:09:54.600409+0000","last_fullsized":"2024-03-21T14:09:54.600409+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.472282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.472282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:57.179210+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.605696+0000","last_change":"2024-03-21T14:09:16.008790+0000","last_active":"2024-03-21T14:09:56.605696+0000","last_peered":"2024-03-21T14:09:56.605696+0000","last_clean":"2024-03-21T14:09:56.605696+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:09:56.605696+0000","last_undegraded":"2024-03-21T14:09:56.605696+0000","last_fullsized":"2024-03-21T14:09:56.605696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.008672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.008672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:39.764024+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598739+0000","last_change":"2024-03-21T14:08:19.403663+0000","last_active":"2024-03-21T14:09:54.598739+0000","last_peered":"2024-03-21T14:09:54.598739+0000","last_clean":"2024-03-21T14:09:54.598739+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:09:54.598739+0000","last_undegraded":"2024-03-21T14:09:54.598739+0000","last_fullsized":"2024-03-21T14:09:54.598739+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:19.403545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:19.403545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":292,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.618330+0000","last_change":"2024-03-21T14:09:55.859340+0000","last_active":"2024-03-21T14:09:56.618330+0000","last_peered":"2024-03-21T14:09:56.618330+0000","last_clean":"2024-03-21T14:09:56.618330+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:09:56.618330+0000","last_undegraded":"2024-03-21T14:09:56.618330+0000","last_fullsized":"2024-03-21T14:09:56.618330+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:55.859206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.859206+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.961510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"173'4972","reported_seq":5003,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.291236+0000","last_change":"2024-03-21T14:09:01.657675+0000","last_active":"2024-03-21T14:09:56.291236+0000","last_peered":"2024-03-21T14:09:56.291236+0000","last_clean":"2024-03-21T14:09:56.291236+0000","last_became_active":"2024-03-21T14:09:01.657373+0000","last_became_peered":"2024-03-21T14:09:01.657373+0000","last_unstale":"2024-03-21T14:09:56.291236+0000","last_undegraded":"2024-03-21T14:09:56.291236+0000","last_fullsized":"2024-03-21T14:09:56.291236+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4972,"log_dups_size":0,"ondisk_log_size":4972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.201751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325844992,"num_objects":4972,"num_object_clones":0,"num_object_copies":14916,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4972,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4972,"num_write_kb":318208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.621284+0000","last_change":"2024-03-21T14:08:12.939745+0000","last_active":"2024-03-21T14:09:56.621284+0000","last_peered":"2024-03-21T14:09:56.621284+0000","last_clean":"2024-03-21T14:09:56.621284+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:09:56.621284+0000","last_undegraded":"2024-03-21T14:09:56.621284+0000","last_fullsized":"2024-03-21T14:09:56.621284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:12.939625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.939625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":272,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598276+0000","last_change":"2024-03-21T14:09:17.993826+0000","last_active":"2024-03-21T14:09:54.598276+0000","last_peered":"2024-03-21T14:09:54.598276+0000","last_clean":"2024-03-21T14:09:54.598276+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:09:54.598276+0000","last_undegraded":"2024-03-21T14:09:54.598276+0000","last_fullsized":"2024-03-21T14:09:54.598276+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:09:17.993708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.993708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.738656+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602728+0000","last_change":"2024-03-21T14:07:11.017293+0000","last_active":"2024-03-21T14:09:54.602728+0000","last_peered":"2024-03-21T14:09:54.602728+0000","last_clean":"2024-03-21T14:09:54.602728+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:09:54.602728+0000","last_undegraded":"2024-03-21T14:09:54.602728+0000","last_fullsized":"2024-03-21T14:09:54.602728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:07:11.013553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:07:11.013553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226042+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"128'39","reported_seq":350,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598514+0000","last_change":"2024-03-21T14:07:21.336517+0000","last_active":"2024-03-21T14:09:54.598514+0000","last_peered":"2024-03-21T14:09:54.598514+0000","last_clean":"2024-03-21T14:09:54.598514+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:09:54.598514+0000","last_undegraded":"2024-03-21T14:09:54.598514+0000","last_fullsized":"2024-03-21T14:09:54.598514+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'39","last_scrub_stamp":"2024-03-21T14:07:21.336399+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:07:21.336399+0000","last_clean_scrub_stamp":"2024-03-21T14:07:21.336399+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":109,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611125+0000","last_change":"2024-03-21T14:08:23.091078+0000","last_active":"2024-03-21T14:09:56.611125+0000","last_peered":"2024-03-21T14:09:56.611125+0000","last_clean":"2024-03-21T14:09:56.611125+0000","last_became_active":"2024-03-21T14:07:23.357608+0000","last_became_peered":"2024-03-21T14:07:23.357608+0000","last_unstale":"2024-03-21T14:09:56.611125+0000","last_undegraded":"2024-03-21T14:09:56.611125+0000","last_fullsized":"2024-03-21T14:09:56.611125+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.090775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.090775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755514+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"173'4772","reported_seq":4804,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.303982+0000","last_change":"2024-03-21T14:09:01.656314+0000","last_active":"2024-03-21T14:09:56.303982+0000","last_peered":"2024-03-21T14:09:56.303982+0000","last_clean":"2024-03-21T14:09:56.303982+0000","last_became_active":"2024-03-21T14:09:01.655845+0000","last_became_peered":"2024-03-21T14:09:01.655845+0000","last_unstale":"2024-03-21T14:09:56.303982+0000","last_undegraded":"2024-03-21T14:09:56.303982+0000","last_fullsized":"2024-03-21T14:09:56.303982+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.346889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312737792,"num_objects":4772,"num_object_clones":0,"num_object_copies":14316,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4772,"num_write_kb":305408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601256+0000","last_change":"2024-03-21T14:08:16.661163+0000","last_active":"2024-03-21T14:09:54.601256+0000","last_peered":"2024-03-21T14:09:54.601256+0000","last_clean":"2024-03-21T14:09:54.601256+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:09:54.601256+0000","last_undegraded":"2024-03-21T14:09:54.601256+0000","last_fullsized":"2024-03-21T14:09:54.601256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.661025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.661025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.251245+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":263,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.614816+0000","last_change":"2024-03-21T14:09:28.482765+0000","last_active":"2024-03-21T14:09:56.614816+0000","last_peered":"2024-03-21T14:09:56.614816+0000","last_clean":"2024-03-21T14:09:56.614816+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:09:56.614816+0000","last_undegraded":"2024-03-21T14:09:56.614816+0000","last_fullsized":"2024-03-21T14:09:56.614816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.480168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.480168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:55.996315+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":273,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.607790+0000","last_change":"2024-03-21T14:08:05.789712+0000","last_active":"2024-03-21T14:09:56.607790+0000","last_peered":"2024-03-21T14:09:56.607790+0000","last_clean":"2024-03-21T14:09:56.607790+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:09:56.607790+0000","last_undegraded":"2024-03-21T14:09:56.607790+0000","last_fullsized":"2024-03-21T14:09:56.607790+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:05.789578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:08:05.789578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.758119+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":309,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.613876+0000","last_change":"2024-03-21T14:09:15.326163+0000","last_active":"2024-03-21T14:09:56.613876+0000","last_peered":"2024-03-21T14:09:56.613876+0000","last_clean":"2024-03-21T14:09:56.613876+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:09:56.613876+0000","last_undegraded":"2024-03-21T14:09:56.613876+0000","last_fullsized":"2024-03-21T14:09:56.613876+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:09:15.325868+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.325868+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:42.864275+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.613978+0000","last_change":"2024-03-21T14:08:14.665078+0000","last_active":"2024-03-21T14:09:56.613978+0000","last_peered":"2024-03-21T14:09:56.613978+0000","last_clean":"2024-03-21T14:09:56.613978+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:09:56.613978+0000","last_undegraded":"2024-03-21T14:09:56.613978+0000","last_fullsized":"2024-03-21T14:09:56.613978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:14.664760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:14.664760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"173'4886","reported_seq":4918,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.343226+0000","last_change":"2024-03-21T14:09:01.451546+0000","last_active":"2024-03-21T14:09:56.343226+0000","last_peered":"2024-03-21T14:09:56.343226+0000","last_clean":"2024-03-21T14:09:56.343226+0000","last_became_active":"2024-03-21T14:09:01.451272+0000","last_became_peered":"2024-03-21T14:09:01.451272+0000","last_unstale":"2024-03-21T14:09:56.343226+0000","last_undegraded":"2024-03-21T14:09:56.343226+0000","last_fullsized":"2024-03-21T14:09:56.343226+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4886,"log_dups_size":0,"ondisk_log_size":4886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.510106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320208896,"num_objects":4886,"num_object_clones":0,"num_object_copies":14658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4886,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4886,"num_write_kb":312704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":241,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598520+0000","last_change":"2024-03-21T14:09:04.091898+0000","last_active":"2024-03-21T14:09:54.598520+0000","last_peered":"2024-03-21T14:09:54.598520+0000","last_clean":"2024-03-21T14:09:54.598520+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:09:54.598520+0000","last_undegraded":"2024-03-21T14:09:54.598520+0000","last_fullsized":"2024-03-21T14:09:54.598520+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.091797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.326382+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":533,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600516+0000","last_change":"2024-03-21T14:09:26.513570+0000","last_active":"2024-03-21T14:09:54.600516+0000","last_peered":"2024-03-21T14:09:54.600516+0000","last_clean":"2024-03-21T14:09:54.600516+0000","last_became_active":"2024-03-21T14:07:25.440544+0000","last_became_peered":"2024-03-21T14:07:25.440544+0000","last_unstale":"2024-03-21T14:09:54.600516+0000","last_undegraded":"2024-03-21T14:09:54.600516+0000","last_fullsized":"2024-03-21T14:09:54.600516+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.512762+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:53.511136+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620290+0000","last_change":"2024-03-21T14:08:34.052880+0000","last_active":"2024-03-21T14:09:56.620290+0000","last_peered":"2024-03-21T14:09:56.620290+0000","last_clean":"2024-03-21T14:09:56.620290+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:09:56.620290+0000","last_undegraded":"2024-03-21T14:09:56.620290+0000","last_fullsized":"2024-03-21T14:09:56.620290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:34.052772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:34.052772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":368,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.592255+0000","last_change":"2024-03-21T14:09:27.011808+0000","last_active":"2024-03-21T14:09:54.592255+0000","last_peered":"2024-03-21T14:09:54.592255+0000","last_clean":"2024-03-21T14:09:54.592255+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:09:54.592255+0000","last_undegraded":"2024-03-21T14:09:54.592255+0000","last_fullsized":"2024-03-21T14:09:54.592255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:09:27.011698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:27.011698+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:44.932762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":246,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.601040+0000","last_change":"2024-03-21T14:08:23.654474+0000","last_active":"2024-03-21T14:09:54.601040+0000","last_peered":"2024-03-21T14:09:54.601040+0000","last_clean":"2024-03-21T14:09:54.601040+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:09:54.601040+0000","last_undegraded":"2024-03-21T14:09:54.601040+0000","last_fullsized":"2024-03-21T14:09:54.601040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.654338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.654338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"174'4867","reported_seq":4899,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.625407+0000","last_change":"2024-03-21T14:09:01.656348+0000","last_active":"2024-03-21T14:09:56.625407+0000","last_peered":"2024-03-21T14:09:56.625407+0000","last_clean":"2024-03-21T14:09:56.625407+0000","last_became_active":"2024-03-21T14:09:01.656033+0000","last_became_peered":"2024-03-21T14:09:01.656033+0000","last_unstale":"2024-03-21T14:09:56.625407+0000","last_undegraded":"2024-03-21T14:09:56.625407+0000","last_fullsized":"2024-03-21T14:09:56.625407+0000","mapping_epoch":162,"log_start":"165'500","ondisk_log_start":"165'500","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4367,"log_dups_size":0,"ondisk_log_size":4367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.312139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318963712,"num_objects":4867,"num_object_clones":0,"num_object_copies":14601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4867,"num_write_kb":311488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.610209+0000","last_change":"2024-03-21T14:08:11.045203+0000","last_active":"2024-03-21T14:09:56.610209+0000","last_peered":"2024-03-21T14:09:56.610209+0000","last_clean":"2024-03-21T14:09:56.610209+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:09:56.610209+0000","last_undegraded":"2024-03-21T14:09:56.610209+0000","last_fullsized":"2024-03-21T14:09:56.610209+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:11.045086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:11.045086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755315+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":262,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.618094+0000","last_change":"2024-03-21T14:08:17.537894+0000","last_active":"2024-03-21T14:09:56.618094+0000","last_peered":"2024-03-21T14:09:56.618094+0000","last_clean":"2024-03-21T14:09:56.618094+0000","last_became_active":"2024-03-21T14:07:23.362137+0000","last_became_peered":"2024-03-21T14:07:23.362137+0000","last_unstale":"2024-03-21T14:09:56.618094+0000","last_undegraded":"2024-03-21T14:09:56.618094+0000","last_fullsized":"2024-03-21T14:09:56.618094+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:17.537763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:17.537763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":702,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:55.053891+0000","last_change":"2024-03-21T14:08:15.687962+0000","last_active":"2024-03-21T14:09:55.053891+0000","last_peered":"2024-03-21T14:09:55.053891+0000","last_clean":"2024-03-21T14:09:55.053891+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:09:55.053891+0000","last_undegraded":"2024-03-21T14:09:55.053891+0000","last_fullsized":"2024-03-21T14:09:55.053891+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:08:15.687866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.687866+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":436,"num_read_kb":431,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":159,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611541+0000","last_change":"2024-03-21T14:08:31.177524+0000","last_active":"2024-03-21T14:09:56.611541+0000","last_peered":"2024-03-21T14:09:56.611541+0000","last_clean":"2024-03-21T14:09:56.611541+0000","last_became_active":"2024-03-21T14:07:23.363490+0000","last_became_peered":"2024-03-21T14:07:23.363490+0000","last_unstale":"2024-03-21T14:09:56.611541+0000","last_undegraded":"2024-03-21T14:09:56.611541+0000","last_fullsized":"2024-03-21T14:09:56.611541+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:08:31.177430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.177430+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.599002+0000","last_change":"2024-03-21T14:09:12.000023+0000","last_active":"2024-03-21T14:09:54.599002+0000","last_peered":"2024-03-21T14:09:54.599002+0000","last_clean":"2024-03-21T14:09:54.599002+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:09:54.599002+0000","last_undegraded":"2024-03-21T14:09:54.599002+0000","last_fullsized":"2024-03-21T14:09:54.599002+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.999855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.999855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:12.909535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"174'4931","reported_seq":4965,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.753075+0000","last_change":"2024-03-21T14:09:01.657541+0000","last_active":"2024-03-21T14:09:56.753075+0000","last_peered":"2024-03-21T14:09:56.753075+0000","last_clean":"2024-03-21T14:09:56.753075+0000","last_became_active":"2024-03-21T14:09:01.656838+0000","last_became_peered":"2024-03-21T14:09:01.656838+0000","last_unstale":"2024-03-21T14:09:56.753075+0000","last_undegraded":"2024-03-21T14:09:56.753075+0000","last_fullsized":"2024-03-21T14:09:56.753075+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4931,"log_dups_size":0,"ondisk_log_size":4931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.830747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323158016,"num_objects":4931,"num_object_clones":0,"num_object_copies":14793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4931,"num_write_kb":315584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611790+0000","last_change":"2024-03-21T14:09:25.416918+0000","last_active":"2024-03-21T14:09:56.611790+0000","last_peered":"2024-03-21T14:09:56.611790+0000","last_clean":"2024-03-21T14:09:56.611790+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:09:56.611790+0000","last_undegraded":"2024-03-21T14:09:56.611790+0000","last_fullsized":"2024-03-21T14:09:56.611790+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.416532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.416532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:45.771830+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":263,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.610428+0000","last_change":"2024-03-21T14:09:23.381952+0000","last_active":"2024-03-21T14:09:56.610428+0000","last_peered":"2024-03-21T14:09:56.610428+0000","last_clean":"2024-03-21T14:09:56.610428+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:09:56.610428+0000","last_undegraded":"2024-03-21T14:09:56.610428+0000","last_fullsized":"2024-03-21T14:09:56.610428+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.381813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:28.987847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":296,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.618324+0000","last_change":"2024-03-21T14:08:29.571072+0000","last_active":"2024-03-21T14:09:56.618324+0000","last_peered":"2024-03-21T14:09:56.618324+0000","last_clean":"2024-03-21T14:09:56.618324+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:09:56.618324+0000","last_undegraded":"2024-03-21T14:09:56.618324+0000","last_fullsized":"2024-03-21T14:09:56.618324+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:08:29.570966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:29.570966+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.591302+0000","last_change":"2024-03-21T14:08:32.024468+0000","last_active":"2024-03-21T14:09:54.591302+0000","last_peered":"2024-03-21T14:09:54.591302+0000","last_clean":"2024-03-21T14:09:54.591302+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:09:54.591302+0000","last_undegraded":"2024-03-21T14:09:54.591302+0000","last_fullsized":"2024-03-21T14:09:54.591302+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.024362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:00.767719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"174'5076","reported_seq":5110,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.807472+0000","last_change":"2024-03-21T14:09:01.657966+0000","last_active":"2024-03-21T14:09:56.807472+0000","last_peered":"2024-03-21T14:09:56.807472+0000","last_clean":"2024-03-21T14:09:56.807472+0000","last_became_active":"2024-03-21T14:09:01.657572+0000","last_became_peered":"2024-03-21T14:09:01.657572+0000","last_unstale":"2024-03-21T14:09:56.807472+0000","last_undegraded":"2024-03-21T14:09:56.807472+0000","last_fullsized":"2024-03-21T14:09:56.807472+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332660736,"num_objects":5076,"num_object_clones":0,"num_object_copies":15228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5076,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5076,"num_write_kb":324864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.617557+0000","last_change":"2024-03-21T14:09:54.876245+0000","last_active":"2024-03-21T14:09:56.617557+0000","last_peered":"2024-03-21T14:09:56.617557+0000","last_clean":"2024-03-21T14:09:56.617557+0000","last_became_active":"2024-03-21T14:08:17.136362+0000","last_became_peered":"2024-03-21T14:08:17.136362+0000","last_unstale":"2024-03-21T14:09:56.617557+0000","last_undegraded":"2024-03-21T14:09:56.617557+0000","last_fullsized":"2024-03-21T14:09:56.617557+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:54.875993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:54.875993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:08.671071+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598762+0000","last_change":"2024-03-21T14:09:05.122336+0000","last_active":"2024-03-21T14:09:54.598762+0000","last_peered":"2024-03-21T14:09:54.598762+0000","last_clean":"2024-03-21T14:09:54.598762+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:09:54.598762+0000","last_undegraded":"2024-03-21T14:09:54.598762+0000","last_fullsized":"2024-03-21T14:09:54.598762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:05.122232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.122232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:32.899878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"152'15","reported_seq":279,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.613769+0000","last_change":"2024-03-21T14:08:25.166241+0000","last_active":"2024-03-21T14:09:56.613769+0000","last_peered":"2024-03-21T14:09:56.613769+0000","last_clean":"2024-03-21T14:09:56.613769+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:09:56.613769+0000","last_undegraded":"2024-03-21T14:09:56.613769+0000","last_fullsized":"2024-03-21T14:09:56.613769+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'15","last_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:08:25.164008+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755401+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":353,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.813865+0000","last_change":"2024-03-21T14:09:56.813911+0000","last_active":"2024-03-21T14:09:56.813865+0000","last_peered":"2024-03-21T14:09:56.813865+0000","last_clean":"2024-03-21T14:09:56.813865+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:09:56.813865+0000","last_undegraded":"2024-03-21T14:09:56.813865+0000","last_fullsized":"2024-03-21T14:09:56.813865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:56.813768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.813768+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.384070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"173'4881","reported_seq":4913,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.319653+0000","last_change":"2024-03-21T14:09:01.658726+0000","last_active":"2024-03-21T14:09:56.319653+0000","last_peered":"2024-03-21T14:09:56.319653+0000","last_clean":"2024-03-21T14:09:56.319653+0000","last_became_active":"2024-03-21T14:09:01.656735+0000","last_became_peered":"2024-03-21T14:09:01.656735+0000","last_unstale":"2024-03-21T14:09:56.319653+0000","last_undegraded":"2024-03-21T14:09:56.319653+0000","last_fullsized":"2024-03-21T14:09:56.319653+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4881,"log_dups_size":0,"ondisk_log_size":4881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.297551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319881216,"num_objects":4881,"num_object_clones":0,"num_object_copies":14643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4881,"num_write_kb":312384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600873+0000","last_change":"2024-03-21T14:09:03.463520+0000","last_active":"2024-03-21T14:09:54.600873+0000","last_peered":"2024-03-21T14:09:54.600873+0000","last_clean":"2024-03-21T14:09:54.600873+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:09:54.600873+0000","last_undegraded":"2024-03-21T14:09:54.600873+0000","last_fullsized":"2024-03-21T14:09:54.600873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:03.463400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.463400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:33.088597+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":120,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.611374+0000","last_change":"2024-03-21T14:09:24.419269+0000","last_active":"2024-03-21T14:09:56.611374+0000","last_peered":"2024-03-21T14:09:56.611374+0000","last_clean":"2024-03-21T14:09:56.611374+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:09:56.611374+0000","last_undegraded":"2024-03-21T14:09:56.611374+0000","last_fullsized":"2024-03-21T14:09:56.611374+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:24.418831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.418831+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:46.285189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":270,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602592+0000","last_change":"2024-03-21T14:08:32.898925+0000","last_active":"2024-03-21T14:09:54.602592+0000","last_peered":"2024-03-21T14:09:54.602592+0000","last_clean":"2024-03-21T14:09:54.602592+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:09:54.602592+0000","last_undegraded":"2024-03-21T14:09:54.602592+0000","last_fullsized":"2024-03-21T14:09:54.602592+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:08:32.898753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.898753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226340+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":345,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.625371+0000","last_change":"2024-03-21T14:09:09.098309+0000","last_active":"2024-03-21T14:09:56.625371+0000","last_peered":"2024-03-21T14:09:56.625371+0000","last_clean":"2024-03-21T14:09:56.625371+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:09:56.625371+0000","last_undegraded":"2024-03-21T14:09:56.625371+0000","last_fullsized":"2024-03-21T14:09:56.625371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:09.098187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.098187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.235333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"174'5011","reported_seq":5045,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.756069+0000","last_change":"2024-03-21T14:09:01.657521+0000","last_active":"2024-03-21T14:09:56.756069+0000","last_peered":"2024-03-21T14:09:56.756069+0000","last_clean":"2024-03-21T14:09:56.756069+0000","last_became_active":"2024-03-21T14:09:01.657075+0000","last_became_peered":"2024-03-21T14:09:01.657075+0000","last_unstale":"2024-03-21T14:09:56.756069+0000","last_undegraded":"2024-03-21T14:09:56.756069+0000","last_fullsized":"2024-03-21T14:09:56.756069+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5011,"log_dups_size":0,"ondisk_log_size":5011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:15.497337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328400896,"num_objects":5011,"num_object_clones":0,"num_object_copies":15033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5011,"num_write_kb":320704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":247,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602568+0000","last_change":"2024-03-21T14:08:31.865055+0000","last_active":"2024-03-21T14:09:54.602568+0000","last_peered":"2024-03-21T14:09:54.602568+0000","last_clean":"2024-03-21T14:09:54.602568+0000","last_became_active":"2024-03-21T14:07:23.364265+0000","last_became_peered":"2024-03-21T14:07:23.364265+0000","last_unstale":"2024-03-21T14:09:54.602568+0000","last_undegraded":"2024-03-21T14:09:54.602568+0000","last_fullsized":"2024-03-21T14:09:54.602568+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:31.864937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.864937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.266592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.622497+0000","last_change":"2024-03-21T14:08:26.921914+0000","last_active":"2024-03-21T14:09:56.622497+0000","last_peered":"2024-03-21T14:09:56.622497+0000","last_clean":"2024-03-21T14:09:56.622497+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:09:56.622497+0000","last_undegraded":"2024-03-21T14:09:56.622497+0000","last_fullsized":"2024-03-21T14:09:56.622497+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:26.921568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.921568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":208,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.600899+0000","last_change":"2024-03-21T14:08:20.594480+0000","last_active":"2024-03-21T14:09:54.600899+0000","last_peered":"2024-03-21T14:09:54.600899+0000","last_clean":"2024-03-21T14:09:54.600899+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:09:54.600899+0000","last_undegraded":"2024-03-21T14:09:54.600899+0000","last_fullsized":"2024-03-21T14:09:54.600899+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:08:20.594343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:20.594343+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.251299+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":237,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602417+0000","last_change":"2024-03-21T14:08:16.845392+0000","last_active":"2024-03-21T14:09:54.602417+0000","last_peered":"2024-03-21T14:09:54.602417+0000","last_clean":"2024-03-21T14:09:54.602417+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:09:54.602417+0000","last_undegraded":"2024-03-21T14:09:54.602417+0000","last_fullsized":"2024-03-21T14:09:54.602417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.845286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.845286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226306+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"174'4978","reported_seq":5012,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.627787+0000","last_change":"2024-03-21T14:09:01.452701+0000","last_active":"2024-03-21T14:09:56.627787+0000","last_peered":"2024-03-21T14:09:56.627787+0000","last_clean":"2024-03-21T14:09:56.627787+0000","last_became_active":"2024-03-21T14:09:01.452387+0000","last_became_peered":"2024-03-21T14:09:01.452387+0000","last_unstale":"2024-03-21T14:09:56.627787+0000","last_undegraded":"2024-03-21T14:09:56.627787+0000","last_fullsized":"2024-03-21T14:09:56.627787+0000","mapping_epoch":162,"log_start":"166'600","ondisk_log_start":"166'600","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.296270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326238208,"num_objects":4978,"num_object_clones":0,"num_object_copies":14934,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4978,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4978,"num_write_kb":318592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598575+0000","last_change":"2024-03-21T14:08:17.203692+0000","last_active":"2024-03-21T14:09:54.598575+0000","last_peered":"2024-03-21T14:09:54.598575+0000","last_clean":"2024-03-21T14:09:54.598575+0000","last_became_active":"2024-03-21T14:08:17.203307+0000","last_became_peered":"2024-03-21T14:08:17.203307+0000","last_unstale":"2024-03-21T14:09:54.598575+0000","last_undegraded":"2024-03-21T14:09:54.598575+0000","last_fullsized":"2024-03-21T14:09:54.598575+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:07:08.306757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:07:08.306757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":342,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603489+0000","last_change":"2024-03-21T14:08:26.932388+0000","last_active":"2024-03-21T14:09:54.603489+0000","last_peered":"2024-03-21T14:09:54.603489+0000","last_clean":"2024-03-21T14:09:54.603489+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:09:54.603489+0000","last_undegraded":"2024-03-21T14:09:54.603489+0000","last_fullsized":"2024-03-21T14:09:54.603489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:08:26.932274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.932274+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226279+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":109,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.614987+0000","last_change":"2024-03-21T14:09:13.247866+0000","last_active":"2024-03-21T14:09:56.614987+0000","last_peered":"2024-03-21T14:09:56.614987+0000","last_clean":"2024-03-21T14:09:56.614987+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:09:56.614987+0000","last_undegraded":"2024-03-21T14:09:56.614987+0000","last_fullsized":"2024-03-21T14:09:56.614987+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:13.247726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:13.247726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.028509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.615841+0000","last_change":"2024-03-21T14:07:22.821297+0000","last_active":"2024-03-21T14:09:56.615841+0000","last_peered":"2024-03-21T14:09:56.615841+0000","last_clean":"2024-03-21T14:09:56.615841+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:09:56.615841+0000","last_undegraded":"2024-03-21T14:09:56.615841+0000","last_fullsized":"2024-03-21T14:09:56.615841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:07:22.821166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:07:22.821166+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:57.755234+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"173'4955","reported_seq":4987,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.315391+0000","last_change":"2024-03-21T14:09:01.451050+0000","last_active":"2024-03-21T14:09:56.315391+0000","last_peered":"2024-03-21T14:09:56.315391+0000","last_clean":"2024-03-21T14:09:56.315391+0000","last_became_active":"2024-03-21T14:09:01.450729+0000","last_became_peered":"2024-03-21T14:09:01.450729+0000","last_unstale":"2024-03-21T14:09:56.315391+0000","last_undegraded":"2024-03-21T14:09:56.315391+0000","last_fullsized":"2024-03-21T14:09:56.315391+0000","mapping_epoch":162,"log_start":"165'200","ondisk_log_start":"165'200","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4755,"log_dups_size":0,"ondisk_log_size":4755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.279375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324730880,"num_objects":4955,"num_object_clones":0,"num_object_copies":14865,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4955,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4955,"num_write_kb":317120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.624052+0000","last_change":"2024-03-21T14:08:11.973211+0000","last_active":"2024-03-21T14:09:56.624052+0000","last_peered":"2024-03-21T14:09:56.624052+0000","last_clean":"2024-03-21T14:09:56.624052+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:09:56.624052+0000","last_undegraded":"2024-03-21T14:09:56.624052+0000","last_fullsized":"2024-03-21T14:09:56.624052+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:11.973087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:11.973087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":245,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.598951+0000","last_change":"2024-03-21T14:09:16.047979+0000","last_active":"2024-03-21T14:09:54.598951+0000","last_peered":"2024-03-21T14:09:54.598951+0000","last_clean":"2024-03-21T14:09:54.598951+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:09:54.598951+0000","last_undegraded":"2024-03-21T14:09:54.598951+0000","last_fullsized":"2024-03-21T14:09:54.598951+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.047852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.047852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.726094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.602418+0000","last_change":"2024-03-21T14:07:23.123778+0000","last_active":"2024-03-21T14:09:54.602418+0000","last_peered":"2024-03-21T14:09:54.602418+0000","last_clean":"2024-03-21T14:09:54.602418+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:09:54.602418+0000","last_undegraded":"2024-03-21T14:09:54.602418+0000","last_fullsized":"2024-03-21T14:09:54.602418+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:07:23.123671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:07:23.123671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:55.226070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":268,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603370+0000","last_change":"2024-03-21T14:09:03.147020+0000","last_active":"2024-03-21T14:09:54.603370+0000","last_peered":"2024-03-21T14:09:54.603370+0000","last_clean":"2024-03-21T14:09:54.603370+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:09:54.603370+0000","last_undegraded":"2024-03-21T14:09:54.603370+0000","last_fullsized":"2024-03-21T14:09:54.603370+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:03.146877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.146877+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:05.861799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"174'4899","reported_seq":4933,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.626680+0000","last_change":"2024-03-21T14:09:01.659005+0000","last_active":"2024-03-21T14:09:56.626680+0000","last_peered":"2024-03-21T14:09:56.626680+0000","last_clean":"2024-03-21T14:09:56.626680+0000","last_became_active":"2024-03-21T14:09:01.656962+0000","last_became_peered":"2024-03-21T14:09:01.656962+0000","last_unstale":"2024-03-21T14:09:56.626680+0000","last_undegraded":"2024-03-21T14:09:56.626680+0000","last_fullsized":"2024-03-21T14:09:56.626680+0000","mapping_epoch":162,"log_start":"165'500","ondisk_log_start":"165'500","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4399,"log_dups_size":0,"ondisk_log_size":4399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.506609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321060864,"num_objects":4899,"num_object_clones":0,"num_object_copies":14697,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4899,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4899,"num_write_kb":313536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":253,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.606291+0000","last_change":"2024-03-21T14:08:30.757032+0000","last_active":"2024-03-21T14:09:56.606291+0000","last_peered":"2024-03-21T14:09:56.606291+0000","last_clean":"2024-03-21T14:09:56.606291+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:09:56.606291+0000","last_undegraded":"2024-03-21T14:09:56.606291+0000","last_fullsized":"2024-03-21T14:09:56.606291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:08:30.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:30.756905+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:09:56.758154+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":255,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.610572+0000","last_change":"2024-03-21T14:09:16.366776+0000","last_active":"2024-03-21T14:09:56.610572+0000","last_peered":"2024-03-21T14:09:56.610572+0000","last_clean":"2024-03-21T14:09:56.610572+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:09:56.610572+0000","last_undegraded":"2024-03-21T14:09:56.610572+0000","last_fullsized":"2024-03-21T14:09:56.610572+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.366566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.366566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.932561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.591378+0000","last_change":"2024-03-21T14:08:24.037320+0000","last_active":"2024-03-21T14:09:54.591378+0000","last_peered":"2024-03-21T14:09:54.591378+0000","last_clean":"2024-03-21T14:09:54.591378+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:09:54.591378+0000","last_undegraded":"2024-03-21T14:09:54.591378+0000","last_fullsized":"2024-03-21T14:09:54.591378+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:24.037202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:24.037202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":277,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603083+0000","last_change":"2024-03-21T14:09:04.171171+0000","last_active":"2024-03-21T14:09:54.603083+0000","last_peered":"2024-03-21T14:09:54.603083+0000","last_clean":"2024-03-21T14:09:54.603083+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:09:54.603083+0000","last_undegraded":"2024-03-21T14:09:54.603083+0000","last_fullsized":"2024-03-21T14:09:54.603083+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:09:04.171069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.171069+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:26.513971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"174'4963","reported_seq":4997,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.730194+0000","last_change":"2024-03-21T14:09:01.451288+0000","last_active":"2024-03-21T14:09:56.730194+0000","last_peered":"2024-03-21T14:09:56.730194+0000","last_clean":"2024-03-21T14:09:56.730194+0000","last_became_active":"2024-03-21T14:09:01.451020+0000","last_became_peered":"2024-03-21T14:09:01.451020+0000","last_unstale":"2024-03-21T14:09:56.730194+0000","last_undegraded":"2024-03-21T14:09:56.730194+0000","last_fullsized":"2024-03-21T14:09:56.730194+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.778288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325255168,"num_objects":4963,"num_object_clones":0,"num_object_copies":14889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4963,"num_write_kb":317632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":241,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603494+0000","last_change":"2024-03-21T14:09:09.117631+0000","last_active":"2024-03-21T14:09:54.603494+0000","last_peered":"2024-03-21T14:09:54.603494+0000","last_clean":"2024-03-21T14:09:54.603494+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:09:54.603494+0000","last_undegraded":"2024-03-21T14:09:54.603494+0000","last_fullsized":"2024-03-21T14:09:54.603494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.117496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.117496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.877971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":354,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.605402+0000","last_change":"2024-03-21T14:09:10.904857+0000","last_active":"2024-03-21T14:09:56.605402+0000","last_peered":"2024-03-21T14:09:56.605402+0000","last_clean":"2024-03-21T14:09:56.605402+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:09:56.605402+0000","last_undegraded":"2024-03-21T14:09:56.605402+0000","last_fullsized":"2024-03-21T14:09:56.605402+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:09:10.904772+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.904772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.436321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":262,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-21T14:09:56.620542+0000","last_change":"2024-03-21T14:09:19.247854+0000","last_active":"2024-03-21T14:09:56.620542+0000","last_peered":"2024-03-21T14:09:56.620542+0000","last_clean":"2024-03-21T14:09:56.620542+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:09:56.620542+0000","last_undegraded":"2024-03-21T14:09:56.620542+0000","last_fullsized":"2024-03-21T14:09:56.620542+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.247746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.247746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.335818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"167'15","reported_seq":269,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-21T14:09:54.603014+0000","last_change":"2024-03-21T14:09:22.120079+0000","last_active":"2024-03-21T14:09:54.603014+0000","last_peered":"2024-03-21T14:09:54.603014+0000","last_clean":"2024-03-21T14:09:54.603014+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:09:54.603014+0000","last_undegraded":"2024-03-21T14:09:54.603014+0000","last_fullsized":"2024-03-21T14:09:54.603014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'15","last_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:09:22.119861+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:22.576029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5155520512,"num_objects":78667,"num_object_clones":0,"num_object_copies":236001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":78667,"num_write_kb":5034688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15456993280,"data_stored":15456993280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76367,"ondisk_log_size":76367,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":436,"num_read_kb":431,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103261,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728300,"kb_used_data":1562464,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92016852,"statfs":{"total":95995035648,"available":94225256448,"internally_reserved":0,"allocated":1599963136,"data_stored":1599532695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":169802186},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430306,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1712392,"kb_used_data":1546560,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92032760,"statfs":{"total":95995035648,"available":94241546240,"internally_reserved":0,"allocated":1583677440,"data_stored":1583277648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2408316,"kb_used_data":2164720,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":91336836,"statfs":{"total":95995035648,"available":93528920064,"internally_reserved":0,"allocated":2216673280,"data_stored":2216226762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1837896,"kb_used_data":1585984,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91907256,"statfs":{"total":95995035648,"available":94113030144,"internally_reserved":0,"allocated":1624047616,"data_stored":1623654463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411447,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2404232,"kb_used_data":2166204,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":91340920,"statfs":{"total":95995035648,"available":93533102080,"internally_reserved":0,"allocated":2218192896,"data_stored":2217796784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182513,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1868632,"kb_used_data":1584784,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91876520,"statfs":{"total":95995035648,"available":94081556480,"internally_reserved":0,"allocated":1622818816,"data_stored":1622402776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":290650567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018989,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1729888,"kb_used_data":1564056,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92015264,"statfs":{"total":95995035648,"available":94223630336,"internally_reserved":0,"allocated":1601593344,"data_stored":1601169266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691944,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2940124,"kb_used_data":2792340,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":90805028,"statfs":{"total":95995035648,"available":92984348672,"internally_reserved":0,"allocated":2859356160,"data_stored":2858936939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1624047616,"data_stored":1624047616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2249916416,"data_stored":2249916416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2252996608,"data_stored":2252996608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1627455488,"data_stored":1627455488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1602551808,"data_stored":1602551808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2902458368,"data_stored":2902458368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1595998208,"data_stored":1595998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1601568768,"data_stored":1601568768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:09:59.630 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-21T14:09:59.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:09:59.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:09:59.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:09:59.811 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:09:59.850 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:59.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:09:59.889 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 82806 82790 110.073 81 0.0138837 0.00908018 2024-03-21T14:09:59.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:09:59.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:09:59.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:09:59 smithi118 ceph-mon[33862]: osdmap e176: 8 total, 8 up, 8 in 2024-03-21T14:09:59.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:09:59.970 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:09:59.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:09:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:09:59.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[29576]: osdmap e176: 8 total, 8 up, 8 in 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:10:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:10:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:10:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:10:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:10:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:10:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:09:59 smithi003 ceph-mon[40510]: osdmap e176: 8 total, 8 up, 8 in 2024-03-21T14:10:00.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:00.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:00.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:00.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:00.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:00.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:00.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:00.533 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:00.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:00.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:00.708 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:00.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:00.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: pgmap v303: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 999 op/s 2024-03-21T14:10:00.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:10:00.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:10:00.730 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:10:00.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:10:00.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-21T14:10:00.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-21T14:10:00.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:10:00.731 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:10:00.731 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: pgmap v303: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 999 op/s 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:10:00.732 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:00 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:10:00.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:00.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:00.890 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 84083 84067 109.441 79.8125 0.0142179 0.00913241 2024-03-21T14:10:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: pgmap v303: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 999 op/s 2024-03-21T14:10:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:10:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:10:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:10:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:10:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set 2024-03-21T14:10:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-21T14:10:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:10:00.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:00 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:10:00.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:00.980 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:00.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:01.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:01.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:01.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:01.221 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:01.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:01.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.401 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:01.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:01.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:01.538 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:01.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:01.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:01.678 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:01.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:01.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:01.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:01.890 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 85338 85322 108.808 78.4375 0.0140631 0.00918509 2024-03-21T14:10:01.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:01.928 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:01 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:10:01.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:01 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:10:01.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:01 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:10:01.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:01 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:10:01.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:01.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:10:01.930 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:10:01.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:01.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:01.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:01.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:02.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:02.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:02.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:02.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:02.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.252 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:02.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:02.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:02.388 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:02.410 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:10:02.411 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:10:02.490 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:02.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:02.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:02.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.672 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:02.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:02.673 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:10:02.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:10:02.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:10:02.690 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: pgmap v305: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: osdmap e177: 8 total, 8 up, 8 in 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2429706974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:10:02.691 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: pgmap v305: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: osdmap e177: 8 total, 8 up, 8 in 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2429706974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:02.692 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:02 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:02.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:02.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:02.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:02.890 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 86547 86531 108.143 75.5625 0.00718488 0.00923924 2024-03-21T14:10:02.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: pgmap v305: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: osdmap e177: 8 total, 8 up, 8 in 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:10:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:10:02.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:10:02.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2429706974' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:02.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:02 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:02.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:02.948 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:02.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:03.049 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:03.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:03.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:03.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:03.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:03.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T14:10:03.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:03.373 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:03.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:03.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:03.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:03.633 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:10:03.651 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:03.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:03.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:03.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:03.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:03.890 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 87824 87808 107.588 79.8125 0.0119408 0.00928968 2024-03-21T14:10:03.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: osdmap e178: 8 total, 8 up, 8 in 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:10:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:03 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:10:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:03.933 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:03.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:03.975 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: osdmap e178: 8 total, 8 up, 8 in 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:10:03.976 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: osdmap e178: 8 total, 8 up, 8 in 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:10:03.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:03 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:10:04.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:04.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:04.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:04.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:04.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:04.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:04.312 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:04.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:04.372 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:10:04.420 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:10:04.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:04.462 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:10:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:04.512 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:10:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:04.551 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:10:04.592 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:10:04.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:04.631 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:10:04.631 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:10:04.631 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-21T14:10:04.631 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-21T14:10:04.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:10:04.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:10:04.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:04.672 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:10:04.675 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:10:04.675 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:10:04.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:10:04.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:10:04.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:10:04.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:10:04.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:10:04.676 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:10:04.749 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:04.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:04.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:04.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:04.890 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 88914 88898 106.829 68.125 0.0134347 0.00935576 2024-03-21T14:10:04.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:04.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 5.18 deep-scrub starts 2024-03-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 5.18 deep-scrub ok 2024-03-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: pgmap v308: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 890 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 4.4 deep-scrub starts 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 4.4 deep-scrub ok 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/380083836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[29576]: osdmap e179: 8 total, 8 up, 8 in 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 5.18 deep-scrub starts 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 5.18 deep-scrub ok 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: pgmap v308: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 890 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 4.4 deep-scrub starts 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 4.4 deep-scrub ok 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:10:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:10:05.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/380083836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:05.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:04 smithi003 ceph-mon[40510]: osdmap e179: 8 total, 8 up, 8 in 2024-03-21T14:10:05.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:05.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:05.102 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 5.18 deep-scrub starts 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 5.18 deep-scrub ok 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: pgmap v308: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 890 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 4.4 deep-scrub starts 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 4.4 deep-scrub ok 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:10:05.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/380083836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:05.104 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:04 smithi118 ceph-mon[33862]: osdmap e179: 8 total, 8 up, 8 in 2024-03-21T14:10:05.177 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.212 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:05.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:05.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:05.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:05.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:05.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:05.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:05.618 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:05.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:10:05.720 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:05.754 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:05.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:05.765 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:05.808 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:10:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:10:05.808 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:10:05.809 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:10:05.809 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:10:05.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:05.891 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 90095 90079 106.205 73.8125 0.0177944 0.00941129 2024-03-21T14:10:05.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:05.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:10:05.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:05 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:10:06.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:06.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:06.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:06.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:05 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:10:06.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:05 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:10:06.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:05 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:10:06.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:05 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:10:06.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:06.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:06.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:06.334 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:06.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:06.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:06.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:06.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T14:10:06.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:06.711 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:06.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:10:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:10:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[29576]: pgmap v310: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 865 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T14:10:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:10:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:10:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:10:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:10:06.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:10:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:10:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[40510]: pgmap v310: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 865 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T14:10:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:10:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:10:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:10:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:06 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:10:06.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.756 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:06.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:06.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:06.891 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 91293 91277 105.624 74.875 0.011583 0.00946232 2024-03-21T14:10:06.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:06.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:06.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:06.903 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:10:07.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:07.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:06 smithi118 ceph-mon[33862]: pgmap v310: 148 pgs: 148 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 865 B/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-21T14:10:07.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:06 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:10:07.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:06 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:10:07.034 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:06 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:10:07.034 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:06 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:10:07.037 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:07.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:07.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:07.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:07.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.310 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:07.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:07.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:07.446 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:07.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:07.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:07.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:07.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:07.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:10:07.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:10:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:07 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:10:07.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:07.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:07.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:07.891 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 92377 92361 104.935 67.75 0.00657012 0.00952479 2024-03-21T14:10:07.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:08.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:08.051 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:08.053 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:08.053 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:10:08.053 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:10:08.053 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:10:08.053 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:10:08.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:10:08.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:07 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:10:08.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:08.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.139 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:08.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:08.240 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:08.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:08.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:08.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:08.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:08.547 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:08.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:08.683 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:08.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:08.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:08.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:08.891 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 93495 93479 104.309 69.875 0.00667337 0.00958241 2024-03-21T14:10:08.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:08.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:08.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:08.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: pgmap v311: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:10:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:10:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:10:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:10:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:10:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: pgmap v311: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:10:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:10:09.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:10:09.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:10:09.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:10:09.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:08 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:10:09.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: pgmap v311: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-21T14:10:09.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:10:09.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:10:09.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:10:09.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:10:09.077 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:10:09.077 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:10:09.077 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:10:09.077 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:10:09.077 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:10:09.077 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:08 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:10:09.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:09.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:09.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:09.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:09.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:09.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:09.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:09.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:09.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:09.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:09.651 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:09.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:09.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:09.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:09.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:09.891 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 94652 94636 103.748 72.3125 0.00383095 0.00963168 2024-03-21T14:10:09.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:09.957 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:09 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:10:09.957 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:09 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:10:09.957 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:09 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:10:09.957 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:09 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:10:09.957 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:09 smithi118 ceph-mon[33862]: osdmap e180: 8 total, 8 up, 8 in 2024-03-21T14:10:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:09.965 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:09.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:10:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:10:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:10:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:10:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[29576]: osdmap e180: 8 total, 8 up, 8 in 2024-03-21T14:10:10.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:10:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:10:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:10:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:10:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:09 smithi003 ceph-mon[40510]: osdmap e180: 8 total, 8 up, 8 in 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154023 7.7 7:ec8384af:::benchmark_data_smithi003_80362_object33006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 199.38658951599999, 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.14729518, 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.011 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154023 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:10:10.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621105+0000", 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.155e-06 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621110+0000", 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5790000000000003e-06 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621119+0000", 2024-03-21T14:10:10.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1670000000000005e-06 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621125+0000", 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9370000000000001e-06 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665294+0000", 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044168488999999998 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665370+0000", 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6055000000000003e-05 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665441+0000", 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1031000000000005e-05 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.668869+0000", 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034284250000000001 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.767406+0000", 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.098536532999999996 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768291+0000", 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00088539800000000002 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768347+0000", 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5769000000000001e-05 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768399+0000", 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1641999999999998e-05 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154716 7.7 7:eb3d6bd8:::benchmark_data_smithi003_80362_object33699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 198.23385948000001, 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.135279182, 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.017 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154716 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773834+0000", 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0399999999999995e-07 2024-03-21T14:10:10.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773838+0000", 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9910000000000004e-06 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773846+0000", 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2209999999999997e-06 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773853+0000", 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3890000000000002e-06 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814039+0000", 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040186573000000003 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814159+0000", 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012002600000000001 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814282+0000", 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000123091 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817930+0000", 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036476630000000002 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831508+0000", 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013578365 2024-03-21T14:10:10.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.908985+0000", 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077476415000000007 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909037+0000", 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1929999999999999e-05 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909113+0000", 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5814000000000003e-05 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154717 7.7 7:e048d7ab:::benchmark_data_smithi003_80362_object33700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 198.233598785, 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13536177999999999, 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.024 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154717 2024-03-21T14:10:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774095+0000", 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774098+0000", 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1640000000000001e-06 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774106+0000", 2024-03-21T14:10:10.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3059999999999998e-06 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774112+0000", 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5740000000000003e-06 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816429+0000", 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042316843 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816497+0000", 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8421000000000001e-05 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816574+0000", 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6460999999999999e-05 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820900+0000", 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043267039999999998 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831554+0000", 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010653673000000001 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909389+0000", 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077835369000000001 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909424+0000", 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4275e-05 2024-03-21T14:10:10.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909456+0000", 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2254e-05 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154718 7.7 7:eaddd76b:::benchmark_data_smithi003_80362_object33701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 198.23285384600001, 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13475282299999999, 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.031 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154718 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774840+0000", 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774844+0000", 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.783e-06 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774853+0000", 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5459999999999997e-06 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774859+0000", 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2639999999999997e-06 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817968+0000", 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043108377000000003 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818046+0000", 2024-03-21T14:10:10.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8118e-05 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818119+0000", 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2712000000000004e-05 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820930+0000", 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028118449999999999 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909502+0000", 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.088571516000000003 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909536+0000", 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3813999999999998e-05 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909563+0000", 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7297e-05 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909592+0000", 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8881e-05 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221486 7.7 7:ee81363c:::benchmark_data_smithi003_80362_object100469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.913279027, 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1032354099999999, 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221486 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094415+0000", 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.164e-06 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094423+0000", 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3129999999999999e-06 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094436+0000", 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2646999999999999e-05 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094445+0000", 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7069999999999998e-06 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094500+0000", 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4882000000000001e-05 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094509+0000", 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9369999999999993e-06 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192346+0000", 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0978373850000001 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192463+0000", 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011746799999999999 2024-03-21T14:10:10.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192545+0000", 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2050000000000002e-05 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197520+0000", 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049742279999999998 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197578+0000", 2024-03-21T14:10:10.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8791e-05 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197589+0000", 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0293e-05 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197620+0000", 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1065000000000002e-05 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197649+0000", 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479999999999999e-05 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221488 7.7 7:e6035441:::benchmark_data_smithi003_80362_object100471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.912000664, 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.107609375, 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.044 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221488 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095693+0000", 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5999999999999991e-07 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095698+0000", 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0239999999999997e-06 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095711+0000", 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2445e-05 2024-03-21T14:10:10.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095719+0000", 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0730000000000005e-06 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095770+0000", 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1650000000000002e-05 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095777+0000", 2024-03-21T14:10:10.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2169999999999999e-06 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194772+0000", 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.09899547 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194854+0000", 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2081000000000002e-05 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194929+0000", 2024-03-21T14:10:10.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5067999999999993e-05 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199113+0000", 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041835680000000004 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203213+0000", 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040997869999999997 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203248+0000", 2024-03-21T14:10:10.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5432999999999998e-05 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203277+0000", 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8952e-05 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203302+0000", 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4647e-05 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221491 7.7 7:eaaedede:::benchmark_data_smithi003_80362_object100474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.90919081200001, 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1086137819999999, 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221491 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098503+0000", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3090000000000001e-06 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098508+0000", 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1899999999999997e-06 2024-03-21T14:10:10.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098518+0000", 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0258000000000001e-05 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098525+0000", 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7229999999999998e-06 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098580+0000", 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5015000000000001e-05 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098590+0000", 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.061e-05 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196184+0000", 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0975940799999999 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196262+0000", 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7281999999999996e-05 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196328+0000", 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6224999999999997e-05 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199116+0000", 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027880639999999998 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203355+0000", 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042392979999999999 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207064+0000", 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037090000000000001 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207093+0000", 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8351999999999999e-05 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207116+0000", 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3376000000000001e-05 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221516 7.7 7:ee74cf53:::benchmark_data_smithi003_80362_object100499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.889722858, 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.095088184, 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.057 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221516 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117972+0000", 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6029999999999999e-06 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117979+0000", 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6650000000000003e-06 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117994+0000", 2024-03-21T14:10:10.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5021999999999999e-05 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118002+0000", 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6129999999999998e-06 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118062+0000", 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9768000000000001e-05 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118072+0000", 2024-03-21T14:10:10.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0028e-05 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197714+0000", 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0796421430000001 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197781+0000", 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6649000000000005e-05 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197849+0000", 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8150000000000003e-05 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203310+0000", 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054610889999999997 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203389+0000", 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8960000000000003e-05 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.212981+0000", 2024-03-21T14:10:10.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095917339999999993 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213031+0000", 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0633999999999997e-05 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213058+0000", 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7126000000000001e-05 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221558 7.7 7:e23945db:::benchmark_data_smithi003_80362_object100541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.852423055, 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0579410659999999, 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221558 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155271+0000", 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155279+0000", 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9899999999999997e-06 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155295+0000", 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5366000000000002e-05 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155303+0000", 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9030000000000004e-06 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155469+0000", 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016613000000000001 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155488+0000", 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9151e-05 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199159+0000", 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0436712960000001 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199258+0000", 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9086999999999995e-05 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199325+0000", 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6824999999999998e-05 2024-03-21T14:10:10.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203317+0000", 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003992125 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213108+0000", 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009791213 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213160+0000", 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1712000000000002e-05 2024-03-21T14:10:10.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213189+0000", 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8915999999999999e-05 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213211+0000", 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1837000000000001e-05 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221566 7.7 7:e351aeff:::benchmark_data_smithi003_80362_object100549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.842580885, 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0550122820000001, 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221566 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.071 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165113+0000", 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1999999999999999e-06 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165118+0000", 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3710000000000002e-06 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165126+0000", 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1980000000000001e-06 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165131+0000", 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1229999999999999e-06 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165172+0000", 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1094999999999999e-05 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165178+0000", 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6659999999999996e-06 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200612+0000", 2024-03-21T14:10:10.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.035434781 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200723+0000", 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011056 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200803+0000", 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9759000000000007e-05 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203324+0000", 2024-03-21T14:10:10.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002521129 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213257+0000", 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099332840000000006 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220034+0000", 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067764390000000004 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220082+0000", 2024-03-21T14:10:10.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8179000000000001e-05 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220124+0000", 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2497999999999999e-05 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221598 7.7 7:e857fd29:::benchmark_data_smithi003_80362_object100581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.773249674, 2024-03-21T14:10:10.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99124171000000005, 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221598 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234445+0000", 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5460000000000001e-06 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234452+0000", 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6590000000000007e-06 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234470+0000", 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7876999999999998e-05 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234480+0000", 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8770000000000005e-06 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234548+0000", 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7343999999999998e-05 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234560+0000", 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2707999999999999e-05 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.201983+0000", 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96742289599999998 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.202047+0000", 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3736000000000001e-05 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.202116+0000", 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.902e-05 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203398+0000", 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001282327 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220150+0000", 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016752229 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225577+0000", 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054261459999999997 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225637+0000", 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0489999999999999e-05 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225685+0000", 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7855000000000002e-05 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221605 7.7 7:e86450e5:::benchmark_data_smithi003_80362_object100588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.737724916, 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96634132500000003, 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.084 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221605 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:10:10.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269969+0000", 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2470000000000001e-06 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269975+0000", 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7320000000000003e-06 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269990+0000", 2024-03-21T14:10:10.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4745e-05 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269999+0000", 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0990000000000005e-06 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.270076+0000", 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7044000000000003e-05 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.270085+0000", 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.55e-06 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203422+0000", 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.93333645799999998 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203489+0000", 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7290000000000001e-05 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203555+0000", 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6123000000000004e-05 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207125+0000", 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035693439999999999 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225732+0000", 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018607716999999999 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236215+0000", 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010482131 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236265+0000", 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0158000000000003e-05 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236309+0000", 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4687000000000001e-05 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221618 7.7 7:e934ecf7:::benchmark_data_smithi003_80362_object100601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.72451703599999, 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95328595999999999, 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221618 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:10.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283177+0000", 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283183+0000", 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5829999999999997e-06 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283199+0000", 2024-03-21T14:10:10.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5786000000000001e-05 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283207+0000", 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6370000000000005e-06 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283266+0000", 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8603000000000001e-05 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283274+0000", 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4249999999999998e-06 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204661+0000", 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.92138625600000001 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204729+0000", 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7949999999999998e-05 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204796+0000", 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7284000000000002e-05 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209602+0000", 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00480606 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227784+0000", 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018181953000000001 2024-03-21T14:10:10.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236361+0000", 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085767780000000002 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236418+0000", 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7538999999999999e-05 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236462+0000", 2024-03-21T14:10:10.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3683000000000002e-05 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221625 7.7 7:e490a489:::benchmark_data_smithi003_80362_object100608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.719656837, 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94858037900000003, 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.099 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221625 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288038+0000", 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288044+0000", 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0360000000000003e-06 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288060+0000", 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6498e-05 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288068+0000", 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079999999999998e-06 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288124+0000", 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6141999999999999e-05 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288133+0000", 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4350000000000003e-06 2024-03-21T14:10:10.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.205877+0000", 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91774332000000003 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.205944+0000", 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7387999999999999e-05 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.206008+0000", 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4078000000000006e-05 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209608+0000", 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035997080000000001 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227901+0000", 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018292940000000001 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236522+0000", 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086215389999999992 2024-03-21T14:10:10.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236574+0000", 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1910000000000003e-05 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236616+0000", 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2265999999999998e-05 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.105 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221680 7.7 7:ec92397d:::benchmark_data_smithi003_80362_object100663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.65556541700001, 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.884639709, 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221680 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:10:10.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352130+0000", 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0250000000000001e-06 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352138+0000", 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5930000000000006e-06 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352157+0000", 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9318e-05 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352168+0000", 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0098000000000001e-05 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352233+0000", 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5362000000000002e-05 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352244+0000", 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0985000000000001e-05 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207147+0000", 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85490348999999999 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207222+0000", 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4796999999999996e-05 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207287+0000", 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4978e-05 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.212923+0000", 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056357789999999996 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227945+0000", 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015022064 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236673+0000", 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087278890000000008 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236725+0000", 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2469999999999997e-05 2024-03-21T14:10:10.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236767+0000", 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1860999999999998e-05 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221689 7.7 7:e59419c3:::benchmark_data_smithi003_80362_object100672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.64208330700001, 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87131467699999998, 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.112 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221689 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365611+0000", 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2020000000000001e-06 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365616+0000", 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3789999999999997e-06 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365628+0000", 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1807000000000001e-05 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365635+0000", 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0899999999999999e-06 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365698+0000", 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2691e-05 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365709+0000", 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0706999999999999e-05 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208374+0000", 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84266594399999994 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208442+0000", 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7799999999999995e-05 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208504+0000", 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1441999999999999e-05 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213123+0000", 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046188710000000001 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227986+0000", 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01486326 2024-03-21T14:10:10.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236829+0000", 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088433829999999998 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236880+0000", 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0977999999999999e-05 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236924+0000", 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4122999999999998e-05 2024-03-21T14:10:10.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221705 7.7 7:e134962a:::benchmark_data_smithi003_80362_object100688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.62138746700001, 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85076709500000003, 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.119 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221705 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386307+0000", 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.336e-06 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386313+0000", 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8209999999999998e-06 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386324+0000", 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0895e-05 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386332+0000", 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2740000000000007e-06 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386407+0000", 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5479e-05 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386417+0000", 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3039999999999994e-06 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209643+0000", 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82322670899999995 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209715+0000", 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1707999999999999e-05 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209777+0000", 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2005e-05 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213129+0000", 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033520189999999999 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.228015+0000", 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014885631 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236988+0000", 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089736309999999993 2024-03-21T14:10:10.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237035+0000", 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7126999999999998e-05 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237073+0000", 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7156000000000001e-05 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221707 7.7 7:ee1b885c:::benchmark_data_smithi003_80362_object100690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.61906489899999, 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84853804399999999, 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221707 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388630+0000", 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4929999999999999e-06 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388636+0000", 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.207e-06 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388650+0000", 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4283e-05 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388659+0000", 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5290000000000008e-06 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388717+0000", 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8374e-05 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388729+0000", 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1666e-05 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.210963+0000", 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82223472099999995 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.211082+0000", 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011889500000000001 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.211202+0000", 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119864 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213270+0000", 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020684119999999999 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.228042+0000", 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014771993000000001 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237112+0000", 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090695329999999994 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237143+0000", 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0642999999999998e-05 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237166+0000", 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3431e-05 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221710 7.7 7:e7a06586:::benchmark_data_smithi003_80362_object100693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.61371520900001, 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84331796999999997, 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.132 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221710 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:10:10.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393979+0000", 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.17e-06 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393985+0000", 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8660000000000002e-06 2024-03-21T14:10:10.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393997+0000", 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2218999999999999e-05 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394006+0000", 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0899999999999994e-06 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394056+0000", 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0389000000000002e-05 2024-03-21T14:10:10.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394063+0000", 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7619999999999998e-06 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213298+0000", 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81923456500000003 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213370+0000", 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1940999999999994e-05 2024-03-21T14:10:10.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213432+0000", 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2206999999999996e-05 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.218798+0000", 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005365791 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.231526+0000", 2024-03-21T14:10:10.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012728385 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237210+0000", 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056836960000000002 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237258+0000", 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8300000000000002e-05 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237296+0000", 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7589000000000002e-05 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221712 7.7 7:efee874f:::benchmark_data_smithi003_80362_object100695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 121.608893143, 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.83858521399999997, 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:10:10.140 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221712 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398801+0000", 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398810+0000", 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7399999999999993e-06 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398823+0000", 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3444e-05 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398832+0000", 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0459999999999994e-06 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398880+0000", 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7713000000000002e-05 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398886+0000", 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2949999999999999e-06 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215610+0000", 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81672409700000004 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215726+0000", 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115151 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215844+0000", 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011856100000000001 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.219013+0000", 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003168424 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.231671+0000", 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012658584000000001 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237340+0000", 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056683879999999999 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237366+0000", 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6338000000000001e-05 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237385+0000", 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9133000000000001e-05 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:10.146 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:10:10.151 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:10:10.153 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:10:10.171 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:10.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:10.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:10.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:10.316 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:10.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:10.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:10.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:10.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:10.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:10.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:10.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:10.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:10.724 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:10.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:10.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:10.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: pgmap v312: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 662 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: 5.16 deep-scrub starts 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: 5.16 deep-scrub ok 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: 4.14 deep-scrub starts 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: 4.14 deep-scrub ok 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: pgmap v312: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 662 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: 5.16 deep-scrub starts 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: 5.16 deep-scrub ok 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: 4.14 deep-scrub starts 2024-03-21T14:10:10.780 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: 4.14 deep-scrub ok 2024-03-21T14:10:10.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:10:10.781 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:10 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:10:10.826 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:10.868 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:10.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:10.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:10.891 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 95648 95632 103.032 62.25 0.011345 0.00970117 2024-03-21T14:10:10.970 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.008 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:11.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:11.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:11.138 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: from='client.25037 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: pgmap v312: 148 pgs: 148 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 662 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:10:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: 5.16 deep-scrub starts 2024-03-21T14:10:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: 5.16 deep-scrub ok 2024-03-21T14:10:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: 4.14 deep-scrub starts 2024-03-21T14:10:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: 4.14 deep-scrub ok 2024-03-21T14:10:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:10:11.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:10 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:10:11.240 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.275 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:11.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:11.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:11.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:11.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:11.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:11.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:11.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:11.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:11.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:11.784 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.822 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:11.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:11.890 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-03-21T14:10:07.377883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5958212008,"num_objects":91104,"num_object_clones":0,"num_object_copies":273312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91104,"num_whiteouts":0,"num_read":1377,"num_read_kb":1223,"num_write":91521,"num_write_kb":5820445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":2363178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82765,"ondisk_log_size":82765,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":19072520,"kb_used_data":17366780,"kb_used_omap":12,"kb_used_meta":1705651,"kb_avail":730888696,"statfs":{"total":767960285184,"available":748430024704,"internally_reserved":0,"allocated":17783582720,"data_stored":17780011648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1746587184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,3,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":620494848,"num_objects":9468,"num_object_clones":0,"num_object_copies":28404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9468,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9468,"num_write_kb":605952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001659"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":252,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463661+0000","last_change":"2024-03-21T14:09:55.330094+0000","last_active":"2024-03-21T14:10:04.463661+0000","last_peered":"2024-03-21T14:10:04.463661+0000","last_clean":"2024-03-21T14:10:04.463661+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:10:04.463661+0000","last_undegraded":"2024-03-21T14:10:04.463661+0000","last_fullsized":"2024-03-21T14:10:04.463661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.329965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.329965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.949341+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491195+0000","last_change":"2024-03-21T14:09:02.129524+0000","last_active":"2024-03-21T14:10:04.491195+0000","last_peered":"2024-03-21T14:10:04.491195+0000","last_clean":"2024-03-21T14:10:04.491195+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:10:04.491195+0000","last_undegraded":"2024-03-21T14:10:04.491195+0000","last_fullsized":"2024-03-21T14:10:04.491195+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:02.129441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.129441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.443888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":118,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471324+0000","last_change":"2024-03-21T14:08:26.447065+0000","last_active":"2024-03-21T14:10:04.471324+0000","last_peered":"2024-03-21T14:10:04.471324+0000","last_clean":"2024-03-21T14:10:04.471324+0000","last_became_active":"2024-03-21T14:07:23.349536+0000","last_became_peered":"2024-03-21T14:07:23.349536+0000","last_unstale":"2024-03-21T14:10:04.471324+0000","last_undegraded":"2024-03-21T14:10:04.471324+0000","last_fullsized":"2024-03-21T14:10:04.471324+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:26.446938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.446938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.462860+0000","last_change":"2024-03-21T14:09:15.098201+0000","last_active":"2024-03-21T14:10:04.462860+0000","last_peered":"2024-03-21T14:10:04.462860+0000","last_clean":"2024-03-21T14:10:04.462860+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:10:04.462860+0000","last_undegraded":"2024-03-21T14:10:04.462860+0000","last_fullsized":"2024-03-21T14:10:04.462860+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:15.098048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.098048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:43.207181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":127,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475206+0000","last_change":"2024-03-21T14:09:59.259150+0000","last_active":"2024-03-21T14:10:04.475206+0000","last_peered":"2024-03-21T14:10:04.475206+0000","last_clean":"2024-03-21T14:10:04.475206+0000","last_became_active":"2024-03-21T14:07:23.355776+0000","last_became_peered":"2024-03-21T14:07:23.355776+0000","last_unstale":"2024-03-21T14:10:04.475206+0000","last_undegraded":"2024-03-21T14:10:04.475206+0000","last_fullsized":"2024-03-21T14:10:04.475206+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.259017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.259017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.072409+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.462955+0000","last_change":"2024-03-21T14:10:03.258278+0000","last_active":"2024-03-21T14:10:04.462955+0000","last_peered":"2024-03-21T14:10:04.462955+0000","last_clean":"2024-03-21T14:10:04.462955+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:10:04.462955+0000","last_undegraded":"2024-03-21T14:10:04.462955+0000","last_fullsized":"2024-03-21T14:10:04.462955+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.258168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.901138+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472230+0000","last_change":"2024-03-21T14:08:31.020727+0000","last_active":"2024-03-21T14:10:04.472230+0000","last_peered":"2024-03-21T14:10:04.472230+0000","last_clean":"2024-03-21T14:10:04.472230+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:10:04.472230+0000","last_undegraded":"2024-03-21T14:10:04.472230+0000","last_fullsized":"2024-03-21T14:10:04.472230+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:31.020641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.020641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":302,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.702425+0000","last_change":"2024-03-21T14:10:06.702481+0000","last_active":"2024-03-21T14:10:06.702425+0000","last_peered":"2024-03-21T14:10:06.702425+0000","last_clean":"2024-03-21T14:10:06.702425+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:10:06.702425+0000","last_undegraded":"2024-03-21T14:10:06.702425+0000","last_fullsized":"2024-03-21T14:10:06.702425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:10:06.702330+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.702330+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:16.820634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":118,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491777+0000","last_change":"2024-03-21T14:08:34.165641+0000","last_active":"2024-03-21T14:10:04.491777+0000","last_peered":"2024-03-21T14:10:04.491777+0000","last_clean":"2024-03-21T14:10:04.491777+0000","last_became_active":"2024-03-21T14:07:23.360834+0000","last_became_peered":"2024-03-21T14:07:23.360834+0000","last_unstale":"2024-03-21T14:10:04.491777+0000","last_undegraded":"2024-03-21T14:10:04.491777+0000","last_fullsized":"2024-03-21T14:10:04.491777+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:34.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:34.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471057+0000","last_change":"2024-03-21T14:09:10.043497+0000","last_active":"2024-03-21T14:10:04.471057+0000","last_peered":"2024-03-21T14:10:04.471057+0000","last_clean":"2024-03-21T14:10:04.471057+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:10:04.471057+0000","last_undegraded":"2024-03-21T14:10:04.471057+0000","last_fullsized":"2024-03-21T14:10:04.471057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.043396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.043396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.977610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471303+0000","last_change":"2024-03-21T14:08:23.436895+0000","last_active":"2024-03-21T14:10:04.471303+0000","last_peered":"2024-03-21T14:10:04.471303+0000","last_clean":"2024-03-21T14:10:04.471303+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:10:04.471303+0000","last_undegraded":"2024-03-21T14:10:04.471303+0000","last_fullsized":"2024-03-21T14:10:04.471303+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.436768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.436768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":269,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463864+0000","last_change":"2024-03-21T14:08:12.828629+0000","last_active":"2024-03-21T14:10:04.463864+0000","last_peered":"2024-03-21T14:10:04.463864+0000","last_clean":"2024-03-21T14:10:04.463864+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:10:04.463864+0000","last_undegraded":"2024-03-21T14:10:04.463864+0000","last_fullsized":"2024-03-21T14:10:04.463864+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:08:12.828494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.828494+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.470790+0000","last_change":"2024-03-21T14:08:10.366349+0000","last_active":"2024-03-21T14:10:04.470790+0000","last_peered":"2024-03-21T14:10:04.470790+0000","last_clean":"2024-03-21T14:10:04.470790+0000","last_became_active":"2024-03-21T14:07:23.350761+0000","last_became_peered":"2024-03-21T14:07:23.350761+0000","last_unstale":"2024-03-21T14:10:04.470790+0000","last_undegraded":"2024-03-21T14:10:04.470790+0000","last_fullsized":"2024-03-21T14:10:04.470790+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.366216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.366216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475420+0000","last_change":"2024-03-21T14:09:25.498525+0000","last_active":"2024-03-21T14:10:04.475420+0000","last_peered":"2024-03-21T14:10:04.475420+0000","last_clean":"2024-03-21T14:10:04.475420+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:10:04.475420+0000","last_undegraded":"2024-03-21T14:10:04.475420+0000","last_fullsized":"2024-03-21T14:10:04.475420+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.498388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.498388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:43.494502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472204+0000","last_change":"2024-03-21T14:09:58.390279+0000","last_active":"2024-03-21T14:10:04.472204+0000","last_peered":"2024-03-21T14:10:04.472204+0000","last_clean":"2024-03-21T14:10:04.472204+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:10:04.472204+0000","last_undegraded":"2024-03-21T14:10:04.472204+0000","last_fullsized":"2024-03-21T14:10:04.472204+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:58.390161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.390161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.872420+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464360+0000","last_change":"2024-03-21T14:09:26.168563+0000","last_active":"2024-03-21T14:10:04.464360+0000","last_peered":"2024-03-21T14:10:04.464360+0000","last_clean":"2024-03-21T14:10:04.464360+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:10:04.464360+0000","last_undegraded":"2024-03-21T14:10:04.464360+0000","last_fullsized":"2024-03-21T14:10:04.464360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:09:26.168400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.168400+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.775561+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":245,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.465836+0000","last_change":"2024-03-21T14:08:18.854622+0000","last_active":"2024-03-21T14:10:04.465836+0000","last_peered":"2024-03-21T14:10:04.465836+0000","last_clean":"2024-03-21T14:10:04.465836+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:10:04.465836+0000","last_undegraded":"2024-03-21T14:10:04.465836+0000","last_fullsized":"2024-03-21T14:10:04.465836+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:18.854505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:18.854505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":264,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.499012+0000","last_change":"2024-03-21T14:10:03.746595+0000","last_active":"2024-03-21T14:10:04.499012+0000","last_peered":"2024-03-21T14:10:04.499012+0000","last_clean":"2024-03-21T14:10:04.499012+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:10:04.499012+0000","last_undegraded":"2024-03-21T14:10:04.499012+0000","last_fullsized":"2024-03-21T14:10:04.499012+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.746480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.746480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.495837+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":118,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464232+0000","last_change":"2024-03-21T14:09:19.179886+0000","last_active":"2024-03-21T14:10:04.464232+0000","last_peered":"2024-03-21T14:10:04.464232+0000","last_clean":"2024-03-21T14:10:04.464232+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:10:04.464232+0000","last_undegraded":"2024-03-21T14:10:04.464232+0000","last_fullsized":"2024-03-21T14:10:04.464232+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.179692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.179692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:38.891081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":303,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.474098+0000","last_change":"2024-03-21T14:09:26.954314+0000","last_active":"2024-03-21T14:10:04.474098+0000","last_peered":"2024-03-21T14:10:04.474098+0000","last_clean":"2024-03-21T14:10:04.474098+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:10:04.474098+0000","last_undegraded":"2024-03-21T14:10:04.474098+0000","last_fullsized":"2024-03-21T14:10:04.474098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:09:26.954184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.954184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.810324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472430+0000","last_change":"2024-03-21T14:09:04.145571+0000","last_active":"2024-03-21T14:10:04.472430+0000","last_peered":"2024-03-21T14:10:04.472430+0000","last_clean":"2024-03-21T14:10:04.472430+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:10:04.472430+0000","last_undegraded":"2024-03-21T14:10:04.472430+0000","last_fullsized":"2024-03-21T14:10:04.472430+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.145471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.145471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.785029+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":256,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464371+0000","last_change":"2024-03-21T14:10:04.249646+0000","last_active":"2024-03-21T14:10:04.464371+0000","last_peered":"2024-03-21T14:10:04.464371+0000","last_clean":"2024-03-21T14:10:04.464371+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:10:04.464371+0000","last_undegraded":"2024-03-21T14:10:04.464371+0000","last_fullsized":"2024-03-21T14:10:04.464371+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:04.249513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:04.249513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:18.620999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463960+0000","last_change":"2024-03-21T14:08:15.817968+0000","last_active":"2024-03-21T14:10:04.463960+0000","last_peered":"2024-03-21T14:10:04.463960+0000","last_clean":"2024-03-21T14:10:04.463960+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:10:04.463960+0000","last_undegraded":"2024-03-21T14:10:04.463960+0000","last_fullsized":"2024-03-21T14:10:04.463960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:15.817858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.817858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472018+0000","last_change":"2024-03-21T14:09:27.347989+0000","last_active":"2024-03-21T14:10:04.472018+0000","last_peered":"2024-03-21T14:10:04.472018+0000","last_clean":"2024-03-21T14:10:04.472018+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:10:04.472018+0000","last_undegraded":"2024-03-21T14:10:04.472018+0000","last_fullsized":"2024-03-21T14:10:04.472018+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:27.347835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:27.347835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.996152+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":263,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.477419+0000","last_change":"2024-03-21T14:09:24.494651+0000","last_active":"2024-03-21T14:10:04.477419+0000","last_peered":"2024-03-21T14:10:04.477419+0000","last_clean":"2024-03-21T14:10:04.477419+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:10:04.477419+0000","last_undegraded":"2024-03-21T14:10:04.477419+0000","last_fullsized":"2024-03-21T14:10:04.477419+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.494516+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:32.193912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":370,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.482718+0000","last_change":"2024-03-21T14:09:58.786565+0000","last_active":"2024-03-21T14:10:04.482718+0000","last_peered":"2024-03-21T14:10:04.482718+0000","last_clean":"2024-03-21T14:10:04.482718+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:10:04.482718+0000","last_undegraded":"2024-03-21T14:10:04.482718+0000","last_fullsized":"2024-03-21T14:10:04.482718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:58.786336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.786336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.024325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491108+0000","last_change":"2024-03-21T14:09:09.213981+0000","last_active":"2024-03-21T14:10:04.491108+0000","last_peered":"2024-03-21T14:10:04.491108+0000","last_clean":"2024-03-21T14:10:04.491108+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:10:04.491108+0000","last_undegraded":"2024-03-21T14:10:04.491108+0000","last_fullsized":"2024-03-21T14:10:04.491108+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:09:09.213878+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.213878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.594067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":321,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481332+0000","last_change":"2024-03-21T14:09:02.781223+0000","last_active":"2024-03-21T14:10:04.481332+0000","last_peered":"2024-03-21T14:10:04.481332+0000","last_clean":"2024-03-21T14:10:04.481332+0000","last_became_active":"2024-03-21T14:08:17.144672+0000","last_became_peered":"2024-03-21T14:08:17.144672+0000","last_unstale":"2024-03-21T14:10:04.481332+0000","last_undegraded":"2024-03-21T14:10:04.481332+0000","last_fullsized":"2024-03-21T14:10:04.481332+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:09:02.781039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.781039+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:13.917935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":260,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.490511+0000","last_change":"2024-03-21T14:09:59.790745+0000","last_active":"2024-03-21T14:10:04.490511+0000","last_peered":"2024-03-21T14:10:04.490511+0000","last_clean":"2024-03-21T14:10:04.490511+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:10:04.490511+0000","last_undegraded":"2024-03-21T14:10:04.490511+0000","last_fullsized":"2024-03-21T14:10:04.490511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.790627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.790627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.885224+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481590+0000","last_change":"2024-03-21T14:08:20.617200+0000","last_active":"2024-03-21T14:10:04.481590+0000","last_peered":"2024-03-21T14:10:04.481590+0000","last_clean":"2024-03-21T14:10:04.481590+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:10:04.481590+0000","last_undegraded":"2024-03-21T14:10:04.481590+0000","last_fullsized":"2024-03-21T14:10:04.481590+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:08:20.617078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:05.652461+0000","last_change":"2024-03-21T14:10:05.652480+0000","last_active":"2024-03-21T14:10:05.652461+0000","last_peered":"2024-03-21T14:10:05.652461+0000","last_clean":"2024-03-21T14:10:05.652461+0000","last_became_active":"2024-03-21T14:07:23.364680+0000","last_became_peered":"2024-03-21T14:07:23.364680+0000","last_unstale":"2024-03-21T14:10:05.652461+0000","last_undegraded":"2024-03-21T14:10:05.652461+0000","last_fullsized":"2024-03-21T14:10:05.652461+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:05.652409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:10:05.652409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.074656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"127'16","reported_seq":293,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.482695+0000","last_change":"2024-03-21T14:09:05.048112+0000","last_active":"2024-03-21T14:10:04.482695+0000","last_peered":"2024-03-21T14:10:04.482695+0000","last_clean":"2024-03-21T14:10:04.482695+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:10:04.482695+0000","last_undegraded":"2024-03-21T14:10:04.482695+0000","last_fullsized":"2024-03-21T14:10:04.482695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'16","last_scrub_stamp":"2024-03-21T14:09:05.047983+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.047983+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.538539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.474972+0000","last_change":"2024-03-21T14:08:11.713805+0000","last_active":"2024-03-21T14:10:04.474972+0000","last_peered":"2024-03-21T14:10:04.474972+0000","last_clean":"2024-03-21T14:10:04.474972+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:10:04.474972+0000","last_undegraded":"2024-03-21T14:10:04.474972+0000","last_fullsized":"2024-03-21T14:10:04.474972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:11.713692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:11.713692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.482014+0000","last_change":"2024-03-21T14:08:18.005058+0000","last_active":"2024-03-21T14:10:04.482014+0000","last_peered":"2024-03-21T14:10:04.482014+0000","last_clean":"2024-03-21T14:10:04.482014+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:10:04.482014+0000","last_undegraded":"2024-03-21T14:10:04.482014+0000","last_fullsized":"2024-03-21T14:10:04.482014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:18.004981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:18.004981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481459+0000","last_change":"2024-03-21T14:08:33.573417+0000","last_active":"2024-03-21T14:10:04.481459+0000","last_peered":"2024-03-21T14:10:04.481459+0000","last_clean":"2024-03-21T14:10:04.481459+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:10:04.481459+0000","last_undegraded":"2024-03-21T14:10:04.481459+0000","last_fullsized":"2024-03-21T14:10:04.481459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:33.573334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.573334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471556+0000","last_change":"2024-03-21T14:08:33.005924+0000","last_active":"2024-03-21T14:10:04.471556+0000","last_peered":"2024-03-21T14:10:04.471556+0000","last_clean":"2024-03-21T14:10:04.471556+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:10:04.471556+0000","last_undegraded":"2024-03-21T14:10:04.471556+0000","last_fullsized":"2024-03-21T14:10:04.471556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:08:33.005817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.005817+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471799+0000","last_change":"2024-03-21T14:09:08.089755+0000","last_active":"2024-03-21T14:10:04.471799+0000","last_peered":"2024-03-21T14:10:04.471799+0000","last_clean":"2024-03-21T14:10:04.471799+0000","last_became_active":"2024-03-21T14:08:17.194651+0000","last_became_peered":"2024-03-21T14:08:17.194651+0000","last_unstale":"2024-03-21T14:10:04.471799+0000","last_undegraded":"2024-03-21T14:10:04.471799+0000","last_fullsized":"2024-03-21T14:10:04.471799+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:08.089654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:08.089654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:11.989970+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475689+0000","last_change":"2024-03-21T14:09:09.475589+0000","last_active":"2024-03-21T14:10:04.475689+0000","last_peered":"2024-03-21T14:10:04.475689+0000","last_clean":"2024-03-21T14:10:04.475689+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:10:04.475689+0000","last_undegraded":"2024-03-21T14:10:04.475689+0000","last_fullsized":"2024-03-21T14:10:04.475689+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.475453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.475453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.336499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":279,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471224+0000","last_change":"2024-03-21T14:09:55.427714+0000","last_active":"2024-03-21T14:10:04.471224+0000","last_peered":"2024-03-21T14:10:04.471224+0000","last_clean":"2024-03-21T14:10:04.471224+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:10:04.471224+0000","last_undegraded":"2024-03-21T14:10:04.471224+0000","last_fullsized":"2024-03-21T14:10:04.471224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:55.427563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.427563+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.673618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":289,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471331+0000","last_change":"2024-03-21T14:09:56.592384+0000","last_active":"2024-03-21T14:10:04.471331+0000","last_peered":"2024-03-21T14:10:04.471331+0000","last_clean":"2024-03-21T14:10:04.471331+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:10:04.471331+0000","last_undegraded":"2024-03-21T14:10:04.471331+0000","last_fullsized":"2024-03-21T14:10:04.471331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:56.592242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.592242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.889308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":262,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.479651+0000","last_change":"2024-03-21T14:09:12.819080+0000","last_active":"2024-03-21T14:10:04.479651+0000","last_peered":"2024-03-21T14:10:04.479651+0000","last_clean":"2024-03-21T14:10:04.479651+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:10:04.479651+0000","last_undegraded":"2024-03-21T14:10:04.479651+0000","last_fullsized":"2024-03-21T14:10:04.479651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:12.818982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:12.818982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.361591+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471720+0000","last_change":"2024-03-21T14:09:10.079022+0000","last_active":"2024-03-21T14:10:04.471720+0000","last_peered":"2024-03-21T14:10:04.471720+0000","last_clean":"2024-03-21T14:10:04.471720+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:10:04.471720+0000","last_undegraded":"2024-03-21T14:10:04.471720+0000","last_fullsized":"2024-03-21T14:10:04.471720+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.078939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.078939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.610355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475776+0000","last_change":"2024-03-21T14:10:02.141197+0000","last_active":"2024-03-21T14:10:04.475776+0000","last_peered":"2024-03-21T14:10:04.475776+0000","last_clean":"2024-03-21T14:10:04.475776+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:10:04.475776+0000","last_undegraded":"2024-03-21T14:10:04.475776+0000","last_fullsized":"2024-03-21T14:10:04.475776+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:32.006676+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":285,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491068+0000","last_change":"2024-03-21T14:09:03.162696+0000","last_active":"2024-03-21T14:10:04.491068+0000","last_peered":"2024-03-21T14:10:04.491068+0000","last_clean":"2024-03-21T14:10:04.491068+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:10:04.491068+0000","last_undegraded":"2024-03-21T14:10:04.491068+0000","last_fullsized":"2024-03-21T14:10:04.491068+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:03.162584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.162584+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:11.004039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":119,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491783+0000","last_change":"2024-03-21T14:08:22.602619+0000","last_active":"2024-03-21T14:10:04.491783+0000","last_peered":"2024-03-21T14:10:04.491783+0000","last_clean":"2024-03-21T14:10:04.491783+0000","last_became_active":"2024-03-21T14:07:23.360741+0000","last_became_peered":"2024-03-21T14:07:23.360741+0000","last_unstale":"2024-03-21T14:10:04.491783+0000","last_undegraded":"2024-03-21T14:10:04.491783+0000","last_fullsized":"2024-03-21T14:10:04.491783+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:22.602499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:22.602499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.490854+0000","last_change":"2024-03-21T14:09:26.436609+0000","last_active":"2024-03-21T14:10:04.490854+0000","last_peered":"2024-03-21T14:10:04.490854+0000","last_clean":"2024-03-21T14:10:04.490854+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:10:04.490854+0000","last_undegraded":"2024-03-21T14:10:04.490854+0000","last_fullsized":"2024-03-21T14:10:04.490854+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:26.436326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.436326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:26.638026+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464123+0000","last_change":"2024-03-21T14:09:23.112108+0000","last_active":"2024-03-21T14:10:04.464123+0000","last_peered":"2024-03-21T14:10:04.464123+0000","last_clean":"2024-03-21T14:10:04.464123+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:10:04.464123+0000","last_undegraded":"2024-03-21T14:10:04.464123+0000","last_fullsized":"2024-03-21T14:10:04.464123+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.111999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.111999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:42.481221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"153'16","reported_seq":303,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464410+0000","last_change":"2024-03-21T14:08:30.872145+0000","last_active":"2024-03-21T14:10:04.464410+0000","last_peered":"2024-03-21T14:10:04.464410+0000","last_clean":"2024-03-21T14:10:04.464410+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:10:04.464410+0000","last_undegraded":"2024-03-21T14:10:04.464410+0000","last_fullsized":"2024-03-21T14:10:04.464410+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:08:30.871984+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":119,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.476095+0000","last_change":"2024-03-21T14:09:10.489473+0000","last_active":"2024-03-21T14:10:04.476095+0000","last_peered":"2024-03-21T14:10:04.476095+0000","last_clean":"2024-03-21T14:10:04.476095+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:10:04.476095+0000","last_undegraded":"2024-03-21T14:10:04.476095+0000","last_fullsized":"2024-03-21T14:10:04.476095+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.489356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.489356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.103897+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":364,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471377+0000","last_change":"2024-03-21T14:09:15.064168+0000","last_active":"2024-03-21T14:10:04.471377+0000","last_peered":"2024-03-21T14:10:04.471377+0000","last_clean":"2024-03-21T14:10:04.471377+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:10:04.471377+0000","last_undegraded":"2024-03-21T14:10:04.471377+0000","last_fullsized":"2024-03-21T14:10:04.471377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:15.063993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.063993+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.135176+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475297+0000","last_change":"2024-03-21T14:08:17.142960+0000","last_active":"2024-03-21T14:10:04.475297+0000","last_peered":"2024-03-21T14:10:04.475297+0000","last_clean":"2024-03-21T14:10:04.475297+0000","last_became_active":"2024-03-21T14:08:17.142668+0000","last_became_peered":"2024-03-21T14:08:17.142668+0000","last_unstale":"2024-03-21T14:10:04.475297+0000","last_undegraded":"2024-03-21T14:10:04.475297+0000","last_fullsized":"2024-03-21T14:10:04.475297+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:12.703435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.703435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"72'10","reported_seq":269,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.478350+0000","last_change":"2024-03-21T14:08:15.879476+0000","last_active":"2024-03-21T14:10:04.478350+0000","last_peered":"2024-03-21T14:10:04.478350+0000","last_clean":"2024-03-21T14:10:04.478350+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:10:04.478350+0000","last_undegraded":"2024-03-21T14:10:04.478350+0000","last_fullsized":"2024-03-21T14:10:04.478350+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:08:15.879333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.879333+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464153+0000","last_change":"2024-03-21T14:09:17.192133+0000","last_active":"2024-03-21T14:10:04.464153+0000","last_peered":"2024-03-21T14:10:04.464153+0000","last_clean":"2024-03-21T14:10:04.464153+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:10:04.464153+0000","last_undegraded":"2024-03-21T14:10:04.464153+0000","last_fullsized":"2024-03-21T14:10:04.464153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:17.192018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.192018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.830635+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"130'9","reported_seq":189,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.477360+0000","last_change":"2024-03-21T14:10:00.156687+0000","last_active":"2024-03-21T14:10:04.477360+0000","last_peered":"2024-03-21T14:10:04.477360+0000","last_clean":"2024-03-21T14:10:04.477360+0000","last_became_active":"2024-03-21T14:07:23.354945+0000","last_became_peered":"2024-03-21T14:07:23.354945+0000","last_unstale":"2024-03-21T14:10:04.477360+0000","last_undegraded":"2024-03-21T14:10:04.477360+0000","last_fullsized":"2024-03-21T14:10:04.477360+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:10:00.156602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:00.156602+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.646196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.484241+0000","last_change":"2024-03-21T14:09:21.855689+0000","last_active":"2024-03-21T14:10:04.484241+0000","last_peered":"2024-03-21T14:10:04.484241+0000","last_clean":"2024-03-21T14:10:04.484241+0000","last_became_active":"2024-03-21T14:07:23.358188+0000","last_became_peered":"2024-03-21T14:07:23.358188+0000","last_unstale":"2024-03-21T14:10:04.484241+0000","last_undegraded":"2024-03-21T14:10:04.484241+0000","last_fullsized":"2024-03-21T14:10:04.484241+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:21.852936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:21.852936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.459798+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":277,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471585+0000","last_change":"2024-03-21T14:09:11.022485+0000","last_active":"2024-03-21T14:10:04.471585+0000","last_peered":"2024-03-21T14:10:04.471585+0000","last_clean":"2024-03-21T14:10:04.471585+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:10:04.471585+0000","last_undegraded":"2024-03-21T14:10:04.471585+0000","last_fullsized":"2024-03-21T14:10:04.471585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:11.022384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.022384+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.993211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":262,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.474375+0000","last_change":"2024-03-21T14:09:14.024358+0000","last_active":"2024-03-21T14:10:04.474375+0000","last_peered":"2024-03-21T14:10:04.474375+0000","last_clean":"2024-03-21T14:10:04.474375+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:10:04.474375+0000","last_undegraded":"2024-03-21T14:10:04.474375+0000","last_fullsized":"2024-03-21T14:10:04.474375+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:14.024237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:14.024237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.900582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":118,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.498874+0000","last_change":"2024-03-21T14:09:11.232306+0000","last_active":"2024-03-21T14:10:04.498874+0000","last_peered":"2024-03-21T14:10:04.498874+0000","last_clean":"2024-03-21T14:10:04.498874+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:10:04.498874+0000","last_undegraded":"2024-03-21T14:10:04.498874+0000","last_fullsized":"2024-03-21T14:10:04.498874+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.232197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.232197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:40.426883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.474969+0000","last_change":"2024-03-21T14:09:55.186771+0000","last_active":"2024-03-21T14:10:04.474969+0000","last_peered":"2024-03-21T14:10:04.474969+0000","last_clean":"2024-03-21T14:10:04.474969+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:10:04.474969+0000","last_undegraded":"2024-03-21T14:10:04.474969+0000","last_fullsized":"2024-03-21T14:10:04.474969+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.186641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.186641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:55.374836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":286,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481432+0000","last_change":"2024-03-21T14:09:05.784892+0000","last_active":"2024-03-21T14:10:04.481432+0000","last_peered":"2024-03-21T14:10:04.481432+0000","last_clean":"2024-03-21T14:10:04.481432+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:10:04.481432+0000","last_undegraded":"2024-03-21T14:10:04.481432+0000","last_fullsized":"2024-03-21T14:10:04.481432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:05.784714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.784714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:33.485395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475284+0000","last_change":"2024-03-21T14:08:10.733076+0000","last_active":"2024-03-21T14:10:04.475284+0000","last_peered":"2024-03-21T14:10:04.475284+0000","last_clean":"2024-03-21T14:10:04.475284+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:10:04.475284+0000","last_undegraded":"2024-03-21T14:10:04.475284+0000","last_fullsized":"2024-03-21T14:10:04.475284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.732971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.732971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471219+0000","last_change":"2024-03-21T14:08:13.407387+0000","last_active":"2024-03-21T14:10:04.471219+0000","last_peered":"2024-03-21T14:10:04.471219+0000","last_clean":"2024-03-21T14:10:04.471219+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:10:04.471219+0000","last_undegraded":"2024-03-21T14:10:04.471219+0000","last_fullsized":"2024-03-21T14:10:04.471219+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:13.407278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:13.407278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":279,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481168+0000","last_change":"2024-03-21T14:09:04.020412+0000","last_active":"2024-03-21T14:10:04.481168+0000","last_peered":"2024-03-21T14:10:04.481168+0000","last_clean":"2024-03-21T14:10:04.481168+0000","last_became_active":"2024-03-21T14:07:23.354457+0000","last_became_peered":"2024-03-21T14:07:23.354457+0000","last_unstale":"2024-03-21T14:10:04.481168+0000","last_undegraded":"2024-03-21T14:10:04.481168+0000","last_fullsized":"2024-03-21T14:10:04.481168+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.020323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.020323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.587036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":155,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475836+0000","last_change":"2024-03-21T14:09:15.585161+0000","last_active":"2024-03-21T14:10:04.475836+0000","last_peered":"2024-03-21T14:10:04.475836+0000","last_clean":"2024-03-21T14:10:04.475836+0000","last_became_active":"2024-03-21T14:07:23.365278+0000","last_became_peered":"2024-03-21T14:07:23.365278+0000","last_unstale":"2024-03-21T14:10:04.475836+0000","last_undegraded":"2024-03-21T14:10:04.475836+0000","last_fullsized":"2024-03-21T14:10:04.475836+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:09:15.584995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.584995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.826538+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"179'5676","reported_seq":5719,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.350367+0000","last_change":"2024-03-21T14:09:01.659122+0000","last_active":"2024-03-21T14:10:06.350367+0000","last_peered":"2024-03-21T14:10:06.350367+0000","last_clean":"2024-03-21T14:10:06.350367+0000","last_became_active":"2024-03-21T14:09:01.658623+0000","last_became_peered":"2024-03-21T14:09:01.658623+0000","last_unstale":"2024-03-21T14:10:06.350367+0000","last_undegraded":"2024-03-21T14:10:06.350367+0000","last_fullsized":"2024-03-21T14:10:06.350367+0000","mapping_epoch":162,"log_start":"166'600","ondisk_log_start":"166'600","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5076,"log_dups_size":0,"ondisk_log_size":5076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":363264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471484+0000","last_change":"2024-03-21T14:08:32.402570+0000","last_active":"2024-03-21T14:10:04.471484+0000","last_peered":"2024-03-21T14:10:04.471484+0000","last_clean":"2024-03-21T14:10:04.471484+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:10:04.471484+0000","last_undegraded":"2024-03-21T14:10:04.471484+0000","last_fullsized":"2024-03-21T14:10:04.471484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:32.402481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.402481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":290,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475796+0000","last_change":"2024-03-21T14:08:24.629488+0000","last_active":"2024-03-21T14:10:04.475796+0000","last_peered":"2024-03-21T14:10:04.475796+0000","last_clean":"2024-03-21T14:10:04.475796+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:10:04.475796+0000","last_undegraded":"2024-03-21T14:10:04.475796+0000","last_fullsized":"2024-03-21T14:10:04.475796+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:08:24.629266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:24.629266+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":286,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471049+0000","last_change":"2024-03-21T14:08:15.390480+0000","last_active":"2024-03-21T14:10:04.471049+0000","last_peered":"2024-03-21T14:10:04.471049+0000","last_clean":"2024-03-21T14:10:04.471049+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:10:04.471049+0000","last_undegraded":"2024-03-21T14:10:04.471049+0000","last_fullsized":"2024-03-21T14:10:04.471049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:08:15.390326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.390326+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":357,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475084+0000","last_change":"2024-03-21T14:09:06.480162+0000","last_active":"2024-03-21T14:10:04.475084+0000","last_peered":"2024-03-21T14:10:04.475084+0000","last_clean":"2024-03-21T14:10:04.475084+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:10:04.475084+0000","last_undegraded":"2024-03-21T14:10:04.475084+0000","last_fullsized":"2024-03-21T14:10:04.475084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:06.480079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:06.480079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.027337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"179'5552","reported_seq":5596,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.366606+0000","last_change":"2024-03-21T14:09:01.656588+0000","last_active":"2024-03-21T14:10:06.366606+0000","last_peered":"2024-03-21T14:10:06.366606+0000","last_clean":"2024-03-21T14:10:06.366606+0000","last_became_active":"2024-03-21T14:09:01.656279+0000","last_became_peered":"2024-03-21T14:09:01.656279+0000","last_unstale":"2024-03-21T14:10:06.366606+0000","last_undegraded":"2024-03-21T14:10:06.366606+0000","last_fullsized":"2024-03-21T14:10:06.366606+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5552,"log_dups_size":0,"ondisk_log_size":5552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363855872,"num_objects":5552,"num_object_clones":0,"num_object_copies":16656,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5552,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5552,"num_write_kb":355328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":77,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.480460+0000","last_change":"2024-03-21T14:10:02.896547+0000","last_active":"2024-03-21T14:10:04.480460+0000","last_peered":"2024-03-21T14:10:04.480460+0000","last_clean":"2024-03-21T14:10:04.480460+0000","last_became_active":"2024-03-21T14:08:17.140381+0000","last_became_peered":"2024-03-21T14:08:17.140381+0000","last_unstale":"2024-03-21T14:10:04.480460+0000","last_undegraded":"2024-03-21T14:10:04.480460+0000","last_fullsized":"2024-03-21T14:10:04.480460+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.896413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.896413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:06.122825+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.476196+0000","last_change":"2024-03-21T14:09:07.478887+0000","last_active":"2024-03-21T14:10:04.476196+0000","last_peered":"2024-03-21T14:10:04.476196+0000","last_clean":"2024-03-21T14:10:04.476196+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:10:04.476196+0000","last_undegraded":"2024-03-21T14:10:04.476196+0000","last_fullsized":"2024-03-21T14:10:04.476196+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:07.478803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:07.478803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:07.531242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":302,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463527+0000","last_change":"2024-03-21T14:09:20.135457+0000","last_active":"2024-03-21T14:10:04.463527+0000","last_peered":"2024-03-21T14:10:04.463527+0000","last_clean":"2024-03-21T14:10:04.463527+0000","last_became_active":"2024-03-21T14:07:23.352151+0000","last_became_peered":"2024-03-21T14:07:23.352151+0000","last_unstale":"2024-03-21T14:10:04.463527+0000","last_undegraded":"2024-03-21T14:10:04.463527+0000","last_fullsized":"2024-03-21T14:10:04.463527+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:20.135288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:20.135288+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:38.512648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.470615+0000","last_change":"2024-03-21T14:09:28.007840+0000","last_active":"2024-03-21T14:10:04.470615+0000","last_peered":"2024-03-21T14:10:04.470615+0000","last_clean":"2024-03-21T14:10:04.470615+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:10:04.470615+0000","last_undegraded":"2024-03-21T14:10:04.470615+0000","last_fullsized":"2024-03-21T14:10:04.470615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.007714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:46.236621+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"179'5630","reported_seq":5673,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.849358+0000","last_change":"2024-03-21T14:09:01.657853+0000","last_active":"2024-03-21T14:10:06.849358+0000","last_peered":"2024-03-21T14:10:06.849358+0000","last_clean":"2024-03-21T14:10:06.849358+0000","last_became_active":"2024-03-21T14:09:01.656858+0000","last_became_peered":"2024-03-21T14:09:01.656858+0000","last_unstale":"2024-03-21T14:10:06.849358+0000","last_undegraded":"2024-03-21T14:10:06.849358+0000","last_fullsized":"2024-03-21T14:10:06.849358+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368967680,"num_objects":5630,"num_object_clones":0,"num_object_copies":16890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5630,"num_write_kb":360320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475140+0000","last_change":"2024-03-21T14:08:21.608247+0000","last_active":"2024-03-21T14:10:04.475140+0000","last_peered":"2024-03-21T14:10:04.475140+0000","last_clean":"2024-03-21T14:10:04.475140+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:10:04.475140+0000","last_undegraded":"2024-03-21T14:10:04.475140+0000","last_fullsized":"2024-03-21T14:10:04.475140+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:21.608128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:21.608128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472059+0000","last_change":"2024-03-21T14:09:04.791742+0000","last_active":"2024-03-21T14:10:04.472059+0000","last_peered":"2024-03-21T14:10:04.472059+0000","last_clean":"2024-03-21T14:10:04.472059+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:10:04.472059+0000","last_undegraded":"2024-03-21T14:10:04.472059+0000","last_fullsized":"2024-03-21T14:10:04.472059+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.791634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.791634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.075282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475899+0000","last_change":"2024-03-21T14:08:33.537783+0000","last_active":"2024-03-21T14:10:04.475899+0000","last_peered":"2024-03-21T14:10:04.475899+0000","last_clean":"2024-03-21T14:10:04.475899+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:10:04.475899+0000","last_undegraded":"2024-03-21T14:10:04.475899+0000","last_fullsized":"2024-03-21T14:10:04.475899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:33.537669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.537669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"151'28","reported_seq":309,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481503+0000","last_change":"2024-03-21T14:10:03.820489+0000","last_active":"2024-03-21T14:10:04.481503+0000","last_peered":"2024-03-21T14:10:04.481503+0000","last_clean":"2024-03-21T14:10:04.481503+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:10:04.481503+0000","last_undegraded":"2024-03-21T14:10:04.481503+0000","last_fullsized":"2024-03-21T14:10:04.481503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'28","last_scrub_stamp":"2024-03-21T14:10:03.820339+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:08:22.013464+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.820339+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:10.338619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"179'5661","reported_seq":5705,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.627914+0000","last_change":"2024-03-21T14:09:01.452705+0000","last_active":"2024-03-21T14:10:06.627914+0000","last_peered":"2024-03-21T14:10:06.627914+0000","last_clean":"2024-03-21T14:10:06.627914+0000","last_became_active":"2024-03-21T14:09:01.452393+0000","last_became_peered":"2024-03-21T14:09:01.452393+0000","last_unstale":"2024-03-21T14:10:06.627914+0000","last_undegraded":"2024-03-21T14:10:06.627914+0000","last_fullsized":"2024-03-21T14:10:06.627914+0000","mapping_epoch":162,"log_start":"167'1300","ondisk_log_start":"167'1300","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.417036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999296,"num_objects":5661,"num_object_clones":0,"num_object_copies":16983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5661,"num_write_kb":362304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":119,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.474810+0000","last_change":"2024-03-21T14:09:28.472395+0000","last_active":"2024-03-21T14:10:04.474810+0000","last_peered":"2024-03-21T14:10:04.474810+0000","last_clean":"2024-03-21T14:10:04.474810+0000","last_became_active":"2024-03-21T14:07:23.354443+0000","last_became_peered":"2024-03-21T14:07:23.354443+0000","last_unstale":"2024-03-21T14:10:04.474810+0000","last_undegraded":"2024-03-21T14:10:04.474810+0000","last_fullsized":"2024-03-21T14:10:04.474810+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.472282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.472282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:57.179210+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":274,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475083+0000","last_change":"2024-03-21T14:09:16.008790+0000","last_active":"2024-03-21T14:10:04.475083+0000","last_peered":"2024-03-21T14:10:04.475083+0000","last_clean":"2024-03-21T14:10:04.475083+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:10:04.475083+0000","last_undegraded":"2024-03-21T14:10:04.475083+0000","last_fullsized":"2024-03-21T14:10:04.475083+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.008672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.008672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:39.764024+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472205+0000","last_change":"2024-03-21T14:08:19.403663+0000","last_active":"2024-03-21T14:10:04.472205+0000","last_peered":"2024-03-21T14:10:04.472205+0000","last_clean":"2024-03-21T14:10:04.472205+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:10:04.472205+0000","last_undegraded":"2024-03-21T14:10:04.472205+0000","last_fullsized":"2024-03-21T14:10:04.472205+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:19.403545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:19.403545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":302,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.480771+0000","last_change":"2024-03-21T14:09:55.859340+0000","last_active":"2024-03-21T14:10:04.480771+0000","last_peered":"2024-03-21T14:10:04.480771+0000","last_clean":"2024-03-21T14:10:04.480771+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:10:04.480771+0000","last_undegraded":"2024-03-21T14:10:04.480771+0000","last_fullsized":"2024-03-21T14:10:04.480771+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:55.859206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.859206+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.961510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"179'5743","reported_seq":5787,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.344691+0000","last_change":"2024-03-21T14:09:01.657675+0000","last_active":"2024-03-21T14:10:06.344691+0000","last_peered":"2024-03-21T14:10:06.344691+0000","last_clean":"2024-03-21T14:10:06.344691+0000","last_became_active":"2024-03-21T14:09:01.657373+0000","last_became_peered":"2024-03-21T14:09:01.657373+0000","last_unstale":"2024-03-21T14:10:06.344691+0000","last_undegraded":"2024-03-21T14:10:06.344691+0000","last_fullsized":"2024-03-21T14:10:06.344691+0000","mapping_epoch":162,"log_start":"166'700","ondisk_log_start":"166'700","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5043,"log_dups_size":0,"ondisk_log_size":5043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.201751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376373248,"num_objects":5743,"num_object_clones":0,"num_object_copies":17229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5743,"num_write_kb":367552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.683116+0000","last_change":"2024-03-21T14:10:04.683158+0000","last_active":"2024-03-21T14:10:04.683116+0000","last_peered":"2024-03-21T14:10:04.683116+0000","last_clean":"2024-03-21T14:10:04.683116+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:10:04.683116+0000","last_undegraded":"2024-03-21T14:10:04.683116+0000","last_fullsized":"2024-03-21T14:10:04.683116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:04.683006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:04.683006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.743596+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":284,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471180+0000","last_change":"2024-03-21T14:09:17.993826+0000","last_active":"2024-03-21T14:10:04.471180+0000","last_peered":"2024-03-21T14:10:04.471180+0000","last_clean":"2024-03-21T14:10:04.471180+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:10:04.471180+0000","last_undegraded":"2024-03-21T14:10:04.471180+0000","last_fullsized":"2024-03-21T14:10:04.471180+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:09:17.993708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.993708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.738656+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463923+0000","last_change":"2024-03-21T14:10:01.249279+0000","last_active":"2024-03-21T14:10:04.463923+0000","last_peered":"2024-03-21T14:10:04.463923+0000","last_clean":"2024-03-21T14:10:04.463923+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:10:04.463923+0000","last_undegraded":"2024-03-21T14:10:04.463923+0000","last_fullsized":"2024-03-21T14:10:04.463923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:01.249165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.249165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.783002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"128'39","reported_seq":372,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472068+0000","last_change":"2024-03-21T14:10:03.405443+0000","last_active":"2024-03-21T14:10:04.472068+0000","last_peered":"2024-03-21T14:10:04.472068+0000","last_clean":"2024-03-21T14:10:04.472068+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:10:04.472068+0000","last_undegraded":"2024-03-21T14:10:04.472068+0000","last_fullsized":"2024-03-21T14:10:04.472068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'39","last_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.405289+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:11.513185+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":119,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.499096+0000","last_change":"2024-03-21T14:08:23.091078+0000","last_active":"2024-03-21T14:10:04.499096+0000","last_peered":"2024-03-21T14:10:04.499096+0000","last_clean":"2024-03-21T14:10:04.499096+0000","last_became_active":"2024-03-21T14:07:23.357608+0000","last_became_peered":"2024-03-21T14:07:23.357608+0000","last_unstale":"2024-03-21T14:10:04.499096+0000","last_undegraded":"2024-03-21T14:10:04.499096+0000","last_fullsized":"2024-03-21T14:10:04.499096+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.090775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.090775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"179'5576","reported_seq":5620,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.307503+0000","last_change":"2024-03-21T14:09:01.656314+0000","last_active":"2024-03-21T14:10:06.307503+0000","last_peered":"2024-03-21T14:10:06.307503+0000","last_clean":"2024-03-21T14:10:06.307503+0000","last_became_active":"2024-03-21T14:09:01.655845+0000","last_became_peered":"2024-03-21T14:09:01.655845+0000","last_unstale":"2024-03-21T14:10:06.307503+0000","last_undegraded":"2024-03-21T14:10:06.307503+0000","last_fullsized":"2024-03-21T14:10:06.307503+0000","mapping_epoch":162,"log_start":"166'800","ondisk_log_start":"166'800","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.346889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365428736,"num_objects":5576,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5576,"num_write_kb":356864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475990+0000","last_change":"2024-03-21T14:08:16.661163+0000","last_active":"2024-03-21T14:10:04.475990+0000","last_peered":"2024-03-21T14:10:04.475990+0000","last_clean":"2024-03-21T14:10:04.475990+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:10:04.475990+0000","last_undegraded":"2024-03-21T14:10:04.475990+0000","last_fullsized":"2024-03-21T14:10:04.475990+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.661025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.661025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.492065+0000","last_change":"2024-03-21T14:09:28.482765+0000","last_active":"2024-03-21T14:10:04.492065+0000","last_peered":"2024-03-21T14:10:04.492065+0000","last_clean":"2024-03-21T14:10:04.492065+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:10:04.492065+0000","last_undegraded":"2024-03-21T14:10:04.492065+0000","last_fullsized":"2024-03-21T14:10:04.492065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.480168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.480168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:55.996315+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":291,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.470426+0000","last_change":"2024-03-21T14:09:58.644132+0000","last_active":"2024-03-21T14:10:04.470426+0000","last_peered":"2024-03-21T14:10:04.470426+0000","last_clean":"2024-03-21T14:10:04.470426+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:10:04.470426+0000","last_undegraded":"2024-03-21T14:10:04.470426+0000","last_fullsized":"2024-03-21T14:10:04.470426+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:58.643445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.643445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:06.955399+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":319,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.498497+0000","last_change":"2024-03-21T14:09:15.326163+0000","last_active":"2024-03-21T14:10:04.498497+0000","last_peered":"2024-03-21T14:10:04.498497+0000","last_clean":"2024-03-21T14:10:04.498497+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:10:04.498497+0000","last_undegraded":"2024-03-21T14:10:04.498497+0000","last_fullsized":"2024-03-21T14:10:04.498497+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:09:15.325868+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.325868+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:42.864275+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471872+0000","last_change":"2024-03-21T14:08:14.665078+0000","last_active":"2024-03-21T14:10:04.471872+0000","last_peered":"2024-03-21T14:10:04.471872+0000","last_clean":"2024-03-21T14:10:04.471872+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:10:04.471872+0000","last_undegraded":"2024-03-21T14:10:04.471872+0000","last_fullsized":"2024-03-21T14:10:04.471872+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:14.664760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:14.664760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"179'5638","reported_seq":5682,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.360008+0000","last_change":"2024-03-21T14:09:01.451546+0000","last_active":"2024-03-21T14:10:06.360008+0000","last_peered":"2024-03-21T14:10:06.360008+0000","last_clean":"2024-03-21T14:10:06.360008+0000","last_became_active":"2024-03-21T14:09:01.451272+0000","last_became_peered":"2024-03-21T14:09:01.451272+0000","last_unstale":"2024-03-21T14:10:06.360008+0000","last_undegraded":"2024-03-21T14:10:06.360008+0000","last_fullsized":"2024-03-21T14:10:06.360008+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5638,"log_dups_size":0,"ondisk_log_size":5638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.510106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369491968,"num_objects":5638,"num_object_clones":0,"num_object_copies":16914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5638,"num_write_kb":360832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471276+0000","last_change":"2024-03-21T14:09:04.091898+0000","last_active":"2024-03-21T14:10:04.471276+0000","last_peered":"2024-03-21T14:10:04.471276+0000","last_clean":"2024-03-21T14:10:04.471276+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:10:04.471276+0000","last_undegraded":"2024-03-21T14:10:04.471276+0000","last_fullsized":"2024-03-21T14:10:04.471276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.091797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.326382+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":545,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475347+0000","last_change":"2024-03-21T14:09:26.513570+0000","last_active":"2024-03-21T14:10:04.475347+0000","last_peered":"2024-03-21T14:10:04.475347+0000","last_clean":"2024-03-21T14:10:04.475347+0000","last_became_active":"2024-03-21T14:07:25.440544+0000","last_became_peered":"2024-03-21T14:07:25.440544+0000","last_unstale":"2024-03-21T14:10:04.475347+0000","last_undegraded":"2024-03-21T14:10:04.475347+0000","last_fullsized":"2024-03-21T14:10:04.475347+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.512762+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:53.511136+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471367+0000","last_change":"2024-03-21T14:08:34.052880+0000","last_active":"2024-03-21T14:10:04.471367+0000","last_peered":"2024-03-21T14:10:04.471367+0000","last_clean":"2024-03-21T14:10:04.471367+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:10:04.471367+0000","last_undegraded":"2024-03-21T14:10:04.471367+0000","last_fullsized":"2024-03-21T14:10:04.471367+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:34.052772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:34.052772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":382,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.482629+0000","last_change":"2024-03-21T14:09:27.011808+0000","last_active":"2024-03-21T14:10:04.482629+0000","last_peered":"2024-03-21T14:10:04.482629+0000","last_clean":"2024-03-21T14:10:04.482629+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:10:04.482629+0000","last_undegraded":"2024-03-21T14:10:04.482629+0000","last_fullsized":"2024-03-21T14:10:04.482629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:09:27.011698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:27.011698+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:44.932762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":258,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.477470+0000","last_change":"2024-03-21T14:08:23.654474+0000","last_active":"2024-03-21T14:10:04.477470+0000","last_peered":"2024-03-21T14:10:04.477470+0000","last_clean":"2024-03-21T14:10:04.477470+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:10:04.477470+0000","last_undegraded":"2024-03-21T14:10:04.477470+0000","last_fullsized":"2024-03-21T14:10:04.477470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.654338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.654338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"179'5607","reported_seq":5651,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.680630+0000","last_change":"2024-03-21T14:09:01.656348+0000","last_active":"2024-03-21T14:10:06.680630+0000","last_peered":"2024-03-21T14:10:06.680630+0000","last_clean":"2024-03-21T14:10:06.680630+0000","last_became_active":"2024-03-21T14:09:01.656033+0000","last_became_peered":"2024-03-21T14:09:01.656033+0000","last_unstale":"2024-03-21T14:10:06.680630+0000","last_undegraded":"2024-03-21T14:10:06.680630+0000","last_fullsized":"2024-03-21T14:10:06.680630+0000","mapping_epoch":162,"log_start":"167'1200","ondisk_log_start":"167'1200","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4407,"log_dups_size":0,"ondisk_log_size":4407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.312139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367460352,"num_objects":5607,"num_object_clones":0,"num_object_copies":16821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5607,"num_write_kb":358848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":261,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.490518+0000","last_change":"2024-03-21T14:10:00.794766+0000","last_active":"2024-03-21T14:10:04.490518+0000","last_peered":"2024-03-21T14:10:04.490518+0000","last_clean":"2024-03-21T14:10:04.490518+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:10:04.490518+0000","last_undegraded":"2024-03-21T14:10:04.490518+0000","last_fullsized":"2024-03-21T14:10:04.490518+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:00.794637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:00.794637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.227279+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491436+0000","last_change":"2024-03-21T14:09:59.812788+0000","last_active":"2024-03-21T14:10:04.491436+0000","last_peered":"2024-03-21T14:10:04.491436+0000","last_clean":"2024-03-21T14:10:04.491436+0000","last_became_active":"2024-03-21T14:07:23.362137+0000","last_became_peered":"2024-03-21T14:07:23.362137+0000","last_unstale":"2024-03-21T14:10:04.491436+0000","last_undegraded":"2024-03-21T14:10:04.491436+0000","last_fullsized":"2024-03-21T14:10:04.491436+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.812533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.812533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:07.401150+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":732,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.216272+0000","last_change":"2024-03-21T14:10:06.216315+0000","last_active":"2024-03-21T14:10:06.216272+0000","last_peered":"2024-03-21T14:10:06.216272+0000","last_clean":"2024-03-21T14:10:06.216272+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:10:06.216272+0000","last_undegraded":"2024-03-21T14:10:06.216272+0000","last_fullsized":"2024-03-21T14:10:06.216272+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:10:06.216180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.216180+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:32.696552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":169,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.498606+0000","last_change":"2024-03-21T14:08:31.177524+0000","last_active":"2024-03-21T14:10:04.498606+0000","last_peered":"2024-03-21T14:10:04.498606+0000","last_clean":"2024-03-21T14:10:04.498606+0000","last_became_active":"2024-03-21T14:07:23.363490+0000","last_became_peered":"2024-03-21T14:07:23.363490+0000","last_unstale":"2024-03-21T14:10:04.498606+0000","last_undegraded":"2024-03-21T14:10:04.498606+0000","last_fullsized":"2024-03-21T14:10:04.498606+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:08:31.177430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.177430+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471517+0000","last_change":"2024-03-21T14:09:12.000023+0000","last_active":"2024-03-21T14:10:04.471517+0000","last_peered":"2024-03-21T14:10:04.471517+0000","last_clean":"2024-03-21T14:10:04.471517+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:10:04.471517+0000","last_undegraded":"2024-03-21T14:10:04.471517+0000","last_fullsized":"2024-03-21T14:10:04.471517+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.999855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.999855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:12.909535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"179'5661","reported_seq":5705,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.731195+0000","last_change":"2024-03-21T14:09:01.657541+0000","last_active":"2024-03-21T14:10:06.731195+0000","last_peered":"2024-03-21T14:10:06.731195+0000","last_clean":"2024-03-21T14:10:06.731195+0000","last_became_active":"2024-03-21T14:09:01.656838+0000","last_became_peered":"2024-03-21T14:09:01.656838+0000","last_unstale":"2024-03-21T14:10:06.731195+0000","last_undegraded":"2024-03-21T14:10:06.731195+0000","last_fullsized":"2024-03-21T14:10:06.731195+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.830747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370999296,"num_objects":5661,"num_object_clones":0,"num_object_copies":16983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5661,"num_write_kb":362304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":261,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.491273+0000","last_change":"2024-03-21T14:09:25.416918+0000","last_active":"2024-03-21T14:10:04.491273+0000","last_peered":"2024-03-21T14:10:04.491273+0000","last_clean":"2024-03-21T14:10:04.491273+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:10:04.491273+0000","last_undegraded":"2024-03-21T14:10:04.491273+0000","last_fullsized":"2024-03-21T14:10:04.491273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.416532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.416532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:45.771830+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.490720+0000","last_change":"2024-03-21T14:09:23.381952+0000","last_active":"2024-03-21T14:10:04.490720+0000","last_peered":"2024-03-21T14:10:04.490720+0000","last_clean":"2024-03-21T14:10:04.490720+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:10:04.490720+0000","last_undegraded":"2024-03-21T14:10:04.490720+0000","last_fullsized":"2024-03-21T14:10:04.490720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.381813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:28.987847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":314,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.480671+0000","last_change":"2024-03-21T14:10:01.815306+0000","last_active":"2024-03-21T14:10:04.480671+0000","last_peered":"2024-03-21T14:10:04.480671+0000","last_clean":"2024-03-21T14:10:04.480671+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:10:04.480671+0000","last_undegraded":"2024-03-21T14:10:04.480671+0000","last_fullsized":"2024-03-21T14:10:04.480671+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:10:01.815146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.815146+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:13.735644+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481611+0000","last_change":"2024-03-21T14:08:32.024468+0000","last_active":"2024-03-21T14:10:04.481611+0000","last_peered":"2024-03-21T14:10:04.481611+0000","last_clean":"2024-03-21T14:10:04.481611+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:10:04.481611+0000","last_undegraded":"2024-03-21T14:10:04.481611+0000","last_fullsized":"2024-03-21T14:10:04.481611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.024362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"179'5868","reported_seq":5912,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.830181+0000","last_change":"2024-03-21T14:09:01.657966+0000","last_active":"2024-03-21T14:10:06.830181+0000","last_peered":"2024-03-21T14:10:06.830181+0000","last_clean":"2024-03-21T14:10:06.830181+0000","last_became_active":"2024-03-21T14:09:01.657572+0000","last_became_peered":"2024-03-21T14:09:01.657572+0000","last_unstale":"2024-03-21T14:10:06.830181+0000","last_undegraded":"2024-03-21T14:10:06.830181+0000","last_fullsized":"2024-03-21T14:10:06.830181+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384565248,"num_objects":5868,"num_object_clones":0,"num_object_copies":17604,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5868,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5868,"num_write_kb":375552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":268,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481622+0000","last_change":"2024-03-21T14:09:54.876245+0000","last_active":"2024-03-21T14:10:04.481622+0000","last_peered":"2024-03-21T14:10:04.481622+0000","last_clean":"2024-03-21T14:10:04.481622+0000","last_became_active":"2024-03-21T14:08:17.136362+0000","last_became_peered":"2024-03-21T14:08:17.136362+0000","last_unstale":"2024-03-21T14:10:04.481622+0000","last_undegraded":"2024-03-21T14:10:04.481622+0000","last_fullsized":"2024-03-21T14:10:04.481622+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:54.875993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:54.875993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:08.671071+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471495+0000","last_change":"2024-03-21T14:09:05.122336+0000","last_active":"2024-03-21T14:10:04.471495+0000","last_peered":"2024-03-21T14:10:04.471495+0000","last_clean":"2024-03-21T14:10:04.471495+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:10:04.471495+0000","last_undegraded":"2024-03-21T14:10:04.471495+0000","last_fullsized":"2024-03-21T14:10:04.471495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:05.122232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.122232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:32.899878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"152'15","reported_seq":297,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.490961+0000","last_change":"2024-03-21T14:10:01.771055+0000","last_active":"2024-03-21T14:10:04.490961+0000","last_peered":"2024-03-21T14:10:04.490961+0000","last_clean":"2024-03-21T14:10:04.490961+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:10:04.490961+0000","last_undegraded":"2024-03-21T14:10:04.490961+0000","last_fullsized":"2024-03-21T14:10:04.490961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'15","last_scrub_stamp":"2024-03-21T14:10:01.770915+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.770915+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.570180+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":365,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.480872+0000","last_change":"2024-03-21T14:09:56.813911+0000","last_active":"2024-03-21T14:10:04.480872+0000","last_peered":"2024-03-21T14:10:04.480872+0000","last_clean":"2024-03-21T14:10:04.480872+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:10:04.480872+0000","last_undegraded":"2024-03-21T14:10:04.480872+0000","last_fullsized":"2024-03-21T14:10:04.480872+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:56.813768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.813768+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.384070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"179'5671","reported_seq":5715,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.335956+0000","last_change":"2024-03-21T14:09:01.658726+0000","last_active":"2024-03-21T14:10:06.335956+0000","last_peered":"2024-03-21T14:10:06.335956+0000","last_clean":"2024-03-21T14:10:06.335956+0000","last_became_active":"2024-03-21T14:09:01.656735+0000","last_became_peered":"2024-03-21T14:09:01.656735+0000","last_unstale":"2024-03-21T14:10:06.335956+0000","last_undegraded":"2024-03-21T14:10:06.335956+0000","last_fullsized":"2024-03-21T14:10:06.335956+0000","mapping_epoch":162,"log_start":"166'600","ondisk_log_start":"166'600","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5071,"log_dups_size":0,"ondisk_log_size":5071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.297551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371654656,"num_objects":5671,"num_object_clones":0,"num_object_copies":17013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5671,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5671,"num_write_kb":362944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475502+0000","last_change":"2024-03-21T14:09:03.463520+0000","last_active":"2024-03-21T14:10:04.475502+0000","last_peered":"2024-03-21T14:10:04.475502+0000","last_clean":"2024-03-21T14:10:04.475502+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:10:04.475502+0000","last_undegraded":"2024-03-21T14:10:04.475502+0000","last_fullsized":"2024-03-21T14:10:04.475502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:03.463400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.463400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:33.088597+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":130,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.498452+0000","last_change":"2024-03-21T14:09:24.419269+0000","last_active":"2024-03-21T14:10:04.498452+0000","last_peered":"2024-03-21T14:10:04.498452+0000","last_clean":"2024-03-21T14:10:04.498452+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:10:04.498452+0000","last_undegraded":"2024-03-21T14:10:04.498452+0000","last_fullsized":"2024-03-21T14:10:04.498452+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:24.418831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.418831+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:46.285189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":282,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463358+0000","last_change":"2024-03-21T14:08:32.898925+0000","last_active":"2024-03-21T14:10:04.463358+0000","last_peered":"2024-03-21T14:10:04.463358+0000","last_clean":"2024-03-21T14:10:04.463358+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:10:04.463358+0000","last_undegraded":"2024-03-21T14:10:04.463358+0000","last_fullsized":"2024-03-21T14:10:04.463358+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:08:32.898753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.898753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":357,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471650+0000","last_change":"2024-03-21T14:09:09.098309+0000","last_active":"2024-03-21T14:10:04.471650+0000","last_peered":"2024-03-21T14:10:04.471650+0000","last_clean":"2024-03-21T14:10:04.471650+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:10:04.471650+0000","last_undegraded":"2024-03-21T14:10:04.471650+0000","last_fullsized":"2024-03-21T14:10:04.471650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:09.098187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.098187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.235333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"179'5814","reported_seq":5858,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.746230+0000","last_change":"2024-03-21T14:09:01.657521+0000","last_active":"2024-03-21T14:10:06.746230+0000","last_peered":"2024-03-21T14:10:06.746230+0000","last_clean":"2024-03-21T14:10:06.746230+0000","last_became_active":"2024-03-21T14:09:01.657075+0000","last_became_peered":"2024-03-21T14:09:01.657075+0000","last_unstale":"2024-03-21T14:10:06.746230+0000","last_undegraded":"2024-03-21T14:10:06.746230+0000","last_fullsized":"2024-03-21T14:10:06.746230+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5814,"log_dups_size":0,"ondisk_log_size":5814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:15.497337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381026304,"num_objects":5814,"num_object_clones":0,"num_object_copies":17442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5814,"num_write_kb":372096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":259,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463682+0000","last_change":"2024-03-21T14:08:31.865055+0000","last_active":"2024-03-21T14:10:04.463682+0000","last_peered":"2024-03-21T14:10:04.463682+0000","last_clean":"2024-03-21T14:10:04.463682+0000","last_became_active":"2024-03-21T14:07:23.364265+0000","last_became_peered":"2024-03-21T14:07:23.364265+0000","last_unstale":"2024-03-21T14:10:04.463682+0000","last_undegraded":"2024-03-21T14:10:04.463682+0000","last_fullsized":"2024-03-21T14:10:04.463682+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:31.864937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.864937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":266,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.472641+0000","last_change":"2024-03-21T14:08:26.921914+0000","last_active":"2024-03-21T14:10:04.472641+0000","last_peered":"2024-03-21T14:10:04.472641+0000","last_clean":"2024-03-21T14:10:04.472641+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:10:04.472641+0000","last_undegraded":"2024-03-21T14:10:04.472641+0000","last_fullsized":"2024-03-21T14:10:04.472641+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:26.921568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:36.591844+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.921568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":220,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.475584+0000","last_change":"2024-03-21T14:08:20.594480+0000","last_active":"2024-03-21T14:10:04.475584+0000","last_peered":"2024-03-21T14:10:04.475584+0000","last_clean":"2024-03-21T14:10:04.475584+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:10:04.475584+0000","last_undegraded":"2024-03-21T14:10:04.475584+0000","last_fullsized":"2024-03-21T14:10:04.475584+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:08:20.594343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:20.594343+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":249,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463138+0000","last_change":"2024-03-21T14:08:16.845392+0000","last_active":"2024-03-21T14:10:04.463138+0000","last_peered":"2024-03-21T14:10:04.463138+0000","last_clean":"2024-03-21T14:10:04.463138+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:10:04.463138+0000","last_undegraded":"2024-03-21T14:10:04.463138+0000","last_fullsized":"2024-03-21T14:10:04.463138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.845286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.845286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"179'5735","reported_seq":5779,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.654995+0000","last_change":"2024-03-21T14:09:01.452701+0000","last_active":"2024-03-21T14:10:06.654995+0000","last_peered":"2024-03-21T14:10:06.654995+0000","last_clean":"2024-03-21T14:10:06.654995+0000","last_became_active":"2024-03-21T14:09:01.452387+0000","last_became_peered":"2024-03-21T14:09:01.452387+0000","last_unstale":"2024-03-21T14:10:06.654995+0000","last_undegraded":"2024-03-21T14:10:06.654995+0000","last_fullsized":"2024-03-21T14:10:06.654995+0000","mapping_epoch":162,"log_start":"167'1300","ondisk_log_start":"167'1300","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.296270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375848960,"num_objects":5735,"num_object_clones":0,"num_object_copies":17205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5735,"num_write_kb":367040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471154+0000","last_change":"2024-03-21T14:09:57.384748+0000","last_active":"2024-03-21T14:10:04.471154+0000","last_peered":"2024-03-21T14:10:04.471154+0000","last_clean":"2024-03-21T14:10:04.471154+0000","last_became_active":"2024-03-21T14:08:17.203307+0000","last_became_peered":"2024-03-21T14:08:17.203307+0000","last_unstale":"2024-03-21T14:10:04.471154+0000","last_undegraded":"2024-03-21T14:10:04.471154+0000","last_fullsized":"2024-03-21T14:10:04.471154+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:57.384552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:09:57.384552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:23.621828+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":356,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.465884+0000","last_change":"2024-03-21T14:08:26.932388+0000","last_active":"2024-03-21T14:10:04.465884+0000","last_peered":"2024-03-21T14:10:04.465884+0000","last_clean":"2024-03-21T14:10:04.465884+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:10:04.465884+0000","last_undegraded":"2024-03-21T14:10:04.465884+0000","last_fullsized":"2024-03-21T14:10:04.465884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:08:26.932274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.932274+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":119,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.492199+0000","last_change":"2024-03-21T14:09:13.247866+0000","last_active":"2024-03-21T14:10:04.492199+0000","last_peered":"2024-03-21T14:10:04.492199+0000","last_clean":"2024-03-21T14:10:04.492199+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:10:04.492199+0000","last_undegraded":"2024-03-21T14:10:04.492199+0000","last_fullsized":"2024-03-21T14:10:04.492199+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:13.247726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:13.247726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.028509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":294,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.498756+0000","last_change":"2024-03-21T14:09:58.762917+0000","last_active":"2024-03-21T14:10:04.498756+0000","last_peered":"2024-03-21T14:10:04.498756+0000","last_clean":"2024-03-21T14:10:04.498756+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:10:04.498756+0000","last_undegraded":"2024-03-21T14:10:04.498756+0000","last_fullsized":"2024-03-21T14:10:04.498756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:09:58.762639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.762639+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.152458+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"179'5709","reported_seq":5753,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.280422+0000","last_change":"2024-03-21T14:09:01.451050+0000","last_active":"2024-03-21T14:10:06.280422+0000","last_peered":"2024-03-21T14:10:06.280422+0000","last_clean":"2024-03-21T14:10:06.280422+0000","last_became_active":"2024-03-21T14:09:01.450729+0000","last_became_peered":"2024-03-21T14:09:01.450729+0000","last_unstale":"2024-03-21T14:10:06.280422+0000","last_undegraded":"2024-03-21T14:10:06.280422+0000","last_fullsized":"2024-03-21T14:10:06.280422+0000","mapping_epoch":162,"log_start":"167'1000","ondisk_log_start":"167'1000","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.279375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374145024,"num_objects":5709,"num_object_clones":0,"num_object_copies":17127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5709,"num_write_kb":365376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":254,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.478110+0000","last_change":"2024-03-21T14:10:03.673038+0000","last_active":"2024-03-21T14:10:04.478110+0000","last_peered":"2024-03-21T14:10:04.478110+0000","last_clean":"2024-03-21T14:10:04.478110+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:10:04.478110+0000","last_undegraded":"2024-03-21T14:10:04.478110+0000","last_fullsized":"2024-03-21T14:10:04.478110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.672903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.672903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.596347+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":257,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471487+0000","last_change":"2024-03-21T14:09:16.047979+0000","last_active":"2024-03-21T14:10:04.471487+0000","last_peered":"2024-03-21T14:10:04.471487+0000","last_clean":"2024-03-21T14:10:04.471487+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:10:04.471487+0000","last_undegraded":"2024-03-21T14:10:04.471487+0000","last_fullsized":"2024-03-21T14:10:04.471487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.047852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.047852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.726094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":273,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.463212+0000","last_change":"2024-03-21T14:10:02.232967+0000","last_active":"2024-03-21T14:10:04.463212+0000","last_peered":"2024-03-21T14:10:04.463212+0000","last_clean":"2024-03-21T14:10:04.463212+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:10:04.463212+0000","last_undegraded":"2024-03-21T14:10:04.463212+0000","last_fullsized":"2024-03-21T14:10:04.463212+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.232818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.232818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.637769+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":280,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.465826+0000","last_change":"2024-03-21T14:09:03.147020+0000","last_active":"2024-03-21T14:10:04.465826+0000","last_peered":"2024-03-21T14:10:04.465826+0000","last_clean":"2024-03-21T14:10:04.465826+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:10:04.465826+0000","last_undegraded":"2024-03-21T14:10:04.465826+0000","last_fullsized":"2024-03-21T14:10:04.465826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:03.146877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.146877+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:05.861799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"179'5650","reported_seq":5694,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.652949+0000","last_change":"2024-03-21T14:09:01.659005+0000","last_active":"2024-03-21T14:10:06.652949+0000","last_peered":"2024-03-21T14:10:06.652949+0000","last_clean":"2024-03-21T14:10:06.652949+0000","last_became_active":"2024-03-21T14:09:01.656962+0000","last_became_peered":"2024-03-21T14:09:01.656962+0000","last_unstale":"2024-03-21T14:10:06.652949+0000","last_undegraded":"2024-03-21T14:10:06.652949+0000","last_fullsized":"2024-03-21T14:10:06.652949+0000","mapping_epoch":162,"log_start":"167'1300","ondisk_log_start":"167'1300","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4350,"log_dups_size":0,"ondisk_log_size":4350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.506609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370278400,"num_objects":5650,"num_object_clones":0,"num_object_copies":16950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5650,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5650,"num_write_kb":361600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":271,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.486490+0000","last_change":"2024-03-21T14:10:03.740909+0000","last_active":"2024-03-21T14:10:04.486490+0000","last_peered":"2024-03-21T14:10:04.486490+0000","last_clean":"2024-03-21T14:10:04.486490+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:10:04.486490+0000","last_undegraded":"2024-03-21T14:10:04.486490+0000","last_fullsized":"2024-03-21T14:10:04.486490+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:10:03.740783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.740783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:12.932320+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.490858+0000","last_change":"2024-03-21T14:09:16.366776+0000","last_active":"2024-03-21T14:10:04.490858+0000","last_peered":"2024-03-21T14:10:04.490858+0000","last_clean":"2024-03-21T14:10:04.490858+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:10:04.490858+0000","last_undegraded":"2024-03-21T14:10:04.490858+0000","last_fullsized":"2024-03-21T14:10:04.490858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.366566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.366566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.932561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":265,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.481986+0000","last_change":"2024-03-21T14:08:24.037320+0000","last_active":"2024-03-21T14:10:04.481986+0000","last_peered":"2024-03-21T14:10:04.481986+0000","last_clean":"2024-03-21T14:10:04.481986+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:10:04.481986+0000","last_undegraded":"2024-03-21T14:10:04.481986+0000","last_fullsized":"2024-03-21T14:10:04.481986+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:24.037202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:24.037202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":289,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464789+0000","last_change":"2024-03-21T14:09:04.171171+0000","last_active":"2024-03-21T14:10:04.464789+0000","last_peered":"2024-03-21T14:10:04.464789+0000","last_clean":"2024-03-21T14:10:04.464789+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:10:04.464789+0000","last_undegraded":"2024-03-21T14:10:04.464789+0000","last_fullsized":"2024-03-21T14:10:04.464789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:09:04.171069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.171069+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:26.513971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"179'5715","reported_seq":5759,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:06.715191+0000","last_change":"2024-03-21T14:09:01.451288+0000","last_active":"2024-03-21T14:10:06.715191+0000","last_peered":"2024-03-21T14:10:06.715191+0000","last_clean":"2024-03-21T14:10:06.715191+0000","last_became_active":"2024-03-21T14:09:01.451020+0000","last_became_peered":"2024-03-21T14:09:01.451020+0000","last_unstale":"2024-03-21T14:10:06.715191+0000","last_undegraded":"2024-03-21T14:10:06.715191+0000","last_fullsized":"2024-03-21T14:10:06.715191+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.778288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374538240,"num_objects":5715,"num_object_clones":0,"num_object_copies":17145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5715,"num_write_kb":365760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":253,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.465943+0000","last_change":"2024-03-21T14:09:09.117631+0000","last_active":"2024-03-21T14:10:04.465943+0000","last_peered":"2024-03-21T14:10:04.465943+0000","last_clean":"2024-03-21T14:10:04.465943+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:10:04.465943+0000","last_undegraded":"2024-03-21T14:10:04.465943+0000","last_fullsized":"2024-03-21T14:10:04.465943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.117496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.117496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.877971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":366,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.473370+0000","last_change":"2024-03-21T14:09:10.904857+0000","last_active":"2024-03-21T14:10:04.473370+0000","last_peered":"2024-03-21T14:10:04.473370+0000","last_clean":"2024-03-21T14:10:04.473370+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:10:04.473370+0000","last_undegraded":"2024-03-21T14:10:04.473370+0000","last_fullsized":"2024-03-21T14:10:04.473370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:09:10.904772+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.904772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.436321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":272,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.471637+0000","last_change":"2024-03-21T14:09:19.247854+0000","last_active":"2024-03-21T14:10:04.471637+0000","last_peered":"2024-03-21T14:10:04.471637+0000","last_clean":"2024-03-21T14:10:04.471637+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:10:04.471637+0000","last_undegraded":"2024-03-21T14:10:04.471637+0000","last_fullsized":"2024-03-21T14:10:04.471637+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.247746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.247746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.335818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"167'15","reported_seq":281,"reported_epoch":179,"state":"active+clean","last_fresh":"2024-03-21T14:10:04.464262+0000","last_change":"2024-03-21T14:09:22.120079+0000","last_active":"2024-03-21T14:10:04.464262+0000","last_peered":"2024-03-21T14:10:04.464262+0000","last_clean":"2024-03-21T14:10:04.464262+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:10:04.464262+0000","last_undegraded":"2024-03-21T14:10:04.464262+0000","last_fullsized":"2024-03-21T14:10:04.464262+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'15","last_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:09:22.119861+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:22.576029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5957615616,"num_objects":90906,"num_object_clones":0,"num_object_copies":272718,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90906,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":90906,"num_write_kb":5817984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17869570048,"data_stored":17869570048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82106,"ondisk_log_size":82106,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103263,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1991344,"kb_used_data":1811624,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91753808,"statfs":{"total":95995035648,"available":93955899392,"internally_reserved":0,"allocated":1855102976,"data_stored":1854641169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430308,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1968788,"kb_used_data":1789064,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":91776364,"statfs":{"total":95995035648,"available":93978996736,"internally_reserved":0,"allocated":1832001536,"data_stored":1831570378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":184023494},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2750720,"kb_used_data":2507128,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":90994432,"statfs":{"total":95995035648,"available":93178298368,"internally_reserved":0,"allocated":2567299072,"data_stored":2566821188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2080708,"kb_used_data":1828796,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":91664444,"statfs":{"total":95995035648,"available":93864390656,"internally_reserved":0,"allocated":1872687104,"data_stored":1872267796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411449,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2769996,"kb_used_data":2544452,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":90975156,"statfs":{"total":95995035648,"available":93158559744,"internally_reserved":0,"allocated":2605518848,"data_stored":2605091370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182515,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2116132,"kb_used_data":1832280,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":91629020,"statfs":{"total":95995035648,"available":93828116480,"internally_reserved":0,"allocated":1876254720,"data_stored":1875807314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":290650567},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018991,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1978984,"kb_used_data":1813152,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91766168,"statfs":{"total":95995035648,"available":93968556032,"internally_reserved":0,"allocated":1856667648,"data_stored":1856212204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691946,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3415848,"kb_used_data":3240284,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":90329304,"statfs":{"total":95995035648,"available":92497207296,"internally_reserved":0,"allocated":3318050816,"data_stored":3317600229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1873346560,"data_stored":1873346560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2597519360,"data_stored":2597519360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2607284224,"data_stored":2607284224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1877213184,"data_stored":1877213184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1857355776,"data_stored":1857355776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3355705344,"data_stored":3355705344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1844314112,"data_stored":1844314112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1856831488,"data_stored":1856831488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:10:11.896 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-21T14:10:11.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 96545 96529 102.235 56.0625 0.0347631 0.00977646 2024-03-21T14:10:11.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:11.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:11.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:11.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 8.5 scrub starts 2024-03-21T14:10:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 3.7 deep-scrub starts 2024-03-21T14:10:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 3.7 deep-scrub ok 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 8.5 scrub starts 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 3.7 deep-scrub starts 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 3.7 deep-scrub ok 2024-03-21T14:10:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:10:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:10:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:10:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:10:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:10:12.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:11 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:10:12.073 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:12.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 8.5 scrub starts 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 3.7 deep-scrub starts 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 3.7 deep-scrub ok 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:10:12.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:11 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:10:12.149 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:12.167 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:10:12.213 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:12.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.263 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:12.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:12.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.405 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:12.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:12.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:12.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:12.588 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:10:12.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:12.683 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:12.784 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:12.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:12.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:12.892 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:10:12.893644+0000 min lat: 0.00121156 max lat: 0.128614 avg lat: 0.00983144 2024-03-21T14:10:12.893 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:10:12.893 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 15 97635 97620 101.667 68.1875 0.0197931 0.00983144 2024-03-21T14:10:12.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:12.964 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:12.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:12.987 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:12 smithi118 ceph-mon[33862]: pgmap v314: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s 2024-03-21T14:10:12.987 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:12 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:10:12.987 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:12 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:10:12.987 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:12 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:10:12.987 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:12 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[29576]: pgmap v314: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[40510]: pgmap v314: 148 pgs: 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 959 op/s 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:10:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:10:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:10:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:12 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:10:13.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:13.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:13.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:13.243 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:13.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:13.345 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.381 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:13.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:13.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:13.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:13.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:13.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:13.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:13.792 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:10:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:13.893 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 98791 98775 101.184 72.1875 0.00753987 0.0098783 2024-03-21T14:10:13.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:13.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:13.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:14.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:10:14.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:10:14.025 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 8.5 scrub ok 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/914571579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:10:14.026 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:13 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:10:14.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:14.068 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:14.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 8.5 scrub ok 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/914571579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:10:14.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 8.5 scrub ok 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/914571579' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:10:14.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:13 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:10:14.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:14.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:14.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.368 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:14.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:14.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:14.507 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:10:14.507 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:99483 8.c e181/162 8:3659558a:::benchmark_data_smithi003_110242_object99482:head v 181'6321, mlcod=181'6318)", 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:10:14.498784+0000", 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0072961429999999997, 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073291600000000004, 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.498784+0000", 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.508 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.498784+0000", 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.498785+0000", 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3820000000000001e-06 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.498823+0000", 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7809999999999999e-05 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.509 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.498841+0000", 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8227e-05 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.498851+0000", 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7489999999999993e-06 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.499057+0000", 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020625799999999999 2024-03-21T14:10:14.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.499111+0000", 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4014000000000002e-05 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:99486 8.8 e181/162 8:13c058c6:::benchmark_data_smithi003_110242_object99485:head v 181'6313, mlcod=181'6311)", 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:10:14.500763+0000", 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0053170559999999997, 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054244089999999998, 2024-03-21T14:10:14.511 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.500763+0000", 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.500763+0000", 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:14.512 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.500764+0000", 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1429999999999999e-06 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.500791+0000", 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7407999999999999e-05 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.500808+0000", 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7187999999999999e-05 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.513 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.500819+0000", 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1022000000000001e-05 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501026+0000", 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020657600000000001 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501136+0000", 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109701 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:14.514 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:99487 8.6 e181/162 8:6380224e:::benchmark_data_smithi003_110242_object99486:head v 181'6148, mlcod=181'6145)", 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:10:14.501214+0000", 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0048655440000000003, 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005041635, 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501214+0000", 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.515 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501214+0000", 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501215+0000", 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6799999999999999e-07 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501265+0000", 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9685000000000003e-05 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.516 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501278+0000", 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3538e-05 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.501296+0000", 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7544999999999999e-05 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.502928+0000", 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016319049999999999 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.517 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.502954+0000", 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.671e-05 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15132.0:99488 8.d e181/162 8:b9747cdc:::benchmark_data_smithi003_110242_object99487:head v 181'6224, mlcod=181'6222)", 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:10:14.504174+0000", 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0019053830000000001, 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021466940000000002, 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:10:14.518 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504174+0000", 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504174+0000", 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504175+0000", 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.08e-06 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504199+0000", 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3476999999999999e-05 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504211+0000", 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2519e-05 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504219+0000", 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5209999999999997e-06 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.520 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504446+0000", 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227271 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:10:14.504492+0000", 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5463999999999997e-05 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 4 2024-03-21T14:10:14.521 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:10:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.523 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:14.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:14.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:14.662 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:14.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:14.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:14.804 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:14.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:14.900 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 99908 99892 100.673 69.8125 0.0254396 0.0099283 2024-03-21T14:10:14.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:14.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:14.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:14.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:15.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: pgmap v315: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: 6.13 deep-scrub starts 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: osdmap e181: 8 total, 8 up, 8 in 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: 6.13 deep-scrub ok 2024-03-21T14:10:15.042 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: pgmap v315: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: 6.13 deep-scrub starts 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: osdmap e181: 8 total, 8 up, 8 in 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: 6.13 deep-scrub ok 2024-03-21T14:10:15.043 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:14 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:10:15.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:15.103 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: pgmap v315: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 79 MiB/s wr, 1.27k op/s 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: 6.13 deep-scrub starts 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: osdmap e181: 8 total, 8 up, 8 in 2024-03-21T14:10:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: 6.13 deep-scrub ok 2024-03-21T14:10:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:14 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:10:15.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:15.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.242 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:15.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:15.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:10:15.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:15.409 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:15.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:15.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:15.552 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:15.654 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:15.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:15.690 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:10:15.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:15.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:15.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 100950 100934 100.108 65.125 0.0109334 0.00998427 2024-03-21T14:10:15.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:15.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:15.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:15.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:16.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:10:16.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:15 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:10:16.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:15 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:10:16.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:16.085 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:10:16.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:15 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:10:16.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:15 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:10:16.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:10:16.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:15 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:10:16.086 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:15 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:10:16.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:16.190 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:16.223 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:16.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:16.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:16.452 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:16.485 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:16.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:16.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:16.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:16.748 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:10:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:10:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:10:16.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:16.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:16.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:16.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 101961 101945 99.5316 63.1875 0.0262549 0.010042 2024-03-21T14:10:16.897 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:10:16.897 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:10:16.897 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-21T14:10:16.897 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-21T14:10:16.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:10:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:16 smithi118 ceph-mon[33862]: pgmap v317: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 666 op/s 2024-03-21T14:10:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:16.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:10:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:10:16.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:17.027 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:17.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:17.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:17.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:17.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:17.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:16 smithi003 ceph-mon[29576]: pgmap v317: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 666 op/s 2024-03-21T14:10:17.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:17.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:16 smithi003 ceph-mon[40510]: pgmap v317: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 42 MiB/s wr, 666 op/s 2024-03-21T14:10:17.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:17.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.338 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:17.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:17.439 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:17.478 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:17.579 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:17.612 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:17.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.758 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:17.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:17.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:17.892 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:17.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:17.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 103085 103069 99.081 70.25 0.0063766 0.0100881 2024-03-21T14:10:17.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:18.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:18.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:18.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:10:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:18.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:18.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:18.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:18.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:18.349 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:10:18.349 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:10:18.349 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:10:18.349 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:10:18.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:18.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:18.383 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:10:18.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:10:18.430 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:18.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:18.463 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:10:18.504 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:10:18.511 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:10:18.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:18.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:18.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:18.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:18.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:18.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:18.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 104064 104048 98.5069 61.1875 0.0113305 0.0101467 2024-03-21T14:10:18.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:18.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:18.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:18 smithi003 ceph-mon[29576]: pgmap v318: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-21T14:10:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:18 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4033845656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:19.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:18 smithi003 ceph-mon[40510]: pgmap v318: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-21T14:10:19.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:18 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4033845656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:19.018 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:10:19.019 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:10:19.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:19.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:18 smithi118 ceph-mon[33862]: pgmap v318: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-21T14:10:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:18 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4033845656' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:19.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:19.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:19.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:19.360 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:19.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:19.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:19.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:19.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:19.643 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:19.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:19.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:19.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:19.922 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 105285 105269 98.1756 76.3125 0.0138668 0.010181 2024-03-21T14:10:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:19.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:20.018 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:20.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:20.095 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:20.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:19 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:10:20.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:19 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:10:20.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:19 smithi003 ceph-mon[40510]: osdmap e182: 8 total, 8 up, 8 in 2024-03-21T14:10:20.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:19 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:10:20.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:19 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:10:20.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:19 smithi003 ceph-mon[29576]: osdmap e182: 8 total, 8 up, 8 in 2024-03-21T14:10:20.136 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:19 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:10:20.136 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:19 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:10:20.136 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:19 smithi118 ceph-mon[33862]: osdmap e182: 8 total, 8 up, 8 in 2024-03-21T14:10:20.185 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:10:20.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:20.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:20.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:20.339 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:20.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:20.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:20.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:10:20.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:20.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:20.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:20.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:20.896 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 106453 106437 97.8053 73 0.0149567 0.0102192 2024-03-21T14:10:20.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:20.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:20.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:20.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:21.024 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:20 smithi118 ceph-mon[33862]: pgmap v320: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-21T14:10:21.050 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:20 smithi003 ceph-mon[29576]: pgmap v320: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-21T14:10:21.050 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:20 smithi003 ceph-mon[40510]: pgmap v320: 148 pgs: 148 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-21T14:10:21.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:21.102 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:21.204 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:21.237 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:21.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:21.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:21.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:21.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:10:21.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:21.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:21.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:21.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:21.787 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:21.815 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:21.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:21.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 107481 107465 97.3179 64.25 0.0299348 0.0102713 2024-03-21T14:10:21.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:21.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:21.957 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:21.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:22.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.110 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:22.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:22.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:22.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:22.330 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:10:22.334 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:10:22.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:22.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:22.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:22.536 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:22.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:22.638 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:22.668 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:22.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:22.691 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:10:22.691 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:10:22.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:22.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:22.897 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 108492 108476 96.8303 63.1875 0.0182375 0.0103224 2024-03-21T14:10:22.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:22.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:22.943 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:22.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:22 smithi003 ceph-mon[29576]: pgmap v321: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 893 op/s 2024-03-21T14:10:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:22 smithi003 ceph-mon[40510]: pgmap v321: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 893 op/s 2024-03-21T14:10:23.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:23.081 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:22 smithi118 ceph-mon[33862]: pgmap v321: 148 pgs: 148 active+clean; 6.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 893 op/s 2024-03-21T14:10:23.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:23.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:23.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:23.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:23.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:23.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:23.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:23.581 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-03-21T14:10:21.381843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6775380392,"num_objects":103573,"num_object_clones":0,"num_object_copies":310719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103573,"num_whiteouts":0,"num_read":1392,"num_read_kb":1238,"num_write":103990,"num_write_kb":6618461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":2363178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86139,"ondisk_log_size":86139,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21578176,"kb_used_data":19822276,"kb_used_omap":12,"kb_used_meta":1755635,"kb_avail":728383040,"statfs":{"total":767960285184,"available":745864232960,"internally_reserved":0,"allocated":20298010624,"data_stored":20294583265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1797770800},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,7,4,8,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":468058112,"num_objects":7142,"num_object_clones":0,"num_object_copies":21426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7142,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7142,"num_write_kb":457088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002228"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813026+0000","last_change":"2024-03-21T14:09:55.330094+0000","last_active":"2024-03-21T14:10:18.813026+0000","last_peered":"2024-03-21T14:10:18.813026+0000","last_clean":"2024-03-21T14:10:18.813026+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:10:18.813026+0000","last_undegraded":"2024-03-21T14:10:18.813026+0000","last_fullsized":"2024-03-21T14:10:18.813026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.329965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.329965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.949341+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":269,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.789033+0000","last_change":"2024-03-21T14:09:02.129524+0000","last_active":"2024-03-21T14:10:13.789033+0000","last_peered":"2024-03-21T14:10:13.789033+0000","last_clean":"2024-03-21T14:10:13.789033+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:10:13.789033+0000","last_undegraded":"2024-03-21T14:10:13.789033+0000","last_fullsized":"2024-03-21T14:10:13.789033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:02.129441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.129441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.443888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":124,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810903+0000","last_change":"2024-03-21T14:08:26.447065+0000","last_active":"2024-03-21T14:10:18.810903+0000","last_peered":"2024-03-21T14:10:18.810903+0000","last_clean":"2024-03-21T14:10:18.810903+0000","last_became_active":"2024-03-21T14:07:23.349536+0000","last_became_peered":"2024-03-21T14:07:23.349536+0000","last_unstale":"2024-03-21T14:10:18.810903+0000","last_undegraded":"2024-03-21T14:10:18.810903+0000","last_fullsized":"2024-03-21T14:10:18.810903+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:26.446938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.446938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.504269+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":283,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812674+0000","last_change":"2024-03-21T14:09:15.098201+0000","last_active":"2024-03-21T14:10:18.812674+0000","last_peered":"2024-03-21T14:10:18.812674+0000","last_clean":"2024-03-21T14:10:18.812674+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:10:18.812674+0000","last_undegraded":"2024-03-21T14:10:18.812674+0000","last_fullsized":"2024-03-21T14:10:18.812674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:15.098048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.098048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:43.207181+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":133,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.819410+0000","last_change":"2024-03-21T14:09:59.259150+0000","last_active":"2024-03-21T14:10:18.819410+0000","last_peered":"2024-03-21T14:10:18.819410+0000","last_clean":"2024-03-21T14:10:18.819410+0000","last_became_active":"2024-03-21T14:07:23.355776+0000","last_became_peered":"2024-03-21T14:07:23.355776+0000","last_unstale":"2024-03-21T14:10:18.819410+0000","last_undegraded":"2024-03-21T14:10:18.819410+0000","last_fullsized":"2024-03-21T14:10:18.819410+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.259017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.259017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.072409+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812726+0000","last_change":"2024-03-21T14:10:03.258278+0000","last_active":"2024-03-21T14:10:18.812726+0000","last_peered":"2024-03-21T14:10:18.812726+0000","last_clean":"2024-03-21T14:10:18.812726+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:10:18.812726+0000","last_undegraded":"2024-03-21T14:10:18.812726+0000","last_fullsized":"2024-03-21T14:10:18.812726+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.258168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.901138+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794244+0000","last_change":"2024-03-21T14:10:12.759166+0000","last_active":"2024-03-21T14:10:13.794244+0000","last_peered":"2024-03-21T14:10:13.794244+0000","last_clean":"2024-03-21T14:10:13.794244+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:10:13.794244+0000","last_undegraded":"2024-03-21T14:10:13.794244+0000","last_fullsized":"2024-03-21T14:10:13.794244+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:12.758536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.758536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.648088+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":306,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.793689+0000","last_change":"2024-03-21T14:10:06.702481+0000","last_active":"2024-03-21T14:10:13.793689+0000","last_peered":"2024-03-21T14:10:13.793689+0000","last_clean":"2024-03-21T14:10:13.793689+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:10:13.793689+0000","last_undegraded":"2024-03-21T14:10:13.793689+0000","last_fullsized":"2024-03-21T14:10:13.793689+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:10:06.702330+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.702330+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:16.820634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":130,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.782431+0000","last_change":"2024-03-21T14:10:06.813347+0000","last_active":"2024-03-21T14:10:13.782431+0000","last_peered":"2024-03-21T14:10:13.782431+0000","last_clean":"2024-03-21T14:10:13.782431+0000","last_became_active":"2024-03-21T14:07:23.360834+0000","last_became_peered":"2024-03-21T14:07:23.360834+0000","last_unstale":"2024-03-21T14:10:13.782431+0000","last_undegraded":"2024-03-21T14:10:13.782431+0000","last_fullsized":"2024-03-21T14:10:13.782431+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:06.813027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.813027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:24.168419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":263,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810515+0000","last_change":"2024-03-21T14:09:10.043497+0000","last_active":"2024-03-21T14:10:18.810515+0000","last_peered":"2024-03-21T14:10:18.810515+0000","last_clean":"2024-03-21T14:10:18.810515+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:10:18.810515+0000","last_undegraded":"2024-03-21T14:10:18.810515+0000","last_fullsized":"2024-03-21T14:10:18.810515+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.043396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.043396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.977610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810588+0000","last_change":"2024-03-21T14:08:23.436895+0000","last_active":"2024-03-21T14:10:18.810588+0000","last_peered":"2024-03-21T14:10:18.810588+0000","last_clean":"2024-03-21T14:10:18.810588+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:10:18.810588+0000","last_undegraded":"2024-03-21T14:10:18.810588+0000","last_fullsized":"2024-03-21T14:10:18.810588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.436768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.436768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.504346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":275,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813277+0000","last_change":"2024-03-21T14:08:12.828629+0000","last_active":"2024-03-21T14:10:18.813277+0000","last_peered":"2024-03-21T14:10:18.813277+0000","last_clean":"2024-03-21T14:10:18.813277+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:10:18.813277+0000","last_undegraded":"2024-03-21T14:10:18.813277+0000","last_fullsized":"2024-03-21T14:10:18.813277+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:08:12.828494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.828494+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401747+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810243+0000","last_change":"2024-03-21T14:08:10.366349+0000","last_active":"2024-03-21T14:10:18.810243+0000","last_peered":"2024-03-21T14:10:18.810243+0000","last_clean":"2024-03-21T14:10:18.810243+0000","last_became_active":"2024-03-21T14:07:23.350761+0000","last_became_peered":"2024-03-21T14:07:23.350761+0000","last_unstale":"2024-03-21T14:10:18.810243+0000","last_undegraded":"2024-03-21T14:10:18.810243+0000","last_fullsized":"2024-03-21T14:10:18.810243+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.366216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.366216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.504297+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":264,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.819198+0000","last_change":"2024-03-21T14:09:25.498525+0000","last_active":"2024-03-21T14:10:18.819198+0000","last_peered":"2024-03-21T14:10:18.819198+0000","last_clean":"2024-03-21T14:10:18.819198+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:10:18.819198+0000","last_undegraded":"2024-03-21T14:10:18.819198+0000","last_fullsized":"2024-03-21T14:10:18.819198+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.498388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.498388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:43.494502+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810595+0000","last_change":"2024-03-21T14:09:58.390279+0000","last_active":"2024-03-21T14:10:18.810595+0000","last_peered":"2024-03-21T14:10:18.810595+0000","last_clean":"2024-03-21T14:10:18.810595+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:10:18.810595+0000","last_undegraded":"2024-03-21T14:10:18.810595+0000","last_fullsized":"2024-03-21T14:10:18.810595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:58.390161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.390161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.872420+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":277,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813070+0000","last_change":"2024-03-21T14:09:26.168563+0000","last_active":"2024-03-21T14:10:18.813070+0000","last_peered":"2024-03-21T14:10:18.813070+0000","last_clean":"2024-03-21T14:10:18.813070+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:10:18.813070+0000","last_undegraded":"2024-03-21T14:10:18.813070+0000","last_fullsized":"2024-03-21T14:10:18.813070+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:09:26.168400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.168400+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.775561+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813432+0000","last_change":"2024-03-21T14:10:11.357570+0000","last_active":"2024-03-21T14:10:18.813432+0000","last_peered":"2024-03-21T14:10:18.813432+0000","last_clean":"2024-03-21T14:10:18.813432+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:10:18.813432+0000","last_undegraded":"2024-03-21T14:10:18.813432+0000","last_fullsized":"2024-03-21T14:10:18.813432+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.357287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.357287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.442137+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":269,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783889+0000","last_change":"2024-03-21T14:10:03.746595+0000","last_active":"2024-03-21T14:10:13.783889+0000","last_peered":"2024-03-21T14:10:13.783889+0000","last_clean":"2024-03-21T14:10:13.783889+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:10:13.783889+0000","last_undegraded":"2024-03-21T14:10:13.783889+0000","last_fullsized":"2024-03-21T14:10:13.783889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.746480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.746480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.495837+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":124,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812977+0000","last_change":"2024-03-21T14:09:19.179886+0000","last_active":"2024-03-21T14:10:18.812977+0000","last_peered":"2024-03-21T14:10:18.812977+0000","last_clean":"2024-03-21T14:10:18.812977+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:10:18.812977+0000","last_undegraded":"2024-03-21T14:10:18.812977+0000","last_fullsized":"2024-03-21T14:10:18.812977+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.179692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.179692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:38.891081+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":307,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.791981+0000","last_change":"2024-03-21T14:09:26.954314+0000","last_active":"2024-03-21T14:10:13.791981+0000","last_peered":"2024-03-21T14:10:13.791981+0000","last_clean":"2024-03-21T14:10:13.791981+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:10:13.791981+0000","last_undegraded":"2024-03-21T14:10:13.791981+0000","last_fullsized":"2024-03-21T14:10:13.791981+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:09:26.954184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.954184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.810324+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.795521+0000","last_change":"2024-03-21T14:09:04.145571+0000","last_active":"2024-03-21T14:10:13.795521+0000","last_peered":"2024-03-21T14:10:13.795521+0000","last_clean":"2024-03-21T14:10:13.795521+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:10:13.795521+0000","last_undegraded":"2024-03-21T14:10:13.795521+0000","last_fullsized":"2024-03-21T14:10:13.795521+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.145471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.145471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.785029+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":263,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813167+0000","last_change":"2024-03-21T14:10:04.249646+0000","last_active":"2024-03-21T14:10:18.813167+0000","last_peered":"2024-03-21T14:10:18.813167+0000","last_clean":"2024-03-21T14:10:18.813167+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:10:18.813167+0000","last_undegraded":"2024-03-21T14:10:18.813167+0000","last_fullsized":"2024-03-21T14:10:18.813167+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:04.249513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:04.249513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:18.620999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813314+0000","last_change":"2024-03-21T14:08:15.817968+0000","last_active":"2024-03-21T14:10:18.813314+0000","last_peered":"2024-03-21T14:10:18.813314+0000","last_clean":"2024-03-21T14:10:18.813314+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:10:18.813314+0000","last_undegraded":"2024-03-21T14:10:18.813314+0000","last_fullsized":"2024-03-21T14:10:18.813314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:15.817858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:08:15.817858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401779+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":289,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794020+0000","last_change":"2024-03-21T14:09:27.347989+0000","last_active":"2024-03-21T14:10:13.794020+0000","last_peered":"2024-03-21T14:10:13.794020+0000","last_clean":"2024-03-21T14:10:13.794020+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:10:13.794020+0000","last_undegraded":"2024-03-21T14:10:13.794020+0000","last_fullsized":"2024-03-21T14:10:13.794020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:27.347835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:27.347835+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.996152+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":270,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820716+0000","last_change":"2024-03-21T14:09:24.494651+0000","last_active":"2024-03-21T14:10:18.820716+0000","last_peered":"2024-03-21T14:10:18.820716+0000","last_clean":"2024-03-21T14:10:18.820716+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:10:18.820716+0000","last_undegraded":"2024-03-21T14:10:18.820716+0000","last_fullsized":"2024-03-21T14:10:18.820716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.494516+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:32.193912+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":376,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784658+0000","last_change":"2024-03-21T14:09:58.786565+0000","last_active":"2024-03-21T14:10:13.784658+0000","last_peered":"2024-03-21T14:10:13.784658+0000","last_clean":"2024-03-21T14:10:13.784658+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:10:13.784658+0000","last_undegraded":"2024-03-21T14:10:13.784658+0000","last_fullsized":"2024-03-21T14:10:13.784658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:58.786336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.786336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.024325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.788889+0000","last_change":"2024-03-21T14:09:09.213981+0000","last_active":"2024-03-21T14:10:13.788889+0000","last_peered":"2024-03-21T14:10:13.788889+0000","last_clean":"2024-03-21T14:10:13.788889+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:10:13.788889+0000","last_undegraded":"2024-03-21T14:10:13.788889+0000","last_fullsized":"2024-03-21T14:10:13.788889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:09:09.213878+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.213878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.594067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":325,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784066+0000","last_change":"2024-03-21T14:09:02.781223+0000","last_active":"2024-03-21T14:10:13.784066+0000","last_peered":"2024-03-21T14:10:13.784066+0000","last_clean":"2024-03-21T14:10:13.784066+0000","last_became_active":"2024-03-21T14:08:17.144672+0000","last_became_peered":"2024-03-21T14:08:17.144672+0000","last_unstale":"2024-03-21T14:10:13.784066+0000","last_undegraded":"2024-03-21T14:10:13.784066+0000","last_fullsized":"2024-03-21T14:10:13.784066+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:09:02.781039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.781039+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:13.917935+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783127+0000","last_change":"2024-03-21T14:09:59.790745+0000","last_active":"2024-03-21T14:10:13.783127+0000","last_peered":"2024-03-21T14:10:13.783127+0000","last_clean":"2024-03-21T14:10:13.783127+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:10:13.783127+0000","last_undegraded":"2024-03-21T14:10:13.783127+0000","last_fullsized":"2024-03-21T14:10:13.783127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.790627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.790627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.885224+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784329+0000","last_change":"2024-03-21T14:10:07.843704+0000","last_active":"2024-03-21T14:10:13.784329+0000","last_peered":"2024-03-21T14:10:13.784329+0000","last_clean":"2024-03-21T14:10:13.784329+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:10:13.784329+0000","last_undegraded":"2024-03-21T14:10:13.784329+0000","last_fullsized":"2024-03-21T14:10:13.784329+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.839111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.839111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:08.766204+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":284,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.801733+0000","last_change":"2024-03-21T14:10:05.652480+0000","last_active":"2024-03-21T14:10:13.801733+0000","last_peered":"2024-03-21T14:10:13.801733+0000","last_clean":"2024-03-21T14:10:13.801733+0000","last_became_active":"2024-03-21T14:07:23.364680+0000","last_became_peered":"2024-03-21T14:07:23.364680+0000","last_unstale":"2024-03-21T14:10:13.801733+0000","last_undegraded":"2024-03-21T14:10:13.801733+0000","last_fullsized":"2024-03-21T14:10:13.801733+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:05.652409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:10:05.652409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.074656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"127'16","reported_seq":297,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784561+0000","last_change":"2024-03-21T14:09:05.048112+0000","last_active":"2024-03-21T14:10:13.784561+0000","last_peered":"2024-03-21T14:10:13.784561+0000","last_clean":"2024-03-21T14:10:13.784561+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:10:13.784561+0000","last_undegraded":"2024-03-21T14:10:13.784561+0000","last_fullsized":"2024-03-21T14:10:13.784561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'16","last_scrub_stamp":"2024-03-21T14:09:05.047983+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.047983+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.538539+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.819109+0000","last_change":"2024-03-21T14:10:07.201542+0000","last_active":"2024-03-21T14:10:18.819109+0000","last_peered":"2024-03-21T14:10:18.819109+0000","last_clean":"2024-03-21T14:10:18.819109+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:10:18.819109+0000","last_undegraded":"2024-03-21T14:10:18.819109+0000","last_fullsized":"2024-03-21T14:10:18.819109+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.201435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.201435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:12.168125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":269,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783668+0000","last_change":"2024-03-21T14:10:07.788804+0000","last_active":"2024-03-21T14:10:13.783668+0000","last_peered":"2024-03-21T14:10:13.783668+0000","last_clean":"2024-03-21T14:10:13.783668+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:10:13.783668+0000","last_undegraded":"2024-03-21T14:10:13.783668+0000","last_fullsized":"2024-03-21T14:10:13.783668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.788680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.788680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:24.873673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.781376+0000","last_change":"2024-03-21T14:10:12.855709+0000","last_active":"2024-03-21T14:10:13.781376+0000","last_peered":"2024-03-21T14:10:13.781376+0000","last_clean":"2024-03-21T14:10:13.781376+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:10:13.781376+0000","last_undegraded":"2024-03-21T14:10:13.781376+0000","last_fullsized":"2024-03-21T14:10:13.781376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:12.855582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.855582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:39.565947+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":280,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.801854+0000","last_change":"2024-03-21T14:10:07.737599+0000","last_active":"2024-03-21T14:10:13.801854+0000","last_peered":"2024-03-21T14:10:13.801854+0000","last_clean":"2024-03-21T14:10:13.801854+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:10:13.801854+0000","last_undegraded":"2024-03-21T14:10:13.801854+0000","last_fullsized":"2024-03-21T14:10:13.801854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:10:07.737472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.737472+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:10.501380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.841073+0000","last_change":"2024-03-21T14:10:13.841129+0000","last_active":"2024-03-21T14:10:13.841073+0000","last_peered":"2024-03-21T14:10:13.841073+0000","last_clean":"2024-03-21T14:10:13.841073+0000","last_became_active":"2024-03-21T14:08:17.194651+0000","last_became_peered":"2024-03-21T14:08:17.194651+0000","last_unstale":"2024-03-21T14:10:13.841073+0000","last_undegraded":"2024-03-21T14:10:13.841073+0000","last_fullsized":"2024-03-21T14:10:13.841073+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:10:13.840591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:14.102381+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":272,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820361+0000","last_change":"2024-03-21T14:09:09.475589+0000","last_active":"2024-03-21T14:10:18.820361+0000","last_peered":"2024-03-21T14:10:18.820361+0000","last_clean":"2024-03-21T14:10:18.820361+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:10:18.820361+0000","last_undegraded":"2024-03-21T14:10:18.820361+0000","last_fullsized":"2024-03-21T14:10:18.820361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.475453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.475453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.336499+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":285,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810584+0000","last_change":"2024-03-21T14:09:55.427714+0000","last_active":"2024-03-21T14:10:18.810584+0000","last_peered":"2024-03-21T14:10:18.810584+0000","last_clean":"2024-03-21T14:10:18.810584+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:10:18.810584+0000","last_undegraded":"2024-03-21T14:10:18.810584+0000","last_fullsized":"2024-03-21T14:10:18.810584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:55.427563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.427563+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.673618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":293,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.793586+0000","last_change":"2024-03-21T14:09:56.592384+0000","last_active":"2024-03-21T14:10:13.793586+0000","last_peered":"2024-03-21T14:10:13.793586+0000","last_clean":"2024-03-21T14:10:13.793586+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:10:13.793586+0000","last_undegraded":"2024-03-21T14:10:13.793586+0000","last_fullsized":"2024-03-21T14:10:13.793586+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:56.592242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.592242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.889308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783837+0000","last_change":"2024-03-21T14:09:12.819080+0000","last_active":"2024-03-21T14:10:13.783837+0000","last_peered":"2024-03-21T14:10:13.783837+0000","last_clean":"2024-03-21T14:10:13.783837+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:10:13.783837+0000","last_undegraded":"2024-03-21T14:10:13.783837+0000","last_fullsized":"2024-03-21T14:10:13.783837+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:12.818982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:12.818982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.361591+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":262,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794124+0000","last_change":"2024-03-21T14:09:10.079022+0000","last_active":"2024-03-21T14:10:13.794124+0000","last_peered":"2024-03-21T14:10:13.794124+0000","last_clean":"2024-03-21T14:10:13.794124+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:10:13.794124+0000","last_undegraded":"2024-03-21T14:10:13.794124+0000","last_fullsized":"2024-03-21T14:10:13.794124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.078939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.078939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.610355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":280,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820233+0000","last_change":"2024-03-21T14:10:02.141197+0000","last_active":"2024-03-21T14:10:18.820233+0000","last_peered":"2024-03-21T14:10:18.820233+0000","last_clean":"2024-03-21T14:10:18.820233+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:10:18.820233+0000","last_undegraded":"2024-03-21T14:10:18.820233+0000","last_fullsized":"2024-03-21T14:10:18.820233+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:32.006676+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":289,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.782099+0000","last_change":"2024-03-21T14:09:03.162696+0000","last_active":"2024-03-21T14:10:13.782099+0000","last_peered":"2024-03-21T14:10:13.782099+0000","last_clean":"2024-03-21T14:10:13.782099+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:10:13.782099+0000","last_undegraded":"2024-03-21T14:10:13.782099+0000","last_fullsized":"2024-03-21T14:10:13.782099+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:03.162584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.162584+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":131,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.781968+0000","last_change":"2024-03-21T14:10:10.915236+0000","last_active":"2024-03-21T14:10:13.781968+0000","last_peered":"2024-03-21T14:10:13.781968+0000","last_clean":"2024-03-21T14:10:13.781968+0000","last_became_active":"2024-03-21T14:07:23.360741+0000","last_became_peered":"2024-03-21T14:07:23.360741+0000","last_unstale":"2024-03-21T14:10:13.781968+0000","last_undegraded":"2024-03-21T14:10:13.781968+0000","last_fullsized":"2024-03-21T14:10:13.781968+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:10.915115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:10.915115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:39.408472+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.782003+0000","last_change":"2024-03-21T14:09:26.436609+0000","last_active":"2024-03-21T14:10:13.782003+0000","last_peered":"2024-03-21T14:10:13.782003+0000","last_clean":"2024-03-21T14:10:13.782003+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:10:13.782003+0000","last_undegraded":"2024-03-21T14:10:13.782003+0000","last_fullsized":"2024-03-21T14:10:13.782003+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:26.436326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.436326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:26.638026+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813281+0000","last_change":"2024-03-21T14:09:23.112108+0000","last_active":"2024-03-21T14:10:18.813281+0000","last_peered":"2024-03-21T14:10:18.813281+0000","last_clean":"2024-03-21T14:10:18.813281+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:10:18.813281+0000","last_undegraded":"2024-03-21T14:10:18.813281+0000","last_fullsized":"2024-03-21T14:10:18.813281+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.111999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.111999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:42.481221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"153'16","reported_seq":309,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813158+0000","last_change":"2024-03-21T14:08:30.872145+0000","last_active":"2024-03-21T14:10:18.813158+0000","last_peered":"2024-03-21T14:10:18.813158+0000","last_clean":"2024-03-21T14:10:18.813158+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:10:18.813158+0000","last_undegraded":"2024-03-21T14:10:18.813158+0000","last_fullsized":"2024-03-21T14:10:18.813158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:08:30.871984+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401812+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":125,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.821074+0000","last_change":"2024-03-21T14:09:10.489473+0000","last_active":"2024-03-21T14:10:18.821074+0000","last_peered":"2024-03-21T14:10:18.821074+0000","last_clean":"2024-03-21T14:10:18.821074+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:10:18.821074+0000","last_undegraded":"2024-03-21T14:10:18.821074+0000","last_fullsized":"2024-03-21T14:10:18.821074+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.489356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.489356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:37.103897+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":373,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810691+0000","last_change":"2024-03-21T14:09:15.064168+0000","last_active":"2024-03-21T14:10:18.810691+0000","last_peered":"2024-03-21T14:10:18.810691+0000","last_clean":"2024-03-21T14:10:18.810691+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:10:18.810691+0000","last_undegraded":"2024-03-21T14:10:18.810691+0000","last_fullsized":"2024-03-21T14:10:18.810691+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:15.063993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.063993+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.135176+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":278,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.819501+0000","last_change":"2024-03-21T14:08:17.142960+0000","last_active":"2024-03-21T14:10:18.819501+0000","last_peered":"2024-03-21T14:10:18.819501+0000","last_clean":"2024-03-21T14:10:18.819501+0000","last_became_active":"2024-03-21T14:08:17.142668+0000","last_became_peered":"2024-03-21T14:08:17.142668+0000","last_unstale":"2024-03-21T14:10:18.819501+0000","last_undegraded":"2024-03-21T14:10:18.819501+0000","last_fullsized":"2024-03-21T14:10:18.819501+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:12.703435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:08:12.703435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.244432+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"180'15","reported_seq":286,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794398+0000","last_change":"2024-03-21T14:10:09.765207+0000","last_active":"2024-03-21T14:10:13.794398+0000","last_peered":"2024-03-21T14:10:13.794398+0000","last_clean":"2024-03-21T14:10:13.794398+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:10:13.794398+0000","last_undegraded":"2024-03-21T14:10:13.794398+0000","last_fullsized":"2024-03-21T14:10:13.794398+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'15","last_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:10:09.765058+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.504477+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813473+0000","last_change":"2024-03-21T14:09:17.192133+0000","last_active":"2024-03-21T14:10:18.813473+0000","last_peered":"2024-03-21T14:10:18.813473+0000","last_clean":"2024-03-21T14:10:18.813473+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:10:18.813473+0000","last_undegraded":"2024-03-21T14:10:18.813473+0000","last_fullsized":"2024-03-21T14:10:18.813473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:17.192018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.192018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.830635+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"130'9","reported_seq":198,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820623+0000","last_change":"2024-03-21T14:10:00.156687+0000","last_active":"2024-03-21T14:10:18.820623+0000","last_peered":"2024-03-21T14:10:18.820623+0000","last_clean":"2024-03-21T14:10:18.820623+0000","last_became_active":"2024-03-21T14:07:23.354945+0000","last_became_peered":"2024-03-21T14:07:23.354945+0000","last_unstale":"2024-03-21T14:10:18.820623+0000","last_undegraded":"2024-03-21T14:10:18.820623+0000","last_fullsized":"2024-03-21T14:10:18.820623+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:10:00.156602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:00.156602+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.646196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":284,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.781344+0000","last_change":"2024-03-21T14:09:21.855689+0000","last_active":"2024-03-21T14:10:13.781344+0000","last_peered":"2024-03-21T14:10:13.781344+0000","last_clean":"2024-03-21T14:10:13.781344+0000","last_became_active":"2024-03-21T14:07:23.358188+0000","last_became_peered":"2024-03-21T14:07:23.358188+0000","last_unstale":"2024-03-21T14:10:13.781344+0000","last_undegraded":"2024-03-21T14:10:13.781344+0000","last_fullsized":"2024-03-21T14:10:13.781344+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:21.852936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:21.852936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:36.459798+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":283,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810861+0000","last_change":"2024-03-21T14:09:11.022485+0000","last_active":"2024-03-21T14:10:18.810861+0000","last_peered":"2024-03-21T14:10:18.810861+0000","last_clean":"2024-03-21T14:10:18.810861+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:10:18.810861+0000","last_undegraded":"2024-03-21T14:10:18.810861+0000","last_fullsized":"2024-03-21T14:10:18.810861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:11.022384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.022384+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.993211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":266,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.792407+0000","last_change":"2024-03-21T14:09:14.024358+0000","last_active":"2024-03-21T14:10:13.792407+0000","last_peered":"2024-03-21T14:10:13.792407+0000","last_clean":"2024-03-21T14:10:13.792407+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:10:13.792407+0000","last_undegraded":"2024-03-21T14:10:13.792407+0000","last_fullsized":"2024-03-21T14:10:13.792407+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:14.024237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:14.024237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.900582+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":122,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783981+0000","last_change":"2024-03-21T14:09:11.232306+0000","last_active":"2024-03-21T14:10:13.783981+0000","last_peered":"2024-03-21T14:10:13.783981+0000","last_clean":"2024-03-21T14:10:13.783981+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:10:13.783981+0000","last_undegraded":"2024-03-21T14:10:13.783981+0000","last_fullsized":"2024-03-21T14:10:13.783981+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.232197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.232197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:40.426883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":280,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.818736+0000","last_change":"2024-03-21T14:09:55.186771+0000","last_active":"2024-03-21T14:10:18.818736+0000","last_peered":"2024-03-21T14:10:18.818736+0000","last_clean":"2024-03-21T14:10:18.818736+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:10:18.818736+0000","last_undegraded":"2024-03-21T14:10:18.818736+0000","last_fullsized":"2024-03-21T14:10:18.818736+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.186641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.186641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:55.374836+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":290,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784168+0000","last_change":"2024-03-21T14:09:05.784892+0000","last_active":"2024-03-21T14:10:13.784168+0000","last_peered":"2024-03-21T14:10:13.784168+0000","last_clean":"2024-03-21T14:10:13.784168+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:10:13.784168+0000","last_undegraded":"2024-03-21T14:10:13.784168+0000","last_fullsized":"2024-03-21T14:10:13.784168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:05.784714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.784714+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:33.485395+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820085+0000","last_change":"2024-03-21T14:08:10.733076+0000","last_active":"2024-03-21T14:10:18.820085+0000","last_peered":"2024-03-21T14:10:18.820085+0000","last_clean":"2024-03-21T14:10:18.820085+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:10:18.820085+0000","last_undegraded":"2024-03-21T14:10:18.820085+0000","last_fullsized":"2024-03-21T14:10:18.820085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:10.732971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:10.732971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.244400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810514+0000","last_change":"2024-03-21T14:10:09.507069+0000","last_active":"2024-03-21T14:10:18.810514+0000","last_peered":"2024-03-21T14:10:18.810514+0000","last_clean":"2024-03-21T14:10:18.810514+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:10:18.810514+0000","last_undegraded":"2024-03-21T14:10:18.810514+0000","last_fullsized":"2024-03-21T14:10:18.810514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:10:09.506940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.590201+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":283,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783044+0000","last_change":"2024-03-21T14:09:04.020412+0000","last_active":"2024-03-21T14:10:13.783044+0000","last_peered":"2024-03-21T14:10:13.783044+0000","last_clean":"2024-03-21T14:10:13.783044+0000","last_became_active":"2024-03-21T14:07:23.354457+0000","last_became_peered":"2024-03-21T14:07:23.354457+0000","last_unstale":"2024-03-21T14:10:13.783044+0000","last_undegraded":"2024-03-21T14:10:13.783044+0000","last_fullsized":"2024-03-21T14:10:13.783044+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.020323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.020323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:14.587036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":161,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820499+0000","last_change":"2024-03-21T14:09:15.585161+0000","last_active":"2024-03-21T14:10:18.820499+0000","last_peered":"2024-03-21T14:10:18.820499+0000","last_clean":"2024-03-21T14:10:18.820499+0000","last_became_active":"2024-03-21T14:07:23.365278+0000","last_became_peered":"2024-03-21T14:07:23.365278+0000","last_unstale":"2024-03-21T14:10:18.820499+0000","last_undegraded":"2024-03-21T14:10:18.820499+0000","last_fullsized":"2024-03-21T14:10:18.820499+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:09:15.584995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.584995+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.826538+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"182'6707","reported_seq":6757,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:21.313718+0000","last_change":"2024-03-21T14:09:01.659122+0000","last_active":"2024-03-21T14:10:21.313718+0000","last_peered":"2024-03-21T14:10:21.313718+0000","last_clean":"2024-03-21T14:10:21.313718+0000","last_became_active":"2024-03-21T14:09:01.658623+0000","last_became_peered":"2024-03-21T14:09:01.658623+0000","last_unstale":"2024-03-21T14:10:21.313718+0000","last_undegraded":"2024-03-21T14:10:21.313718+0000","last_fullsized":"2024-03-21T14:10:21.313718+0000","mapping_epoch":162,"log_start":"167'1700","ondisk_log_start":"167'1700","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439549952,"num_objects":6707,"num_object_clones":0,"num_object_copies":20121,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6707,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6707,"num_write_kb":429248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.811660+0000","last_change":"2024-03-21T14:08:32.402570+0000","last_active":"2024-03-21T14:10:18.811660+0000","last_peered":"2024-03-21T14:10:18.811660+0000","last_clean":"2024-03-21T14:10:18.811660+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:10:18.811660+0000","last_undegraded":"2024-03-21T14:10:18.811660+0000","last_fullsized":"2024-03-21T14:10:18.811660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:32.402481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.402481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.504322+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":304,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820380+0000","last_change":"2024-03-21T14:10:08.205353+0000","last_active":"2024-03-21T14:10:18.820380+0000","last_peered":"2024-03-21T14:10:18.820380+0000","last_clean":"2024-03-21T14:10:18.820380+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:10:18.820380+0000","last_undegraded":"2024-03-21T14:10:18.820380+0000","last_fullsized":"2024-03-21T14:10:18.820380+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:10:08.205222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:08.205222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:25.723255+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":300,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.811384+0000","last_change":"2024-03-21T14:10:11.474876+0000","last_active":"2024-03-21T14:10:18.811384+0000","last_peered":"2024-03-21T14:10:18.811384+0000","last_clean":"2024-03-21T14:10:18.811384+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:10:18.811384+0000","last_undegraded":"2024-03-21T14:10:18.811384+0000","last_fullsized":"2024-03-21T14:10:18.811384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:10:11.474732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.474732+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.236235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":366,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.818939+0000","last_change":"2024-03-21T14:09:06.480162+0000","last_active":"2024-03-21T14:10:18.818939+0000","last_peered":"2024-03-21T14:10:18.818939+0000","last_clean":"2024-03-21T14:10:18.818939+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:10:18.818939+0000","last_undegraded":"2024-03-21T14:10:18.818939+0000","last_fullsized":"2024-03-21T14:10:18.818939+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:06.480079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:06.480079+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.027337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"181'6240","reported_seq":6357,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.267847+0000","last_change":"2024-03-21T14:10:12.650114+0000","last_active":"2024-03-21T14:10:16.267847+0000","last_peered":"2024-03-21T14:10:16.267847+0000","last_clean":"2024-03-21T14:10:16.267847+0000","last_became_active":"2024-03-21T14:09:01.656279+0000","last_became_peered":"2024-03-21T14:09:01.656279+0000","last_unstale":"2024-03-21T14:10:16.267847+0000","last_undegraded":"2024-03-21T14:10:16.267847+0000","last_fullsized":"2024-03-21T14:10:16.267847+0000","mapping_epoch":162,"log_start":"166'600","ondisk_log_start":"166'600","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'5980","last_scrub_stamp":"2024-03-21T14:10:12.649926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.649926+0000","objects_scrubbed":5922,"log_size":5640,"log_dups_size":0,"ondisk_log_size":5640,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:13.781877+0000","scrub_duration":1885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408944640,"num_objects":6240,"num_object_clones":0,"num_object_copies":18720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6240,"num_write_kb":399360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":81,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784366+0000","last_change":"2024-03-21T14:10:02.896547+0000","last_active":"2024-03-21T14:10:13.784366+0000","last_peered":"2024-03-21T14:10:13.784366+0000","last_clean":"2024-03-21T14:10:13.784366+0000","last_became_active":"2024-03-21T14:08:17.140381+0000","last_became_peered":"2024-03-21T14:08:17.140381+0000","last_unstale":"2024-03-21T14:10:13.784366+0000","last_undegraded":"2024-03-21T14:10:13.784366+0000","last_fullsized":"2024-03-21T14:10:13.784366+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.896413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.896413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:06.122825+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":280,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.821224+0000","last_change":"2024-03-21T14:09:07.478887+0000","last_active":"2024-03-21T14:10:18.821224+0000","last_peered":"2024-03-21T14:10:18.821224+0000","last_clean":"2024-03-21T14:10:18.821224+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:10:18.821224+0000","last_undegraded":"2024-03-21T14:10:18.821224+0000","last_fullsized":"2024-03-21T14:10:18.821224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:07.478803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:07.478803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.244523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":308,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812987+0000","last_change":"2024-03-21T14:09:20.135457+0000","last_active":"2024-03-21T14:10:18.812987+0000","last_peered":"2024-03-21T14:10:18.812987+0000","last_clean":"2024-03-21T14:10:18.812987+0000","last_became_active":"2024-03-21T14:07:23.352151+0000","last_became_peered":"2024-03-21T14:07:23.352151+0000","last_unstale":"2024-03-21T14:10:18.812987+0000","last_undegraded":"2024-03-21T14:10:18.812987+0000","last_fullsized":"2024-03-21T14:10:18.812987+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:20.135288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:20.135288+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:38.512648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.791466+0000","last_change":"2024-03-21T14:09:28.007840+0000","last_active":"2024-03-21T14:10:13.791466+0000","last_peered":"2024-03-21T14:10:13.791466+0000","last_clean":"2024-03-21T14:10:13.791466+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:10:13.791466+0000","last_undegraded":"2024-03-21T14:10:13.791466+0000","last_fullsized":"2024-03-21T14:10:13.791466+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.007714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:46.236621+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"181'6303","reported_seq":6350,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.851395+0000","last_change":"2024-03-21T14:09:01.657853+0000","last_active":"2024-03-21T14:10:16.851395+0000","last_peered":"2024-03-21T14:10:16.851395+0000","last_clean":"2024-03-21T14:10:16.851395+0000","last_became_active":"2024-03-21T14:09:01.656858+0000","last_became_peered":"2024-03-21T14:09:01.656858+0000","last_unstale":"2024-03-21T14:10:16.851395+0000","last_undegraded":"2024-03-21T14:10:16.851395+0000","last_fullsized":"2024-03-21T14:10:16.851395+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.988279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413073408,"num_objects":6303,"num_object_clones":0,"num_object_copies":18909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6303,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6303,"num_write_kb":403392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":268,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.819821+0000","last_change":"2024-03-21T14:10:11.210321+0000","last_active":"2024-03-21T14:10:18.819821+0000","last_peered":"2024-03-21T14:10:18.819821+0000","last_clean":"2024-03-21T14:10:18.819821+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:10:18.819821+0000","last_undegraded":"2024-03-21T14:10:18.819821+0000","last_fullsized":"2024-03-21T14:10:18.819821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.210194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.210194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:23.155153+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":270,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.791243+0000","last_change":"2024-03-21T14:09:04.791742+0000","last_active":"2024-03-21T14:10:13.791243+0000","last_peered":"2024-03-21T14:10:13.791243+0000","last_clean":"2024-03-21T14:10:13.791243+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:10:13.791243+0000","last_undegraded":"2024-03-21T14:10:13.791243+0000","last_fullsized":"2024-03-21T14:10:13.791243+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.791634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.791634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.075282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":272,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820435+0000","last_change":"2024-03-21T14:08:33.537783+0000","last_active":"2024-03-21T14:10:18.820435+0000","last_peered":"2024-03-21T14:10:18.820435+0000","last_clean":"2024-03-21T14:10:18.820435+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:10:18.820435+0000","last_undegraded":"2024-03-21T14:10:18.820435+0000","last_fullsized":"2024-03-21T14:10:18.820435+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:33.537669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:33.537669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.244461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"151'28","reported_seq":313,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783628+0000","last_change":"2024-03-21T14:10:03.820489+0000","last_active":"2024-03-21T14:10:13.783628+0000","last_peered":"2024-03-21T14:10:13.783628+0000","last_clean":"2024-03-21T14:10:13.783628+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:10:13.783628+0000","last_undegraded":"2024-03-21T14:10:13.783628+0000","last_fullsized":"2024-03-21T14:10:13.783628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'28","last_scrub_stamp":"2024-03-21T14:10:03.820339+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:08:22.013464+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.820339+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:10.338619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"181'6304","reported_seq":6352,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.664901+0000","last_change":"2024-03-21T14:09:01.452705+0000","last_active":"2024-03-21T14:10:16.664901+0000","last_peered":"2024-03-21T14:10:16.664901+0000","last_clean":"2024-03-21T14:10:16.664901+0000","last_became_active":"2024-03-21T14:09:01.452393+0000","last_became_peered":"2024-03-21T14:09:01.452393+0000","last_unstale":"2024-03-21T14:10:16.664901+0000","last_undegraded":"2024-03-21T14:10:16.664901+0000","last_fullsized":"2024-03-21T14:10:16.664901+0000","mapping_epoch":162,"log_start":"167'1900","ondisk_log_start":"167'1900","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4404,"log_dups_size":0,"ondisk_log_size":4404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.417036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":413138944,"num_objects":6304,"num_object_clones":0,"num_object_copies":18912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6304,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6304,"num_write_kb":403456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":125,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.818485+0000","last_change":"2024-03-21T14:09:28.472395+0000","last_active":"2024-03-21T14:10:18.818485+0000","last_peered":"2024-03-21T14:10:18.818485+0000","last_clean":"2024-03-21T14:10:18.818485+0000","last_became_active":"2024-03-21T14:07:23.354443+0000","last_became_peered":"2024-03-21T14:07:23.354443+0000","last_unstale":"2024-03-21T14:10:18.818485+0000","last_undegraded":"2024-03-21T14:10:18.818485+0000","last_fullsized":"2024-03-21T14:10:18.818485+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.472282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.472282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:57.179210+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.791225+0000","last_change":"2024-03-21T14:09:16.008790+0000","last_active":"2024-03-21T14:10:13.791225+0000","last_peered":"2024-03-21T14:10:13.791225+0000","last_clean":"2024-03-21T14:10:13.791225+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:10:13.791225+0000","last_undegraded":"2024-03-21T14:10:13.791225+0000","last_fullsized":"2024-03-21T14:10:13.791225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.008672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.008672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:39.764024+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810684+0000","last_change":"2024-03-21T14:10:12.496525+0000","last_active":"2024-03-21T14:10:18.810684+0000","last_peered":"2024-03-21T14:10:18.810684+0000","last_clean":"2024-03-21T14:10:18.810684+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:10:18.810684+0000","last_undegraded":"2024-03-21T14:10:18.810684+0000","last_fullsized":"2024-03-21T14:10:18.810684+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:12.496111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.496111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:41.002562+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":306,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784631+0000","last_change":"2024-03-21T14:09:55.859340+0000","last_active":"2024-03-21T14:10:13.784631+0000","last_peered":"2024-03-21T14:10:13.784631+0000","last_clean":"2024-03-21T14:10:13.784631+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:10:13.784631+0000","last_undegraded":"2024-03-21T14:10:13.784631+0000","last_fullsized":"2024-03-21T14:10:13.784631+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:55.859206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.859206+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.961510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"182'6785","reported_seq":6835,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:21.362534+0000","last_change":"2024-03-21T14:09:01.657675+0000","last_active":"2024-03-21T14:10:21.362534+0000","last_peered":"2024-03-21T14:10:21.362534+0000","last_clean":"2024-03-21T14:10:21.362534+0000","last_became_active":"2024-03-21T14:09:01.657373+0000","last_became_peered":"2024-03-21T14:09:01.657373+0000","last_unstale":"2024-03-21T14:10:21.362534+0000","last_undegraded":"2024-03-21T14:10:21.362534+0000","last_fullsized":"2024-03-21T14:10:21.362534+0000","mapping_epoch":162,"log_start":"167'1700","ondisk_log_start":"167'1700","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5085,"log_dups_size":0,"ondisk_log_size":5085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.201751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":444661760,"num_objects":6785,"num_object_clones":0,"num_object_copies":20355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6785,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6785,"num_write_kb":434240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794478+0000","last_change":"2024-03-21T14:10:04.683158+0000","last_active":"2024-03-21T14:10:13.794478+0000","last_peered":"2024-03-21T14:10:13.794478+0000","last_clean":"2024-03-21T14:10:13.794478+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:10:13.794478+0000","last_undegraded":"2024-03-21T14:10:13.794478+0000","last_fullsized":"2024-03-21T14:10:13.794478+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:04.683006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:04.683006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.743596+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":290,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.811567+0000","last_change":"2024-03-21T14:09:17.993826+0000","last_active":"2024-03-21T14:10:18.811567+0000","last_peered":"2024-03-21T14:10:18.811567+0000","last_clean":"2024-03-21T14:10:18.811567+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:10:18.811567+0000","last_undegraded":"2024-03-21T14:10:18.811567+0000","last_fullsized":"2024-03-21T14:10:18.811567+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:09:17.993708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.993708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.738656+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812879+0000","last_change":"2024-03-21T14:10:01.249279+0000","last_active":"2024-03-21T14:10:18.812879+0000","last_peered":"2024-03-21T14:10:18.812879+0000","last_clean":"2024-03-21T14:10:18.812879+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:10:18.812879+0000","last_undegraded":"2024-03-21T14:10:18.812879+0000","last_fullsized":"2024-03-21T14:10:18.812879+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:01.249165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.249165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.783002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"128'39","reported_seq":381,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810415+0000","last_change":"2024-03-21T14:10:03.405443+0000","last_active":"2024-03-21T14:10:18.810415+0000","last_peered":"2024-03-21T14:10:18.810415+0000","last_clean":"2024-03-21T14:10:18.810415+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:10:18.810415+0000","last_undegraded":"2024-03-21T14:10:18.810415+0000","last_fullsized":"2024-03-21T14:10:18.810415+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'39","last_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.405289+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:11.513185+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":131,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783987+0000","last_change":"2024-03-21T14:10:07.847117+0000","last_active":"2024-03-21T14:10:13.783987+0000","last_peered":"2024-03-21T14:10:13.783987+0000","last_clean":"2024-03-21T14:10:13.783987+0000","last_became_active":"2024-03-21T14:07:23.357608+0000","last_became_peered":"2024-03-21T14:07:23.357608+0000","last_unstale":"2024-03-21T14:10:13.783987+0000","last_undegraded":"2024-03-21T14:10:13.783987+0000","last_fullsized":"2024-03-21T14:10:13.783987+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.847006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.847006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.406528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"182'6572","reported_seq":6622,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:21.300281+0000","last_change":"2024-03-21T14:09:01.656314+0000","last_active":"2024-03-21T14:10:21.300281+0000","last_peered":"2024-03-21T14:10:21.300281+0000","last_clean":"2024-03-21T14:10:21.300281+0000","last_became_active":"2024-03-21T14:09:01.655845+0000","last_became_peered":"2024-03-21T14:09:01.655845+0000","last_unstale":"2024-03-21T14:10:21.300281+0000","last_undegraded":"2024-03-21T14:10:21.300281+0000","last_fullsized":"2024-03-21T14:10:21.300281+0000","mapping_epoch":162,"log_start":"167'1800","ondisk_log_start":"167'1800","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.346889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430702592,"num_objects":6572,"num_object_clones":0,"num_object_copies":19716,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6572,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6572,"num_write_kb":420608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820579+0000","last_change":"2024-03-21T14:08:16.661163+0000","last_active":"2024-03-21T14:10:18.820579+0000","last_peered":"2024-03-21T14:10:18.820579+0000","last_clean":"2024-03-21T14:10:18.820579+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:10:18.820579+0000","last_undegraded":"2024-03-21T14:10:18.820579+0000","last_fullsized":"2024-03-21T14:10:18.820579+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.661025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.661025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.244367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.802161+0000","last_change":"2024-03-21T14:09:28.482765+0000","last_active":"2024-03-21T14:10:13.802161+0000","last_peered":"2024-03-21T14:10:13.802161+0000","last_clean":"2024-03-21T14:10:13.802161+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:10:13.802161+0000","last_undegraded":"2024-03-21T14:10:13.802161+0000","last_fullsized":"2024-03-21T14:10:13.802161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.480168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.480168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:55.996315+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":295,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.791336+0000","last_change":"2024-03-21T14:09:58.644132+0000","last_active":"2024-03-21T14:10:13.791336+0000","last_peered":"2024-03-21T14:10:13.791336+0000","last_clean":"2024-03-21T14:10:13.791336+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:10:13.791336+0000","last_undegraded":"2024-03-21T14:10:13.791336+0000","last_fullsized":"2024-03-21T14:10:13.791336+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:58.643445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.643445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:06.955399+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":323,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783786+0000","last_change":"2024-03-21T14:09:15.326163+0000","last_active":"2024-03-21T14:10:13.783786+0000","last_peered":"2024-03-21T14:10:13.783786+0000","last_clean":"2024-03-21T14:10:13.783786+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:10:13.783786+0000","last_undegraded":"2024-03-21T14:10:13.783786+0000","last_fullsized":"2024-03-21T14:10:13.783786+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:09:15.325868+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.325868+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:42.864275+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.795284+0000","last_change":"2024-03-21T14:10:08.638321+0000","last_active":"2024-03-21T14:10:13.795284+0000","last_peered":"2024-03-21T14:10:13.795284+0000","last_clean":"2024-03-21T14:10:13.795284+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:10:13.795284+0000","last_undegraded":"2024-03-21T14:10:13.795284+0000","last_fullsized":"2024-03-21T14:10:13.795284+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:08.638191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:08.638191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:33.293386+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"181'6287","reported_seq":6335,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.225076+0000","last_change":"2024-03-21T14:09:01.451546+0000","last_active":"2024-03-21T14:10:16.225076+0000","last_peered":"2024-03-21T14:10:16.225076+0000","last_clean":"2024-03-21T14:10:16.225076+0000","last_became_active":"2024-03-21T14:09:01.451272+0000","last_became_peered":"2024-03-21T14:09:01.451272+0000","last_unstale":"2024-03-21T14:10:16.225076+0000","last_undegraded":"2024-03-21T14:10:16.225076+0000","last_fullsized":"2024-03-21T14:10:16.225076+0000","mapping_epoch":162,"log_start":"166'700","ondisk_log_start":"166'700","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5587,"log_dups_size":0,"ondisk_log_size":5587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:29.510106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412024832,"num_objects":6287,"num_object_clones":0,"num_object_copies":18861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6287,"num_write_kb":402368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810812+0000","last_change":"2024-03-21T14:09:04.091898+0000","last_active":"2024-03-21T14:10:18.810812+0000","last_peered":"2024-03-21T14:10:18.810812+0000","last_clean":"2024-03-21T14:10:18.810812+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:10:18.810812+0000","last_undegraded":"2024-03-21T14:10:18.810812+0000","last_fullsized":"2024-03-21T14:10:18.810812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.091797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.504425+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":551,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820023+0000","last_change":"2024-03-21T14:09:26.513570+0000","last_active":"2024-03-21T14:10:18.820023+0000","last_peered":"2024-03-21T14:10:18.820023+0000","last_clean":"2024-03-21T14:10:18.820023+0000","last_became_active":"2024-03-21T14:07:25.440544+0000","last_became_peered":"2024-03-21T14:07:25.440544+0000","last_unstale":"2024-03-21T14:10:18.820023+0000","last_undegraded":"2024-03-21T14:10:18.820023+0000","last_fullsized":"2024-03-21T14:10:18.820023+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.512762+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:53.511136+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.793799+0000","last_change":"2024-03-21T14:10:11.760727+0000","last_active":"2024-03-21T14:10:13.793799+0000","last_peered":"2024-03-21T14:10:13.793799+0000","last_clean":"2024-03-21T14:10:13.793799+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:10:13.793799+0000","last_undegraded":"2024-03-21T14:10:13.793799+0000","last_fullsized":"2024-03-21T14:10:13.793799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.760585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.760585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:31.868157+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":388,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783877+0000","last_change":"2024-03-21T14:09:27.011808+0000","last_active":"2024-03-21T14:10:13.783877+0000","last_peered":"2024-03-21T14:10:13.783877+0000","last_clean":"2024-03-21T14:10:13.783877+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:10:13.783877+0000","last_undegraded":"2024-03-21T14:10:13.783877+0000","last_fullsized":"2024-03-21T14:10:13.783877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:09:27.011698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:27.011698+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:44.932762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":272,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820804+0000","last_change":"2024-03-21T14:10:10.232175+0000","last_active":"2024-03-21T14:10:18.820804+0000","last_peered":"2024-03-21T14:10:18.820804+0000","last_clean":"2024-03-21T14:10:18.820804+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:10:18.820804+0000","last_undegraded":"2024-03-21T14:10:18.820804+0000","last_fullsized":"2024-03-21T14:10:18.820804+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:10.231809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:10.231809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:25.178394+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"181'6288","reported_seq":6336,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.662937+0000","last_change":"2024-03-21T14:09:01.656348+0000","last_active":"2024-03-21T14:10:16.662937+0000","last_peered":"2024-03-21T14:10:16.662937+0000","last_clean":"2024-03-21T14:10:16.662937+0000","last_became_active":"2024-03-21T14:09:01.656033+0000","last_became_peered":"2024-03-21T14:09:01.656033+0000","last_unstale":"2024-03-21T14:10:16.662937+0000","last_undegraded":"2024-03-21T14:10:16.662937+0000","last_fullsized":"2024-03-21T14:10:16.662937+0000","mapping_epoch":162,"log_start":"167'1900","ondisk_log_start":"167'1900","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4388,"log_dups_size":0,"ondisk_log_size":4388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.312139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":412090368,"num_objects":6288,"num_object_clones":0,"num_object_copies":18864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6288,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6288,"num_write_kb":402432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.782875+0000","last_change":"2024-03-21T14:10:00.794766+0000","last_active":"2024-03-21T14:10:13.782875+0000","last_peered":"2024-03-21T14:10:13.782875+0000","last_clean":"2024-03-21T14:10:13.782875+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:10:13.782875+0000","last_undegraded":"2024-03-21T14:10:13.782875+0000","last_fullsized":"2024-03-21T14:10:13.782875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:00.794637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:00.794637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.227279+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":284,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.782070+0000","last_change":"2024-03-21T14:09:59.812788+0000","last_active":"2024-03-21T14:10:13.782070+0000","last_peered":"2024-03-21T14:10:13.782070+0000","last_clean":"2024-03-21T14:10:13.782070+0000","last_became_active":"2024-03-21T14:07:23.362137+0000","last_became_peered":"2024-03-21T14:07:23.362137+0000","last_unstale":"2024-03-21T14:10:13.782070+0000","last_undegraded":"2024-03-21T14:10:13.782070+0000","last_fullsized":"2024-03-21T14:10:13.782070+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.812533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.812533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:07.401150+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":753,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:20.096644+0000","last_change":"2024-03-21T14:10:06.216315+0000","last_active":"2024-03-21T14:10:20.096644+0000","last_peered":"2024-03-21T14:10:20.096644+0000","last_clean":"2024-03-21T14:10:20.096644+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:10:20.096644+0000","last_undegraded":"2024-03-21T14:10:20.096644+0000","last_fullsized":"2024-03-21T14:10:20.096644+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:10:06.216180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.216180+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:32.696552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":181,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783650+0000","last_change":"2024-03-21T14:10:12.821439+0000","last_active":"2024-03-21T14:10:13.783650+0000","last_peered":"2024-03-21T14:10:13.783650+0000","last_clean":"2024-03-21T14:10:13.783650+0000","last_became_active":"2024-03-21T14:07:23.363490+0000","last_became_peered":"2024-03-21T14:07:23.363490+0000","last_unstale":"2024-03-21T14:10:13.783650+0000","last_undegraded":"2024-03-21T14:10:13.783650+0000","last_fullsized":"2024-03-21T14:10:13.783650+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:10:12.821292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.821292+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:39.357062+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":263,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.811048+0000","last_change":"2024-03-21T14:09:12.000023+0000","last_active":"2024-03-21T14:10:18.811048+0000","last_peered":"2024-03-21T14:10:18.811048+0000","last_clean":"2024-03-21T14:10:18.811048+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:10:18.811048+0000","last_undegraded":"2024-03-21T14:10:18.811048+0000","last_fullsized":"2024-03-21T14:10:18.811048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.999855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.999855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:20.504400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"181'6327","reported_seq":6374,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.703827+0000","last_change":"2024-03-21T14:09:01.657541+0000","last_active":"2024-03-21T14:10:16.703827+0000","last_peered":"2024-03-21T14:10:16.703827+0000","last_clean":"2024-03-21T14:10:16.703827+0000","last_became_active":"2024-03-21T14:09:01.656838+0000","last_became_peered":"2024-03-21T14:09:01.656838+0000","last_unstale":"2024-03-21T14:10:16.703827+0000","last_undegraded":"2024-03-21T14:10:16.703827+0000","last_fullsized":"2024-03-21T14:10:16.703827+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":6327,"log_dups_size":0,"ondisk_log_size":6327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.830747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414646272,"num_objects":6327,"num_object_clones":0,"num_object_copies":18981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6327,"num_write_kb":404928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":265,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.789124+0000","last_change":"2024-03-21T14:09:25.416918+0000","last_active":"2024-03-21T14:10:13.789124+0000","last_peered":"2024-03-21T14:10:13.789124+0000","last_clean":"2024-03-21T14:10:13.789124+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:10:13.789124+0000","last_undegraded":"2024-03-21T14:10:13.789124+0000","last_fullsized":"2024-03-21T14:10:13.789124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.416532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.416532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:45.771830+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783280+0000","last_change":"2024-03-21T14:09:23.381952+0000","last_active":"2024-03-21T14:10:13.783280+0000","last_peered":"2024-03-21T14:10:13.783280+0000","last_clean":"2024-03-21T14:10:13.783280+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:10:13.783280+0000","last_undegraded":"2024-03-21T14:10:13.783280+0000","last_fullsized":"2024-03-21T14:10:13.783280+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.381813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:28.987847+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":318,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784537+0000","last_change":"2024-03-21T14:10:01.815306+0000","last_active":"2024-03-21T14:10:13.784537+0000","last_peered":"2024-03-21T14:10:13.784537+0000","last_clean":"2024-03-21T14:10:13.784537+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:10:13.784537+0000","last_undegraded":"2024-03-21T14:10:13.784537+0000","last_fullsized":"2024-03-21T14:10:13.784537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:10:01.815146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.815146+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:13.735644+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783729+0000","last_change":"2024-03-21T14:10:13.768757+0000","last_active":"2024-03-21T14:10:13.783729+0000","last_peered":"2024-03-21T14:10:13.783729+0000","last_clean":"2024-03-21T14:10:13.783729+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:10:13.783729+0000","last_undegraded":"2024-03-21T14:10:13.783729+0000","last_fullsized":"2024-03-21T14:10:13.783729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:13.768639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:10:13.768639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.476557+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"181'6491","reported_seq":6539,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.719629+0000","last_change":"2024-03-21T14:09:01.657966+0000","last_active":"2024-03-21T14:10:16.719629+0000","last_peered":"2024-03-21T14:10:16.719629+0000","last_clean":"2024-03-21T14:10:16.719629+0000","last_became_active":"2024-03-21T14:09:01.657572+0000","last_became_peered":"2024-03-21T14:09:01.657572+0000","last_unstale":"2024-03-21T14:10:16.719629+0000","last_undegraded":"2024-03-21T14:10:16.719629+0000","last_fullsized":"2024-03-21T14:10:16.719629+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":6491,"log_dups_size":0,"ondisk_log_size":6491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.486234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425394176,"num_objects":6491,"num_object_clones":0,"num_object_copies":19473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6491,"num_write_kb":415424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":272,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783946+0000","last_change":"2024-03-21T14:09:54.876245+0000","last_active":"2024-03-21T14:10:13.783946+0000","last_peered":"2024-03-21T14:10:13.783946+0000","last_clean":"2024-03-21T14:10:13.783946+0000","last_became_active":"2024-03-21T14:08:17.136362+0000","last_became_peered":"2024-03-21T14:08:17.136362+0000","last_unstale":"2024-03-21T14:10:13.783946+0000","last_undegraded":"2024-03-21T14:10:13.783946+0000","last_fullsized":"2024-03-21T14:10:13.783946+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:54.875993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:54.875993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:08.671071+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":271,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810785+0000","last_change":"2024-03-21T14:09:05.122336+0000","last_active":"2024-03-21T14:10:18.810785+0000","last_peered":"2024-03-21T14:10:18.810785+0000","last_clean":"2024-03-21T14:10:18.810785+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:10:18.810785+0000","last_undegraded":"2024-03-21T14:10:18.810785+0000","last_fullsized":"2024-03-21T14:10:18.810785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:05.122232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.122232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:32.899878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"152'15","reported_seq":301,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.782052+0000","last_change":"2024-03-21T14:10:01.771055+0000","last_active":"2024-03-21T14:10:13.782052+0000","last_peered":"2024-03-21T14:10:13.782052+0000","last_clean":"2024-03-21T14:10:13.782052+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:10:13.782052+0000","last_undegraded":"2024-03-21T14:10:13.782052+0000","last_fullsized":"2024-03-21T14:10:13.782052+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'15","last_scrub_stamp":"2024-03-21T14:10:01.770915+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.770915+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.570180+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":371,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.784723+0000","last_change":"2024-03-21T14:09:56.813911+0000","last_active":"2024-03-21T14:10:13.784723+0000","last_peered":"2024-03-21T14:10:13.784723+0000","last_clean":"2024-03-21T14:10:13.784723+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:10:13.784723+0000","last_undegraded":"2024-03-21T14:10:13.784723+0000","last_fullsized":"2024-03-21T14:10:13.784723+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:56.813768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.813768+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.384070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"182'6684","reported_seq":6734,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:21.349813+0000","last_change":"2024-03-21T14:09:01.658726+0000","last_active":"2024-03-21T14:10:21.349813+0000","last_peered":"2024-03-21T14:10:21.349813+0000","last_clean":"2024-03-21T14:10:21.349813+0000","last_became_active":"2024-03-21T14:09:01.656735+0000","last_became_peered":"2024-03-21T14:09:01.656735+0000","last_unstale":"2024-03-21T14:10:21.349813+0000","last_undegraded":"2024-03-21T14:10:21.349813+0000","last_fullsized":"2024-03-21T14:10:21.349813+0000","mapping_epoch":162,"log_start":"167'1600","ondisk_log_start":"167'1600","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5084,"log_dups_size":0,"ondisk_log_size":5084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438042624,"num_objects":6684,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6684,"num_write_kb":427776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":260,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.819693+0000","last_change":"2024-03-21T14:09:03.463520+0000","last_active":"2024-03-21T14:10:18.819693+0000","last_peered":"2024-03-21T14:10:18.819693+0000","last_clean":"2024-03-21T14:10:18.819693+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:10:18.819693+0000","last_undegraded":"2024-03-21T14:10:18.819693+0000","last_fullsized":"2024-03-21T14:10:18.819693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:03.463400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.463400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:33.088597+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":134,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783497+0000","last_change":"2024-03-21T14:09:24.419269+0000","last_active":"2024-03-21T14:10:13.783497+0000","last_peered":"2024-03-21T14:10:13.783497+0000","last_clean":"2024-03-21T14:10:13.783497+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:10:13.783497+0000","last_undegraded":"2024-03-21T14:10:13.783497+0000","last_fullsized":"2024-03-21T14:10:13.783497+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:24.418831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.418831+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:46.285189+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":288,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812877+0000","last_change":"2024-03-21T14:08:32.898925+0000","last_active":"2024-03-21T14:10:18.812877+0000","last_peered":"2024-03-21T14:10:18.812877+0000","last_clean":"2024-03-21T14:10:18.812877+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:10:18.812877+0000","last_undegraded":"2024-03-21T14:10:18.812877+0000","last_fullsized":"2024-03-21T14:10:18.812877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:08:32.898753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.898753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401912+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":363,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.801976+0000","last_change":"2024-03-21T14:09:09.098309+0000","last_active":"2024-03-21T14:10:13.801976+0000","last_peered":"2024-03-21T14:10:13.801976+0000","last_clean":"2024-03-21T14:10:13.801976+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:10:13.801976+0000","last_undegraded":"2024-03-21T14:10:13.801976+0000","last_fullsized":"2024-03-21T14:10:13.801976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:09.098187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.098187+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.235333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"181'6490","reported_seq":6538,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.717989+0000","last_change":"2024-03-21T14:09:01.657521+0000","last_active":"2024-03-21T14:10:16.717989+0000","last_peered":"2024-03-21T14:10:16.717989+0000","last_clean":"2024-03-21T14:10:16.717989+0000","last_became_active":"2024-03-21T14:09:01.657075+0000","last_became_peered":"2024-03-21T14:09:01.657075+0000","last_unstale":"2024-03-21T14:10:16.717989+0000","last_undegraded":"2024-03-21T14:10:16.717989+0000","last_fullsized":"2024-03-21T14:10:16.717989+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":6490,"log_dups_size":0,"ondisk_log_size":6490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.671232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425328640,"num_objects":6490,"num_object_clones":0,"num_object_copies":19470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6490,"num_write_kb":415360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":265,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813211+0000","last_change":"2024-03-21T14:08:31.865055+0000","last_active":"2024-03-21T14:10:18.813211+0000","last_peered":"2024-03-21T14:10:18.813211+0000","last_clean":"2024-03-21T14:10:18.813211+0000","last_became_active":"2024-03-21T14:07:23.364265+0000","last_became_peered":"2024-03-21T14:07:23.364265+0000","last_unstale":"2024-03-21T14:10:18.813211+0000","last_undegraded":"2024-03-21T14:10:18.813211+0000","last_fullsized":"2024-03-21T14:10:18.813211+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:31.864937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.864937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":278,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.795746+0000","last_change":"2024-03-21T14:10:10.780682+0000","last_active":"2024-03-21T14:10:13.795746+0000","last_peered":"2024-03-21T14:10:13.795746+0000","last_clean":"2024-03-21T14:10:13.795746+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:10:13.795746+0000","last_undegraded":"2024-03-21T14:10:13.795746+0000","last_fullsized":"2024-03-21T14:10:13.795746+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:10:10.780543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:25.376223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":226,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.820256+0000","last_change":"2024-03-21T14:08:20.594480+0000","last_active":"2024-03-21T14:10:18.820256+0000","last_peered":"2024-03-21T14:10:18.820256+0000","last_clean":"2024-03-21T14:10:18.820256+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:10:18.820256+0000","last_undegraded":"2024-03-21T14:10:18.820256+0000","last_fullsized":"2024-03-21T14:10:18.820256+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:08:20.594343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:20.594343+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.244495+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":255,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812740+0000","last_change":"2024-03-21T14:08:16.845392+0000","last_active":"2024-03-21T14:10:18.812740+0000","last_peered":"2024-03-21T14:10:18.812740+0000","last_clean":"2024-03-21T14:10:18.812740+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:10:18.812740+0000","last_undegraded":"2024-03-21T14:10:18.812740+0000","last_fullsized":"2024-03-21T14:10:18.812740+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.845286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.845286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401873+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"181'6429","reported_seq":6477,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.681667+0000","last_change":"2024-03-21T14:09:01.452701+0000","last_active":"2024-03-21T14:10:16.681667+0000","last_peered":"2024-03-21T14:10:16.681667+0000","last_clean":"2024-03-21T14:10:16.681667+0000","last_became_active":"2024-03-21T14:09:01.452387+0000","last_became_peered":"2024-03-21T14:09:01.452387+0000","last_unstale":"2024-03-21T14:10:16.681667+0000","last_undegraded":"2024-03-21T14:10:16.681667+0000","last_fullsized":"2024-03-21T14:10:16.681667+0000","mapping_epoch":162,"log_start":"167'2000","ondisk_log_start":"167'2000","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4429,"log_dups_size":0,"ondisk_log_size":4429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.296270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421330944,"num_objects":6429,"num_object_clones":0,"num_object_copies":19287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6429,"num_write_kb":411456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.810481+0000","last_change":"2024-03-21T14:09:57.384748+0000","last_active":"2024-03-21T14:10:18.810481+0000","last_peered":"2024-03-21T14:10:18.810481+0000","last_clean":"2024-03-21T14:10:18.810481+0000","last_became_active":"2024-03-21T14:08:17.203307+0000","last_became_peered":"2024-03-21T14:08:17.203307+0000","last_unstale":"2024-03-21T14:10:18.810481+0000","last_undegraded":"2024-03-21T14:10:18.810481+0000","last_fullsized":"2024-03-21T14:10:18.810481+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:57.384552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:09:57.384552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:23.621828+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":365,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813515+0000","last_change":"2024-03-21T14:08:26.932388+0000","last_active":"2024-03-21T14:10:18.813515+0000","last_peered":"2024-03-21T14:10:18.813515+0000","last_clean":"2024-03-21T14:10:18.813515+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:10:18.813515+0000","last_undegraded":"2024-03-21T14:10:18.813515+0000","last_fullsized":"2024-03-21T14:10:18.813515+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:08:26.932274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:26.932274+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401844+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":123,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.802321+0000","last_change":"2024-03-21T14:09:13.247866+0000","last_active":"2024-03-21T14:10:13.802321+0000","last_peered":"2024-03-21T14:10:13.802321+0000","last_clean":"2024-03-21T14:10:13.802321+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:10:13.802321+0000","last_undegraded":"2024-03-21T14:10:13.802321+0000","last_fullsized":"2024-03-21T14:10:13.802321+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:13.247726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:13.247726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:30.028509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":298,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783880+0000","last_change":"2024-03-21T14:09:58.762917+0000","last_active":"2024-03-21T14:10:13.783880+0000","last_peered":"2024-03-21T14:10:13.783880+0000","last_clean":"2024-03-21T14:10:13.783880+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:10:13.783880+0000","last_undegraded":"2024-03-21T14:10:13.783880+0000","last_fullsized":"2024-03-21T14:10:13.783880+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:09:58.762639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.762639+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.152458+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"182'6805","reported_seq":6855,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:21.308726+0000","last_change":"2024-03-21T14:09:01.451050+0000","last_active":"2024-03-21T14:10:21.308726+0000","last_peered":"2024-03-21T14:10:21.308726+0000","last_clean":"2024-03-21T14:10:21.308726+0000","last_became_active":"2024-03-21T14:09:01.450729+0000","last_became_peered":"2024-03-21T14:09:01.450729+0000","last_unstale":"2024-03-21T14:10:21.308726+0000","last_undegraded":"2024-03-21T14:10:21.308726+0000","last_fullsized":"2024-03-21T14:10:21.308726+0000","mapping_epoch":162,"log_start":"167'2100","ondisk_log_start":"167'2100","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.446182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":445972480,"num_objects":6805,"num_object_clones":0,"num_object_copies":20415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6805,"num_write_kb":435520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":258,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794191+0000","last_change":"2024-03-21T14:10:03.673038+0000","last_active":"2024-03-21T14:10:13.794191+0000","last_peered":"2024-03-21T14:10:13.794191+0000","last_clean":"2024-03-21T14:10:13.794191+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:10:13.794191+0000","last_undegraded":"2024-03-21T14:10:13.794191+0000","last_fullsized":"2024-03-21T14:10:13.794191+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.672903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.672903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.596347+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":263,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.811017+0000","last_change":"2024-03-21T14:09:16.047979+0000","last_active":"2024-03-21T14:10:18.811017+0000","last_peered":"2024-03-21T14:10:18.811017+0000","last_clean":"2024-03-21T14:10:18.811017+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:10:18.811017+0000","last_undegraded":"2024-03-21T14:10:18.811017+0000","last_fullsized":"2024-03-21T14:10:18.811017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.047852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.047852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.726094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":279,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.812753+0000","last_change":"2024-03-21T14:10:02.232967+0000","last_active":"2024-03-21T14:10:18.812753+0000","last_peered":"2024-03-21T14:10:18.812753+0000","last_clean":"2024-03-21T14:10:18.812753+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:10:18.812753+0000","last_undegraded":"2024-03-21T14:10:18.812753+0000","last_fullsized":"2024-03-21T14:10:18.812753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.232818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.232818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.637769+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":286,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813355+0000","last_change":"2024-03-21T14:09:03.147020+0000","last_active":"2024-03-21T14:10:18.813355+0000","last_peered":"2024-03-21T14:10:18.813355+0000","last_clean":"2024-03-21T14:10:18.813355+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:10:18.813355+0000","last_undegraded":"2024-03-21T14:10:18.813355+0000","last_fullsized":"2024-03-21T14:10:18.813355+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:03.146877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.146877+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:19.401974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"181'6322","reported_seq":6368,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.681324+0000","last_change":"2024-03-21T14:09:01.659005+0000","last_active":"2024-03-21T14:10:16.681324+0000","last_peered":"2024-03-21T14:10:16.681324+0000","last_clean":"2024-03-21T14:10:16.681324+0000","last_became_active":"2024-03-21T14:09:01.656962+0000","last_became_peered":"2024-03-21T14:09:01.656962+0000","last_unstale":"2024-03-21T14:10:16.681324+0000","last_undegraded":"2024-03-21T14:10:16.681324+0000","last_fullsized":"2024-03-21T14:10:16.681324+0000","mapping_epoch":162,"log_start":"167'1900","ondisk_log_start":"167'1900","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4422,"log_dups_size":0,"ondisk_log_size":4422,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:18.506609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414318592,"num_objects":6322,"num_object_clones":0,"num_object_copies":18966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6322,"num_write_kb":404608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":275,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.792094+0000","last_change":"2024-03-21T14:10:03.740909+0000","last_active":"2024-03-21T14:10:13.792094+0000","last_peered":"2024-03-21T14:10:13.792094+0000","last_clean":"2024-03-21T14:10:13.792094+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:10:13.792094+0000","last_undegraded":"2024-03-21T14:10:13.792094+0000","last_fullsized":"2024-03-21T14:10:13.792094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:10:03.740783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.740783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:12.932320+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":269,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783429+0000","last_change":"2024-03-21T14:09:16.366776+0000","last_active":"2024-03-21T14:10:13.783429+0000","last_peered":"2024-03-21T14:10:13.783429+0000","last_clean":"2024-03-21T14:10:13.783429+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:10:13.783429+0000","last_undegraded":"2024-03-21T14:10:13.783429+0000","last_fullsized":"2024-03-21T14:10:13.783429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.366566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.366566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:24.932561+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":277,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.783818+0000","last_change":"2024-03-21T14:10:08.767362+0000","last_active":"2024-03-21T14:10:13.783818+0000","last_peered":"2024-03-21T14:10:13.783818+0000","last_clean":"2024-03-21T14:10:13.783818+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:10:13.783818+0000","last_undegraded":"2024-03-21T14:10:13.783818+0000","last_fullsized":"2024-03-21T14:10:13.783818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:08.767256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:08.767256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:11.594705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":295,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813255+0000","last_change":"2024-03-21T14:09:04.171171+0000","last_active":"2024-03-21T14:10:18.813255+0000","last_peered":"2024-03-21T14:10:18.813255+0000","last_clean":"2024-03-21T14:10:18.813255+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:10:18.813255+0000","last_undegraded":"2024-03-21T14:10:18.813255+0000","last_fullsized":"2024-03-21T14:10:18.813255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:09:04.171069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.171069+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:26.513971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"181'6341","reported_seq":6389,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:16.761334+0000","last_change":"2024-03-21T14:09:01.451288+0000","last_active":"2024-03-21T14:10:16.761334+0000","last_peered":"2024-03-21T14:10:16.761334+0000","last_clean":"2024-03-21T14:10:16.761334+0000","last_became_active":"2024-03-21T14:09:01.451020+0000","last_became_peered":"2024-03-21T14:09:01.451020+0000","last_unstale":"2024-03-21T14:10:16.761334+0000","last_undegraded":"2024-03-21T14:10:16.761334+0000","last_fullsized":"2024-03-21T14:10:16.761334+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":6341,"log_dups_size":0,"ondisk_log_size":6341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:21.671212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415563776,"num_objects":6341,"num_object_clones":0,"num_object_copies":19023,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6341,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6341,"num_write_kb":405824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":259,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813541+0000","last_change":"2024-03-21T14:09:09.117631+0000","last_active":"2024-03-21T14:10:18.813541+0000","last_peered":"2024-03-21T14:10:18.813541+0000","last_clean":"2024-03-21T14:10:18.813541+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:10:18.813541+0000","last_undegraded":"2024-03-21T14:10:18.813541+0000","last_fullsized":"2024-03-21T14:10:18.813541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.117496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.117496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:23.877971+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":372,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.791667+0000","last_change":"2024-03-21T14:09:10.904857+0000","last_active":"2024-03-21T14:10:13.791667+0000","last_peered":"2024-03-21T14:10:13.791667+0000","last_clean":"2024-03-21T14:10:13.791667+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:10:13.791667+0000","last_undegraded":"2024-03-21T14:10:13.791667+0000","last_fullsized":"2024-03-21T14:10:13.791667+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:09:10.904772+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.904772+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.436321+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":276,"reported_epoch":181,"state":"active+clean","last_fresh":"2024-03-21T14:10:13.794037+0000","last_change":"2024-03-21T14:09:19.247854+0000","last_active":"2024-03-21T14:10:13.794037+0000","last_peered":"2024-03-21T14:10:13.794037+0000","last_clean":"2024-03-21T14:10:13.794037+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:10:13.794037+0000","last_undegraded":"2024-03-21T14:10:13.794037+0000","last_fullsized":"2024-03-21T14:10:13.794037+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.247746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.247746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:27.335818+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"167'15","reported_seq":287,"reported_epoch":182,"state":"active+clean","last_fresh":"2024-03-21T14:10:18.813086+0000","last_change":"2024-03-21T14:09:22.120079+0000","last_active":"2024-03-21T14:10:18.813086+0000","last_peered":"2024-03-21T14:10:18.813086+0000","last_clean":"2024-03-21T14:10:18.813086+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:10:18.813086+0000","last_undegraded":"2024-03-21T14:10:18.813086+0000","last_fullsized":"2024-03-21T14:10:18.813086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'15","last_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:09:22.119861+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:10:22.576029+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":6774784000,"num_objects":103375,"num_object_clones":0,"num_object_copies":310125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":103375,"num_write_kb":6616000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20406861824,"data_stored":20406861824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85475,"ondisk_log_size":85475,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103266,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2322784,"kb_used_data":2138904,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91422368,"statfs":{"total":95995035648,"available":93616504832,"internally_reserved":0,"allocated":2190237696,"data_stored":2189841330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":188283338},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430310,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2179372,"kb_used_data":2003808,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91565780,"statfs":{"total":95995035648,"available":93763358720,"internally_reserved":0,"allocated":2051899392,"data_stored":2051457712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179763654},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574977,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3051096,"kb_used_data":2807504,"kb_used_omap":1,"kb_used_meta":243582,"kb_avail":90694056,"statfs":{"total":95995035648,"available":92870713344,"internally_reserved":0,"allocated":2874884096,"data_stored":2874395690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":249428421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738502,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2288988,"kb_used_data":2027348,"kb_used_omap":1,"kb_used_meta":261630,"kb_avail":91456164,"statfs":{"total":95995035648,"available":93651111936,"internally_reserved":0,"allocated":2076004352,"data_stored":2075574522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":267909573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,0,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411451,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3068772,"kb_used_data":2843228,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":90676380,"statfs":{"total":95995035648,"available":92852613120,"internally_reserved":0,"allocated":2911465472,"data_stored":2911027472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":230947269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182517,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2335288,"kb_used_data":2043120,"kb_used_omap":1,"kb_used_meta":292158,"kb_avail":91409864,"statfs":{"total":95995035648,"available":93603700736,"internally_reserved":0,"allocated":2092154880,"data_stored":2091696952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":299170247},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018994,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2305680,"kb_used_data":2121616,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":91439472,"statfs":{"total":95995035648,"available":93634019328,"internally_reserved":0,"allocated":2172534784,"data_stored":2172144781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":188283333},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691949,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4026196,"kb_used_data":3836748,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":89718956,"statfs":{"total":95995035648,"available":91872210944,"internally_reserved":0,"allocated":3928829952,"data_stored":3928444806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2092695552,"data_stored":2092695552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2901540864,"data_stored":2901540864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2911502336,"data_stored":2911502336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2092498944,"data_stored":2092498944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2192179200,"data_stored":2192179200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3962503168,"data_stored":3962503168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2063859712,"data_stored":2063859712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2190082048,"data_stored":2190082048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:10:23.588 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-21T14:10:23.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:23.596 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:23.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:23.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:23.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:10:23.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:23.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:23.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:23.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 109567 109551 96.4122 67.1875 0.0302277 0.0103664 2024-03-21T14:10:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:23 smithi118 ceph-mon[33862]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:23.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:24.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:24.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:23 smithi003 ceph-mon[29576]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:24.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:23 smithi003 ceph-mon[40510]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:10:24.202 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:24.238 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:24.340 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:24.371 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:24.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:24.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:24.505 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:24.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:24.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:24.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:24.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:24.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:24.892 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:24.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 110667 110651 96.028 68.75 0.00821505 0.0104089 2024-03-21T14:10:24.925 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:24.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:25.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:25.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:25.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:24 smithi118 ceph-mon[33862]: pgmap v322: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T14:10:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:25.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:24 smithi003 ceph-mon[29576]: pgmap v322: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T14:10:25.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:24 smithi003 ceph-mon[40510]: pgmap v322: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-21T14:10:25.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:25.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:25.442 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:25.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:25.492 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:25.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:25.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:25.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:25.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:25.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:25.869 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:10:25.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:25.898 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 111815 111799 95.6954 71.75 0.0222337 0.0104438 2024-03-21T14:10:25.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:25.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:25.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:25.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1606014925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:25.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:25 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:25.979 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:25 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1606014925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:25.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:25 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:26.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:26.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:26.119 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:10:26.153 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:10:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:25 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1606014925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:25 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:26.181 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:26.192 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:10:26.218 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:26.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:26.226 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:10:26.266 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:10:26.307 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:10:26.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:26.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:10:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:26.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:26.392 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:10:26.400 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:10:26.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:26.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:26.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:26.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:26.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:26.711 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:10:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:10:26.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:26.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:26.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:26.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 112974 112958 95.3794 72.4375 0.00748561 0.0104796 2024-03-21T14:10:26.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:26.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:26.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[29576]: pgmap v323: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[29576]: osdmap e183: 8 total, 8 up, 8 in 2024-03-21T14:10:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1613943997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[40510]: pgmap v323: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[40510]: osdmap e183: 8 total, 8 up, 8 in 2024-03-21T14:10:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1613943997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:27.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:27.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:27.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:27.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:27.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:10:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:10:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:26 smithi118 ceph-mon[33862]: pgmap v323: 148 pgs: 148 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:26 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:26 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:26 smithi118 ceph-mon[33862]: osdmap e183: 8 total, 8 up, 8 in 2024-03-21T14:10:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1613943997' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:27.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:27.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:27.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:27.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:27.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:27.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:27.488 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:27.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:27.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:27.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:27.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:27.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:27.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:27.899 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 114195 114179 95.1251 76.3125 0.00634133 0.0105081 2024-03-21T14:10:27.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:27.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:27.982 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:27.982 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:28.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:28.052 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:28.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:28.185 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:28.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:28.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:28.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:28.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:28.467 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:28.569 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:28.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:10:28.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:10:28.589 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-21T14:10:28.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T14:10:28.610 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:28.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:28.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:28.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:10:28.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:28.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 115375 115359 94.8438 73.75 0.0128053 0.0105388 2024-03-21T14:10:28.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:28.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:29.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:29.158 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:28 smithi003 ceph-mon[40510]: pgmap v325: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-21T14:10:29.159 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:28 smithi003 ceph-mon[29576]: pgmap v325: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-21T14:10:29.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:29.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:28 smithi118 ceph-mon[33862]: pgmap v325: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.27k op/s 2024-03-21T14:10:29.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:29.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:29.360 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:29.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:29.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:29.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.502 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:29.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:29.592 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:115815 8.1 e183/162 8:871eaeb7:::benchmark_data_smithi003_110242_object115814:head v 183'7218, mlcod=183'7215)", 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:10:29.356888+0000", 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00084154599999999998, 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00087739899999999995, 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.356888+0000", 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.356888+0000", 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.356889+0000", 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1119999999999999e-06 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:10:29.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.356913+0000", 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3893999999999999e-05 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.356945+0000", 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1819999999999997e-05 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.356955+0000", 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0429e-05 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:10:29.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357376+0000", 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00042053000000000001 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357395+0000", 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9079000000000001e-05 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:115817 8.2 e183/162 8:4d5ca92b:::benchmark_data_smithi003_110242_object115816:head v 183'7379, mlcod=183'7377)", 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:10:29.357024+0000", 2024-03-21T14:10:29.596 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00070597499999999996, 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00081494, 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357024+0000", 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357024+0000", 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357024+0000", 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5199999999999996e-07 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357107+0000", 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2736000000000002e-05 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357121+0000", 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3572e-05 2024-03-21T14:10:29.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357128+0000", 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2389999999999996e-06 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357191+0000", 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3330000000000005e-05 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357215+0000", 2024-03-21T14:10:29.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3776000000000001e-05 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15132.0:115810 8.e e183/162)", 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:10:29.357208+0000", 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00052167999999999995, 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00068916300000000002, 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:10:29.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357208+0000", 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357208+0000", 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357208+0000", 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3000000000000001e-07 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357212+0000", 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4829999999999999e-06 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357220+0000", 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8280000000000003e-06 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.357225+0000", 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7320000000000003e-06 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:29.602 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:115810 8.e 8:77728758:::benchmark_data_smithi003_110242_object115809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:10:29.350486+0000", 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0072433640000000004, 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007474095, 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:10:29.603 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 115810 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.350486+0000", 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.350486+0000", 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.350488+0000", 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5579999999999999e-06 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.350511+0000", 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3348000000000001e-05 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.350525+0000", 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3387000000000001e-05 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.350532+0000", 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6909999999999997e-06 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.352002+0000", 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001470177 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.352481+0000", 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047891299999999997 2024-03-21T14:10:29.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:29.352888+0000", 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040699200000000001 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 4 2024-03-21T14:10:29.607 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:10:29.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:29.646 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:29.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:29.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:29.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:29.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:10:29.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:29.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 116468 116452 94.4974 68.3125 0.0156167 0.0105777 2024-03-21T14:10:29.918 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:29.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:29.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:30.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.052 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:30.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:30.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:30.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:30.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.323 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:30.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:30.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:30.459 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:30.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:10:30.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:30.590 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:30.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:30.828 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:30.856 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:30.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 117527 117511 94.1341 66.1875 0.00583839 0.010617 2024-03-21T14:10:30.958 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:30.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:30.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:31.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:31.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:30 smithi118 ceph-mon[33862]: pgmap v326: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:31.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:30 smithi003 ceph-mon[29576]: pgmap v326: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:30 smithi003 ceph-mon[40510]: pgmap v326: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:10:31.274 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:31.290 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:31.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:31.416 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:31.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:31.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:31.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:31.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:31.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:31.831 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:31.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 118718 118702 93.8847 74.4375 0.0105116 0.0106468 2024-03-21T14:10:31.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:31.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:31.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:32.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:32.101 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:32.110 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:32.182 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:32.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:32.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:32.233 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:32.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:32.369 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:32.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:32.470 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:32.498 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:32.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:32.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:32.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:32.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:32.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.789 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:32.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:32.890 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:32.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:10:32.902807+0000 min lat: 0.00121156 max lat: 0.128614 avg lat: 0.0106756 2024-03-21T14:10:32.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:10:32.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 119891 119875 93.6275 73.3125 0.0155728 0.0106756 2024-03-21T14:10:32.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:32.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:33.029 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:33.059 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:33.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:33.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:32 smithi118 ceph-mon[33862]: pgmap v327: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-21T14:10:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:33.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:32 smithi003 ceph-mon[29576]: pgmap v327: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-21T14:10:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:32 smithi003 ceph-mon[40510]: pgmap v327: 148 pgs: 148 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-21T14:10:33.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:33.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:33.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:33.483 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:33.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:33.610 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:33.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:33.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:33.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:33.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:33.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:33.901 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 121073 121057 93.3836 73.875 0.0191182 0.0107041 2024-03-21T14:10:33.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:34.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:34.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:34.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:34.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:34.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:34.315 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:34.417 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:34.455 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:34.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:34.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:34.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.734 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:34.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:34.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:34.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:34.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 122236 122220 93.1313 72.6875 0.0113546 0.0107322 2024-03-21T14:10:34.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:34.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:34.999 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:34.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:35.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:35.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:35.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:34 smithi118 ceph-mon[33862]: pgmap v328: 148 pgs: 148 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:35.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:34 smithi003 ceph-mon[29576]: pgmap v328: 148 pgs: 148 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:35.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:34 smithi003 ceph-mon[40510]: pgmap v328: 148 pgs: 148 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:35.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:35.380 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:35.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:35.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:35.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:35.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:35.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:35.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:35.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:35.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:35.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:35.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 123316 123300 92.8224 67.5 0.016447 0.0107682 2024-03-21T14:10:35.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:35.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:35.962 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:35.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:36.008 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:36.063 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:36.097 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:36.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:36.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:36.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:36.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:36.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:36.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:36.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:36.533 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:10:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:10:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:10:36.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:36.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:36.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:36.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:36.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:36.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 124432 124416 92.5477 69.75 0.00417514 0.0107994 2024-03-21T14:10:36.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:36.954 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:36.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:37.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:37.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:37.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:36 smithi003 ceph-mon[29576]: pgmap v329: 148 pgs: 148 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:37.130 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:36 smithi003 ceph-mon[40510]: pgmap v329: 148 pgs: 148 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:36 smithi118 ceph-mon[33862]: pgmap v329: 148 pgs: 148 active+clean; 7.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:37.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:10:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:10:37.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:37.219 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 183, 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-03-21T13:56:54.530125+0000", 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-03-21T14:10:25.840077+0000", 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2024-03-21T14:02:19.321454+0000", 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2024-03-21T14:02:01.236118+0000", 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5806080, 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2024-03-21T14:10:37.220 INFO:teuthology.orchestra.run.smithi003.stdout: "nodeep-scrub", 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "noscrub", 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 18, 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 8, 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2024-03-21T14:10:37.221 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "luminous", 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "luminous", 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "squid", 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_crimson": false, 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".mgr", 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:00:12.925535+0000", 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.222 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.223 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "122", 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.224 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.225 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.226 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_max": 32, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.227 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": {} 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 7.8899998664855957, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 7.8899998664855957, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 3, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 3, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "datapool", 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.373659+0000", 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.229 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 3, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 3, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 3, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 3, 2024-03-21T14:10:37.230 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 3, 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:10:37.231 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 2, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 68, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.232 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.233 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.234 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.6500000953674316, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.6500000953674316, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-03-21T14:10:37.235 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".rgw.root", 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.794831+0000", 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.236 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "66", 2024-03-21T14:10:37.237 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.238 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.239 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:10:37.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.5, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.5, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.241 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 4, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.log", 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:33.750394+0000", 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.242 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.243 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.244 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.245 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.246 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.25, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.25, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.247 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 5, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.control", 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:35.711498+0000", 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.248 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "70", 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.249 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.250 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.251 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.252 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.5, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.5, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 6, 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.meta", 2024-03-21T14:10:37.253 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:37.763124+0000", 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.254 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.255 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "72", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.256 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:10:37.258 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.25, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.25, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 8, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "unique_pool_1", 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:08:59.821602+0000", 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:10:37.260 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 16, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 16, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 16, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 16, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 16, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:10:37.261 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "165", 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:10:37.262 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:10:37.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:10:37.264 INFO:teuthology.orchestra.run.smithi003.stdout: "rados": {} 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:10:37.265 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb", 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.73016357421875, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 8, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 162, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.266 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:10:37.267 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.268 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6801/930096433", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6802/930096433", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6804/930096433", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6803/930096433", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d", 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 13, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 162, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.270 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.271 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.271 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:37.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.273 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6805/1526485482", 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6806/1526485482", 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6808/1526485482", 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/1526485482", 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.275 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c", 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 18, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 143, 2024-03-21T14:10:37.276 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.277 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6809/3751378277", 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6810/3751378277", 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6812/3751378277", 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6811/3751378277", 2024-03-21T14:10:37.279 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c", 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.6562042236328125, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 25, 2024-03-21T14:10:37.280 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 162, 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.282 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6813/1918209225", 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6814/1918209225", 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6816/1918209225", 2024-03-21T14:10:37.283 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6815/1918209225", 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5", 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.284 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 30, 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 162, 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6800", 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.285 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6801", 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6803", 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:10:37.286 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6802", 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6800/296472230", 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6801/296472230", 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6803/296472230", 2024-03-21T14:10:37.287 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6802/296472230", 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "bff24605-38ce-47ce-9142-4ced3766476e", 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.288 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 40, 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 162, 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6804", 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.289 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6805", 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6807", 2024-03-21T14:10:37.290 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6806", 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.291 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6804/326385780", 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6805/326385780", 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6807/326385780", 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6806/326385780", 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "73a781e7-8660-4885-b976-2aeb8515109a", 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:10:37.292 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 48, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 162, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6808", 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.293 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6809", 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.294 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6811", 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6810", 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.295 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6808/1322208109", 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6809/1322208109", 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6811/1322208109", 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6810/1322208109", 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0", 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:10:37.296 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 58, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 149, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6812", 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.297 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6813", 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.298 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6815", 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6814", 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.299 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6812/2128678671", 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6813/2128678671", 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6815/2128678671", 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6814/2128678671", 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.300 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:10:18.475650+0000", 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.301 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:10:02.424982+0000", 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:09:46.508440+0000", 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.302 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:09:50.391357+0000", 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.303 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:09:55.670351+0000", 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:09:47.861354+0000", 2024-03-21T14:10:37.304 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:10:00.738672+0000", 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:10:37.305 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:09:57.253168+0000", 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [ 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.13", 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.306 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.0", 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.307 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.4", 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 5 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.b", 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.308 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.13", 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.309 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.1e", 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.0", 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:10:37.310 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.311 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.311 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.b", 2024-03-21T14:10:37.311 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.311 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.d", 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.312 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.13", 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_primaries": [], 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2024-03-21T14:10:37.313 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklist": { 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/45234419": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/384540368": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3883474825": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/1598547836": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3687155565": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3884352348": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2319227492": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2154949954": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/4162970769": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2292819669": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3411869253": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/520312891": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:10:37.314 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2267426691": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3211426112": "2024-03-22T13:57:56.576834+0000" 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "range_blocklist": {}, 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2024-03-21T14:10:37.315 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {}, 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": { 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_enabled": false, 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_bucket_count": 0, 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "degraded_stretch_mode": 0, 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "recovering_stretch_mode": 0, 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_bucket": 0 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:10:37.316 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:10:37.374 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:37.405 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:37.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:37.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:37.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:37.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:37.684 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:37.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:37.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:37.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 125536 125520 92.2707 69 0.0105302 0.0108331 2024-03-21T14:10:37.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:37.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:37.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:37.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:38.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:38.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4016073085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:10:38.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4016073085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:10:38.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:38.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4016073085' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:10:38.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:38.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:38.392 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:38.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:38.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:38.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:38.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.680 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:38.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:38.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:38.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:38.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 126619 126603 91.9848 67.6875 0.0130382 0.0108667 2024-03-21T14:10:38.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:38.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:38.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:39.063 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:39.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:39.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:38 smithi118 ceph-mon[33862]: pgmap v330: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:10:39.206 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:39.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:38 smithi003 ceph-mon[29576]: pgmap v330: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:10:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:38 smithi003 ceph-mon[40510]: pgmap v330: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-21T14:10:39.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:39.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:39.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:39.522 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:39.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:39.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:39.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:39.804 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:39.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:39.902 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 127700 127684 91.704 67.5625 0.0114832 0.0108996 2024-03-21T14:10:39.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:39.942 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:39.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:40.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:40.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:40.142 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:10:40.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:40.186 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:10:40.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:40.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:40.231 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:10:40.275 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:10:40.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:40.329 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:10:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:40.389 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:10:40.443 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:10:40.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:40.494 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:10:40.494 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:10:40.495 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:10:40.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:40.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:40.508 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:10:40.509 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:10:40.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:40.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:40.751 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:40.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:40.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:40.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 128689 128673 91.3644 61.8125 0.018385 0.0109405 2024-03-21T14:10:40.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:40.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:40 smithi003 ceph-mon[29576]: pgmap v331: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-21T14:10:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:40 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/977484999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:40 smithi003 ceph-mon[40510]: pgmap v331: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-21T14:10:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/977484999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:41.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:41.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:41.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:40 smithi118 ceph-mon[33862]: pgmap v331: 148 pgs: 148 active+clean; 7.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 960 op/s 2024-03-21T14:10:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:40 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/977484999' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:41.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:41.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:41.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:41.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:41.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:41.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:41.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:41.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:41.713 INFO:tasks.thrashosds.thrasher:j is {'epoch': 183, 'fsid': 'c60e082a-e78a-11ee-95cd-87774f69a715', 'created': '2024-03-21T13:56:54.530125+0000', 'modified': '2024-03-21T14:10:25.840077+0000', 'last_up_change': '2024-03-21T14:02:19.321454+0000', 'last_in_change': '2024-03-21T14:02:01.236118+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T14:00:12.925535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T14:02:31.373659+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T14:02:31.794831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T14:02:33.750394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T14:02:35.711498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T14:02:37.763124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-21T14:08:59.821602+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'daf9bc3c-4e73-4379-afb6-7ee181b686fb', 'up': 1, 'in': 1, 'weight': 0.73016357421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6801', 'nonce': 930096433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6802', 'nonce': 930096433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6804', 'nonce': 930096433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6803', 'nonce': 930096433}]}, 'public_addr': '172.21.15.3:6801/930096433', 'cluster_addr': '172.21.15.3:6802/930096433', 'heartbeat_back_addr': '172.21.15.3:6804/930096433', 'heartbeat_front_addr': '172.21.15.3:6803/930096433', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1652f24d-aa0e-471c-bf54-03814b1ee11d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6805', 'nonce': 1526485482}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6806', 'nonce': 1526485482}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6808', 'nonce': 1526485482}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6807', 'nonce': 1526485482}]}, 'public_addr': '172.21.15.3:6805/1526485482', 'cluster_addr': '172.21.15.3:6806/1526485482', 'heartbeat_back_addr': '172.21.15.3:6808/1526485482', 'heartbeat_front_addr': '172.21.15.3:6807/1526485482', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5e67d907-f8d1-4ecc-85d3-df88da5c364c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6809', 'nonce': 3751378277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6810', 'nonce': 3751378277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6812', 'nonce': 3751378277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6811', 'nonce': 3751378277}]}, 'public_addr': '172.21.15.3:6809/3751378277', 'cluster_addr': '172.21.15.3:6810/3751378277', 'heartbeat_back_addr': '172.21.15.3:6812/3751378277', 'heartbeat_front_addr': '172.21.15.3:6811/3751378277', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'ad8dabdf-fce5-4312-9e80-dbbb6ba2947c', 'up': 1, 'in': 1, 'weight': 0.6562042236328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6813', 'nonce': 1918209225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6814', 'nonce': 1918209225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6816', 'nonce': 1918209225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6815', 'nonce': 1918209225}]}, 'public_addr': '172.21.15.3:6813/1918209225', 'cluster_addr': '172.21.15.3:6814/1918209225', 'heartbeat_back_addr': '172.21.15.3:6816/1918209225', 'heartbeat_front_addr': '172.21.15.3:6815/1918209225', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b5d486b1-2cb6-46c7-8664-994af07ff7b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 296472230}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 296472230}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 296472230}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 296472230}]}, 'public_addr': '172.21.15.118:6800/296472230', 'cluster_addr': '172.21.15.118:6801/296472230', 'heartbeat_back_addr': '172.21.15.118:6803/296472230', 'heartbeat_front_addr': '172.21.15.118:6802/296472230', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bff24605-38ce-47ce-9142-4ced3766476e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 326385780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 326385780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 326385780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 326385780}]}, 'public_addr': '172.21.15.118:6804/326385780', 'cluster_addr': '172.21.15.118:6805/326385780', 'heartbeat_back_addr': '172.21.15.118:6807/326385780', 'heartbeat_front_addr': '172.21.15.118:6806/326385780', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '73a781e7-8660-4885-b976-2aeb8515109a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 162, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 1322208109}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 1322208109}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 1322208109}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 1322208109}]}, 'public_addr': '172.21.15.118:6808/1322208109', 'cluster_addr': '172.21.15.118:6809/1322208109', 'heartbeat_back_addr': '172.21.15.118:6811/1322208109', 'heartbeat_front_addr': '172.21.15.118:6810/1322208109', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3e95559b-79f6-40ba-9d98-d350bcdd60e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 149, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 2128678671}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 2128678671}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 2128678671}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 2128678671}]}, 'public_addr': '172.21.15.118:6812/2128678671', 'cluster_addr': '172.21.15.118:6813/2128678671', 'heartbeat_back_addr': '172.21.15.118:6815/2128678671', 'heartbeat_front_addr': '172.21.15.118:6814/2128678671', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:10:18.475650+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:10:02.424982+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:09:46.508440+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:09:50.391357+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:09:55.670351+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:09:47.861354+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:10:00.738672+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:09:57.253168+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.13', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.0', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.0', 'mappings': [{'from': 3, 'to': 6}]}, {'pgid': '6.b', 'mappings': [{'from': 3, 'to': 0}]}, {'pgid': '6.d', 'mappings': [{'from': 2, 'to': 6}]}, {'pgid': '6.13', 'mappings': [{'from': 2, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.3:0/45234419': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/384540368': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3883474825': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/1598547836': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:6800/3687155565': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3884352348': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2319227492': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2154949954': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/4162970769': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/2292819669': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/3411869253': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/520312891': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2267426691': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/3211426112': '2024-03-22T13:57:56.576834+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T14:10:41.713 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.b 2024-03-21T14:10:41.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd rm-pg-upmap-items 6.b 2024-03-21T14:10:41.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:41.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:41.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:41.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 129863 129847 91.1608 73.375 0.0150879 0.0109648 2024-03-21T14:10:41.904 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:41.946 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:41.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:41.967 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:10:42.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:42.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:42.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:10:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/613856025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:10:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:41 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:10:42.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:42.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/613856025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:10:42.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:41 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:10:42.218 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/613856025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:10:42.218 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:41 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:10:42.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:42.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:42.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:42.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:42.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:42.552 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:42.655 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:42.688 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:42.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:42.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:42.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.823 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:42.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0176 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 130914 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 90.8948 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 33.1007 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 211.875 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 56.0625 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1454 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 529.611 2024-03-21T14:10:42.904 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3390 2024-03-21T14:10:42.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 897 2024-03-21T14:10:42.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0109969 2024-03-21T14:10:42.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.00810413 2024-03-21T14:10:42.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.128614 2024-03-21T14:10:42.905 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00121156 2024-03-21T14:10:42.908 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T14:10:42.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:42.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:43.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:43.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: pgmap v332: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: osdmap e184: 8 total, 8 up, 8 in 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: 8.6 scrub starts 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:10:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:42 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:10:43.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: pgmap v332: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: osdmap e184: 8 total, 8 up, 8 in 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: 8.6 scrub starts 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: pgmap v332: 148 pgs: 148 active+clean; 7.6 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: osdmap e184: 8 total, 8 up, 8 in 2024-03-21T14:10:43.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: 8.6 scrub starts 2024-03-21T14:10:43.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:10:43.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:42 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:10:43.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:43.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.254 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:43.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:43.355 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:43.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:43.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:43.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:43.620 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:43.637 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:43.645 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.673 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:43.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:43.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:43.810 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:43.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:43.941 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:43.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:43.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:44.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:44.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:43 smithi003 ceph-mon[29576]: 8.0 scrub starts 2024-03-21T14:10:44.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:43 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:10:44.058 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:43 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:10:44.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:43 smithi003 ceph-mon[40510]: 8.0 scrub starts 2024-03-21T14:10:44.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:43 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:10:44.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:43 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:10:44.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:44.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:44.214 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:44.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:44.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:44.349 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:43 smithi118 ceph-mon[33862]: 8.0 scrub starts 2024-03-21T14:10:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:43 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:10:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:43 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:10:44.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:44.482 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:44.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:44.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:44.618 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:44.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:44.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:44.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:44.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:44.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:44.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:44.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:10:44.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:45.051 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:45.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:45.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:45.188 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:45.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[29576]: pgmap v334: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:10:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:10:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:10:45.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[40510]: pgmap v334: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:45.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:10:45.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:10:45.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:44 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:10:45.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:45.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:45.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:45.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:44 smithi118 ceph-mon[33862]: pgmap v334: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:45.555 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:44 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:10:45.555 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:44 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:10:45.555 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:44 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:10:45.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:45.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:10:45.586 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:45.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:45.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:45.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:45.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:45.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:45.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:45.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:45.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:46.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:46.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:10:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:10:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:10:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[29576]: 8.c scrub starts 2024-03-21T14:10:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:10:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:10:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:10:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:45 smithi003 ceph-mon[40510]: 8.c scrub starts 2024-03-21T14:10:46.268 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:46.297 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:46.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:46.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:46.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:45 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:10:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:45 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:10:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:45 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:10:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:45 smithi118 ceph-mon[33862]: 8.c scrub starts 2024-03-21T14:10:46.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:46.559 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:46.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:46.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:10:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:10:46.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:46.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:46.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:46.982 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:46.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:46.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:47.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:47.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:47.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: pgmap v335: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: 8.6 scrub ok 2024-03-21T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: 8.0 scrub ok 2024-03-21T14:10:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:47.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:46 smithi118 ceph-mon[33862]: 8.e scrub starts 2024-03-21T14:10:47.227 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: pgmap v335: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: 8.6 scrub ok 2024-03-21T14:10:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:10:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:10:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: 8.0 scrub ok 2024-03-21T14:10:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[29576]: 8.e scrub starts 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: pgmap v335: 148 pgs: 148 active+clean; 7.9 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: 8.6 scrub ok 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: 8.0 scrub ok 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:10:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:46 smithi003 ceph-mon[40510]: 8.e scrub starts 2024-03-21T14:10:47.260 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:47.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:47.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:47.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:47.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:47.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:47.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:47.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:47.801 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:47.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:47.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:47.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:47.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:48.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:48.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:48.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:48.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:48.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:10:48.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 8.f scrub starts 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[29576]: 8.c scrub ok 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 8.f scrub starts 2024-03-21T14:10:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:47 smithi003 ceph-mon[40510]: 8.c scrub ok 2024-03-21T14:10:48.323 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.353 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:48.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:10:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:10:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:10:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:10:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:10:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:10:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 8.f scrub starts 2024-03-21T14:10:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:47 smithi118 ceph-mon[33862]: 8.c scrub ok 2024-03-21T14:10:48.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:48.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:48.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:48.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:48.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:48.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:48.764 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:48.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:48.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:48.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:48.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[29576]: pgmap v336: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 1.04k op/s 2024-03-21T14:10:48.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:10:48.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:10:48.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:10:48.965 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:10:49.037 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:49.067 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:49.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:49.169 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:49.202 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:49.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[40510]: pgmap v336: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 1.04k op/s 2024-03-21T14:10:49.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:10:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:10:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:10:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:48 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:10:49.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:49.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:48 smithi118 ceph-mon[33862]: pgmap v336: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 1.04k op/s 2024-03-21T14:10:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:48 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:10:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:48 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:10:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:48 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:10:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:48 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:10:49.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.481 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:49.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:49.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:49.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.614 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:49.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:49.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:49.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:49.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:49.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:49.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:49.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:49.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:50.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.032 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:50.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:50.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:50.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:10:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:10:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:49 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:10:50.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:50.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:50.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:10:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:10:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:10:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:10:50.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:10:50.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:10:50.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:49 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:10:50.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.468 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:50.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:50.569 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.602 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:50.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:50.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:50.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:50.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:50.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:50.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:50.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:51.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:51.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:51.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:51.142 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:51.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: pgmap v337: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 1.04k op/s 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: 3.1a scrub starts 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: 3.1a scrub ok 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: 8.e scrub ok 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[29576]: 8.f scrub ok 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: pgmap v337: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 1.04k op/s 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: 3.1a scrub starts 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: 3.1a scrub ok 2024-03-21T14:10:51.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: 8.e scrub ok 2024-03-21T14:10:51.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:10:51.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:50 smithi003 ceph-mon[40510]: 8.f scrub ok 2024-03-21T14:10:51.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:51.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:51.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:51.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:51.425 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:51.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: pgmap v337: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.8 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 43 MiB/s wr, 1.04k op/s 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: 3.1a scrub starts 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: 3.1a scrub ok 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: 8.e scrub ok 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:10:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:50 smithi118 ceph-mon[33862]: 8.f scrub ok 2024-03-21T14:10:51.462 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:10:51.462 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:10:51.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:51.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:51.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:51.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:51.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:51.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:51.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:51.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:51.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:52.000 INFO:teuthology.orchestra.run.smithi003.stderr:clear 6.b pg_upmap_items mapping 2024-03-21T14:10:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:52.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:52.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:52.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:10:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:10:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:10:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:10:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:10:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4134016611' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]: dispatch 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]: dispatch 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:10:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:10:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:10:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:10:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:10:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:10:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4134016611' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]: dispatch 2024-03-21T14:10:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:51 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]: dispatch 2024-03-21T14:10:52.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:52.297 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:52.398 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.432 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:52.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:10:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:10:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:10:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:10:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:10:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:10:52.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:10:52.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:10:52.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:10:52.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4134016611' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]: dispatch 2024-03-21T14:10:52.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:51 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]: dispatch 2024-03-21T14:10:52.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:52.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:52.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:52.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:52.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:52.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:52.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:52.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:52.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:52.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:52.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:52.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: pgmap v338: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 1.03k op/s 2024-03-21T14:10:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]': finished 2024-03-21T14:10:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: osdmap e185: 8 total, 8 up, 8 in 2024-03-21T14:10:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:10:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:10:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: pgmap v338: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 1.03k op/s 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]': finished 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: osdmap e185: 8 total, 8 up, 8 in 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:10:52.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:52 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:10:53.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:53.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:53.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:53.392 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:53.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: pgmap v338: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 7.7 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 1.03k op/s 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "6.b"}]': finished 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: osdmap e185: 8 total, 8 up, 8 in 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:10:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:52 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:10:53.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:53.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:53.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:53.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:53.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:53.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:53.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:53.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:54.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:54.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.090 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:54.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:54.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:54.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:54 smithi003 ceph-mon[29576]: osdmap e186: 8 total, 8 up, 8 in 2024-03-21T14:10:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:54 smithi003 ceph-mon[29576]: 8.9 scrub starts 2024-03-21T14:10:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:54 smithi003 ceph-mon[29576]: pgmap v341: 148 pgs: 148 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-21T14:10:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:54 smithi003 ceph-mon[40510]: osdmap e186: 8 total, 8 up, 8 in 2024-03-21T14:10:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:54 smithi003 ceph-mon[40510]: 8.9 scrub starts 2024-03-21T14:10:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:54 smithi003 ceph-mon[40510]: pgmap v341: 148 pgs: 148 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-21T14:10:54.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:54.366 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:54.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:54.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:10:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:54 smithi118 ceph-mon[33862]: osdmap e186: 8 total, 8 up, 8 in 2024-03-21T14:10:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:54 smithi118 ceph-mon[33862]: 8.9 scrub starts 2024-03-21T14:10:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:54 smithi118 ceph-mon[33862]: pgmap v341: 148 pgs: 148 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 11 MiB/s wr, 1.26k op/s 2024-03-21T14:10:54.469 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:54.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:54.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:54.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:54.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:54.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:54.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:54.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:54.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:54.934 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:54.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:55.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:55.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:55.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:55.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.203 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:55.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:55.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:55.337 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:55.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:55.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:55.470 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:55.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:55.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:55.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:55.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:55.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:55.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[29576]: 8.9 scrub ok 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[40510]: 8.9 scrub ok 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:10:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:55 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:10:55.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:55.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:55.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:55 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:10:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:55 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:10:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:55 smithi118 ceph-mon[33862]: 8.9 scrub ok 2024-03-21T14:10:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:55 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:10:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:55 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:10:55.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:56.003 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:56.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:10:56.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:56.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:10:56.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:56.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:56.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:56.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:56.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:56.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:56.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:56.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:56.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:56.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:10:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:10:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:10:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:10:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[29576]: pgmap v342: 148 pgs: 148 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 632 op/s 2024-03-21T14:10:56.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:10:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:10:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:10:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:10:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:10:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:10:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:56 smithi003 ceph-mon[40510]: pgmap v342: 148 pgs: 148 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 632 op/s 2024-03-21T14:10:56.810 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:56.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:56 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:10:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:56 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:10:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:56 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:10:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:56 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:10:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:56 smithi118 ceph-mon[33862]: pgmap v342: 148 pgs: 148 active+clean; 7.5 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 632 op/s 2024-03-21T14:10:56.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:10:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:10:56.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:56.973 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:56.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:57.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:57.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:57.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:57.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:57.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:57.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:57.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.384 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:57.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:57.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:57.537 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:57.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:57.671 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:57.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:10:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:10:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:10:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:10:57.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:10:57.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:57.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:57.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:57.874 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:10:57.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:10:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:10:57.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:57.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:57.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:58.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:58.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:58.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:10:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:58.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:10:58.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:58.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.382 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:58.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:58.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:58.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:58.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:58.525 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:10:58.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:10:58.638 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:10:58.638 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:10:58.638 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:10:58.655 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:10:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:58.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:10:58.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:10:58.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:10:58.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: pgmap v343: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T14:10:58.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:10:58.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:10:58.703 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1675047728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:58.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4192369155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:58.704 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: pgmap v343: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1675047728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:58.705 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4192369155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:58.746 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:10:58.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:58.785 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:10:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:58.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:58.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:10:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:10:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:10:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:10:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:10:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:10:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:10:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:10:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:10:58.910 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: pgmap v343: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1675047728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:10:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4192369155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:10:58.946 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:58.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:59.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:59.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:59.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:59.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:59.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:59.307 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:10:59.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.343 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:59.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:10:59.444 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:10:59.477 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:10:59.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:10:59.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:59.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:10:59.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:10:59.756 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:10:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:10:59.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1675047728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: osdmap e187: 8 total, 8 up, 8 in 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:10:59.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1675047728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: osdmap e187: 8 total, 8 up, 8 in 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:10:59.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:10:59 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:10:59.819 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:10:59.819 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:10:59.820 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-03-21T14:10:59.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:10:59.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:10:59.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:10:59.924 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:10:59.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1675047728' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: osdmap e187: 8 total, 8 up, 8 in 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:10:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:10:59 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:11:00.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:00.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:00.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:00.208 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:00.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:00.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:00.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:00.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:00.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:00.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:00.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:00.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:00.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:00.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:00 smithi118 ceph-mon[33862]: pgmap v345: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 768 op/s 2024-03-21T14:11:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:00 smithi118 ceph-mon[33862]: 8.1 scrub starts 2024-03-21T14:11:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:00 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:11:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:00 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:11:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:00 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:11:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:00 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:11:00.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:00.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:00.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[29576]: pgmap v345: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 768 op/s 2024-03-21T14:11:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[29576]: 8.1 scrub starts 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[40510]: pgmap v345: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 768 op/s 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[40510]: 8.1 scrub starts 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:11:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:00 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:11:01.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:01.065 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:01.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:01.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:01.243 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:01.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:01.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:01.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:01.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:01.516 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:01.547 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:01.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:01.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:01.704 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:11:01.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:01.830 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:01.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:01.932 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:01.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:01.962 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:01.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:02.064 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:02.094 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:02.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:11:02.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:02.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:151360 8.1 8:85670c6c:::benchmark_data_smithi003_110242_object20443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.071917+0000", 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.02612604, 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026158921000000002, 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:11:02.270 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151360 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.071917+0000", 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.071917+0000", 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.071919+0000", 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.64e-06 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.071924+0000", 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5280000000000003e-06 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.071968+0000", 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3718999999999999e-05 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.071976+0000", 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7649999999999993e-06 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.075750+0000", 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00377363 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:02.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.075858+0000", 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108291 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.075986+0000", 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127768 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.079213+0000", 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032273520000000002 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.090710+0000", 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011496777999999999 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:151370 8.f e187/162 8:f035bfc2:::benchmark_data_smithi003_110242_object20453:head v 187'9378, mlcod=187'9376)", 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.086001+0000", 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.012042456999999999, 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012174256, 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:02.275 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086001+0000", 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086001+0000", 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086001+0000", 2024-03-21T14:11:02.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.58e-07 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086007+0000", 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6810000000000003e-06 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086014+0000", 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9140000000000002e-06 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086019+0000", 2024-03-21T14:11:02.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3149999999999999e-06 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086147+0000", 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128045 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.086165+0000", 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8114999999999999e-05 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:151376 8.0 e187/162 8:0b00e2cf:::benchmark_data_smithi003_110242_object20459:head v 187'9523, mlcod=187'9520)", 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.091234+0000", 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0068088239999999998, 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070046329999999997, 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091234+0000", 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.279 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091234+0000", 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091235+0000", 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7299999999999995e-07 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091241+0000", 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5600000000000001e-06 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.280 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091255+0000", 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4469e-05 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091263+0000", 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9920000000000007e-06 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091363+0000", 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9444000000000005e-05 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:02.281 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091384+0000", 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1284000000000002e-05 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:151372 8.f e187/162 8:ffecca3e:::benchmark_data_smithi003_110242_object20455:head v 187'9379, mlcod=187'9376)", 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.091282+0000", 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0067609590000000004, 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070181779999999999, 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.282 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091282+0000", 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091282+0000", 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091283+0000", 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5199999999999998e-07 2024-03-21T14:11:02.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091287+0000", 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6120000000000001e-06 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091294+0000", 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4899999999999997e-06 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.091299+0000", 2024-03-21T14:11:02.284 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4999999999999999e-06 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.093811+0000", 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025118599999999999 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.093833+0000", 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1413999999999999e-05 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15132.0:151360 8.1 e187/162)", 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.092228+0000", 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0058148619999999996, 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061323699999999998, 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092228+0000", 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.286 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092228+0000", 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092229+0000", 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2600000000000002e-07 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092234+0000", 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5970000000000002e-06 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.287 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092246+0000", 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2696e-05 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092254+0000", 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0700000000000007e-06 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:151383 8.7 8:e692a0cc:::benchmark_data_smithi003_110242_object20466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-21T14:11:02.288 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.092503+0000", 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0055403149999999996, 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059101220000000003, 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151383 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092503+0000", 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.289 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092503+0000", 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092504+0000", 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2589999999999999e-06 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092509+0000", 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6700000000000002e-06 2024-03-21T14:11:02.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092519+0000", 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0225e-05 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.092527+0000", 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4889999999999998e-06 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.097428+0000", 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049018000000000004 2024-03-21T14:11:02.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.097546+0000", 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011770200000000001 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,6", 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.097669+0000", 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122977 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.292 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:151379 8.0 e187/162 8:0b9ce83a:::benchmark_data_smithi003_110242_object20462:head v 187'9524, mlcod=187'9521)", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:02.096638+0000", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001405545, 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001851021, 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.096638+0000", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.096638+0000", 2024-03-21T14:11:02.293 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.096639+0000", 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.54e-07 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.096647+0000", 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8849999999999999e-06 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.096656+0000", 2024-03-21T14:11:02.294 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3870000000000001e-06 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.096663+0000", 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5129999999999999e-06 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.097790+0000", 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011266480000000001 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:02.295 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:02.097813+0000", 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2633000000000002e-05 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 7 2024-03-21T14:11:02.296 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:11:02.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.382 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:02.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:02.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:02.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: 6.1c scrub starts 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: 6.1c scrub ok 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: pgmap v346: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:11:02.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: 6.1c scrub starts 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: 6.1c scrub ok 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: pgmap v346: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:11:02.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:02 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:11:02.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.518 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:02.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:02.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:02.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.661 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:02.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:11:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:11:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: 6.1c scrub starts 2024-03-21T14:11:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: 6.1c scrub ok 2024-03-21T14:11:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:02.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:02.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: pgmap v346: 148 pgs: 148 active+clean; 7.1 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 760 op/s 2024-03-21T14:11:02.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:02.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:02.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:11:02.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:02 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:11:02.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:02.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:02.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:02.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:02.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:03.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:03.081 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:03.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:03.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:03.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:03.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:11:03.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:03.354 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:03.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:11:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:11:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 6.3 deep-scrub starts 2024-03-21T14:11:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 6.3 deep-scrub ok 2024-03-21T14:11:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 8.1 scrub ok 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: osdmap e188: 8 total, 8 up, 8 in 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[29576]: 8.3 scrub starts 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 6.3 deep-scrub starts 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 6.3 deep-scrub ok 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 8.1 scrub ok 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: osdmap e188: 8 total, 8 up, 8 in 2024-03-21T14:11:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:11:03.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:11:03.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:03 smithi003 ceph-mon[40510]: 8.3 scrub starts 2024-03-21T14:11:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:03.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:03.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:03.650 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 6.3 deep-scrub starts 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 6.3 deep-scrub ok 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 8.1 scrub ok 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: osdmap e188: 8 total, 8 up, 8 in 2024-03-21T14:11:03.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:11:03.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:11:03.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:03 smithi118 ceph-mon[33862]: 8.3 scrub starts 2024-03-21T14:11:03.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:03.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:03.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:03.938 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:03.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:04.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.075 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:04.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:04.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:04.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:04.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:04.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.345 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:04.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:04.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:04.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:04.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:04.646 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:04.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:04.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:04.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[29576]: osdmap e189: 8 total, 8 up, 8 in 2024-03-21T14:11:04.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[29576]: pgmap v349: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 944 op/s 2024-03-21T14:11:04.661 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:04.661 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:04.661 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[40510]: osdmap e189: 8 total, 8 up, 8 in 2024-03-21T14:11:04.661 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:04 smithi003 ceph-mon[40510]: pgmap v349: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 944 op/s 2024-03-21T14:11:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:04 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:04 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:04 smithi118 ceph-mon[33862]: osdmap e189: 8 total, 8 up, 8 in 2024-03-21T14:11:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:04 smithi118 ceph-mon[33862]: pgmap v349: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 944 op/s 2024-03-21T14:11:04.752 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:04.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:04.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:04.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:04.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:04.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:04.983 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:04.984 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:05.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:05.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:05.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.225 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:05.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:05.327 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:05.357 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:05.458 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:11:05.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:05.466 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:11:05.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.505 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:05.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:05.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:05 smithi003 ceph-mon[40510]: 8.4 scrub starts 2024-03-21T14:11:05.552 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:05 smithi003 ceph-mon[40510]: 8.3 scrub ok 2024-03-21T14:11:05.552 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:05 smithi003 ceph-mon[29576]: 8.4 scrub starts 2024-03-21T14:11:05.552 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:05 smithi003 ceph-mon[29576]: 8.3 scrub ok 2024-03-21T14:11:05.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:05.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:05.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:05 smithi118 ceph-mon[33862]: 8.4 scrub starts 2024-03-21T14:11:05.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:05 smithi118 ceph-mon[33862]: 8.3 scrub ok 2024-03-21T14:11:05.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:05.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:05.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:05.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:05.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:06.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:06.054 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:06.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:06.194 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:06.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:06.318 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-03-21T14:11:05.397695+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7268473280,"num_objects":111098,"num_object_clones":0,"num_object_copies":333294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":111098,"num_whiteouts":0,"num_read":1433,"num_read_kb":1279,"num_write":151545,"num_write_kb":8380958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":2363178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89679,"ondisk_log_size":89679,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23458928,"kb_used_data":21417192,"kb_used_omap":12,"kb_used_meta":2041587,"kb_avail":726502288,"statfs":{"total":767960285184,"available":743938342912,"internally_reserved":0,"allocated":21931204608,"data_stored":21928182576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2090585648},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3,6,4,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-371130368,"num_objects":-5663,"num_object_clones":0,"num_object_copies":-16989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5663,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5663,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.877297"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510431+0000","last_change":"2024-03-21T14:09:55.330094+0000","last_active":"2024-03-21T14:10:58.510431+0000","last_peered":"2024-03-21T14:10:58.510431+0000","last_clean":"2024-03-21T14:10:58.510431+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:10:58.510431+0000","last_undegraded":"2024-03-21T14:10:58.510431+0000","last_fullsized":"2024-03-21T14:10:58.510431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.329965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.329965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.949341+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522447+0000","last_change":"2024-03-21T14:09:02.129524+0000","last_active":"2024-03-21T14:10:58.522447+0000","last_peered":"2024-03-21T14:10:58.522447+0000","last_clean":"2024-03-21T14:10:58.522447+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:10:58.522447+0000","last_undegraded":"2024-03-21T14:10:58.522447+0000","last_fullsized":"2024-03-21T14:10:58.522447+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:02.129441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:30.718605+0000","last_clean_scrub_stamp":"2024-03-21T14:09:02.129441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":142,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516170+0000","last_change":"2024-03-21T14:10:46.826204+0000","last_active":"2024-03-21T14:10:58.516170+0000","last_peered":"2024-03-21T14:10:58.516170+0000","last_clean":"2024-03-21T14:10:58.516170+0000","last_became_active":"2024-03-21T14:07:23.349536+0000","last_became_peered":"2024-03-21T14:07:23.349536+0000","last_unstale":"2024-03-21T14:10:58.516170+0000","last_undegraded":"2024-03-21T14:10:58.516170+0000","last_fullsized":"2024-03-21T14:10:58.516170+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:46.826075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:46.826075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:11.568226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"72'9","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510500+0000","last_change":"2024-03-21T14:09:15.098201+0000","last_active":"2024-03-21T14:10:58.510500+0000","last_peered":"2024-03-21T14:10:58.510500+0000","last_clean":"2024-03-21T14:10:58.510500+0000","last_became_active":"2024-03-21T14:02:35.448372+0000","last_became_peered":"2024-03-21T14:02:35.448372+0000","last_unstale":"2024-03-21T14:10:58.510500+0000","last_undegraded":"2024-03-21T14:10:58.510500+0000","last_fullsized":"2024-03-21T14:10:58.510500+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:15.098048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:15.098048+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":143,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.520609+0000","last_change":"2024-03-21T14:09:59.259150+0000","last_active":"2024-03-21T14:10:58.520609+0000","last_peered":"2024-03-21T14:10:58.520609+0000","last_clean":"2024-03-21T14:10:58.520609+0000","last_became_active":"2024-03-21T14:07:23.355776+0000","last_became_peered":"2024-03-21T14:07:23.355776+0000","last_unstale":"2024-03-21T14:10:58.520609+0000","last_undegraded":"2024-03-21T14:10:58.520609+0000","last_fullsized":"2024-03-21T14:10:58.520609+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.259017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.259017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.072409+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510603+0000","last_change":"2024-03-21T14:10:03.258278+0000","last_active":"2024-03-21T14:10:58.510603+0000","last_peered":"2024-03-21T14:10:58.510603+0000","last_clean":"2024-03-21T14:10:58.510603+0000","last_became_active":"2024-03-21T14:02:37.457900+0000","last_became_peered":"2024-03-21T14:02:37.457900+0000","last_unstale":"2024-03-21T14:10:58.510603+0000","last_undegraded":"2024-03-21T14:10:58.510603+0000","last_fullsized":"2024-03-21T14:10:58.510603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.258168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.901138+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511483+0000","last_change":"2024-03-21T14:10:12.759166+0000","last_active":"2024-03-21T14:10:58.511483+0000","last_peered":"2024-03-21T14:10:58.511483+0000","last_clean":"2024-03-21T14:10:58.511483+0000","last_became_active":"2024-03-21T14:02:33.440235+0000","last_became_peered":"2024-03-21T14:02:33.440235+0000","last_unstale":"2024-03-21T14:10:58.511483+0000","last_undegraded":"2024-03-21T14:10:58.511483+0000","last_fullsized":"2024-03-21T14:10:58.511483+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:12.758536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.758536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.648088+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"83'22","reported_seq":318,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.509846+0000","last_change":"2024-03-21T14:10:06.702481+0000","last_active":"2024-03-21T14:10:58.509846+0000","last_peered":"2024-03-21T14:10:58.509846+0000","last_clean":"2024-03-21T14:10:58.509846+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:10:58.509846+0000","last_undegraded":"2024-03-21T14:10:58.509846+0000","last_fullsized":"2024-03-21T14:10:58.509846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'22","last_scrub_stamp":"2024-03-21T14:10:06.702330+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.702330+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:16.820634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":142,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.536748+0000","last_change":"2024-03-21T14:10:06.813347+0000","last_active":"2024-03-21T14:10:58.536748+0000","last_peered":"2024-03-21T14:10:58.536748+0000","last_clean":"2024-03-21T14:10:58.536748+0000","last_became_active":"2024-03-21T14:07:23.360834+0000","last_became_peered":"2024-03-21T14:07:23.360834+0000","last_unstale":"2024-03-21T14:10:58.536748+0000","last_undegraded":"2024-03-21T14:10:58.536748+0000","last_fullsized":"2024-03-21T14:10:58.536748+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:06.813027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.813027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:24.168419+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516578+0000","last_change":"2024-03-21T14:09:10.043497+0000","last_active":"2024-03-21T14:10:58.516578+0000","last_peered":"2024-03-21T14:10:58.516578+0000","last_clean":"2024-03-21T14:10:58.516578+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:10:58.516578+0000","last_undegraded":"2024-03-21T14:10:58.516578+0000","last_fullsized":"2024-03-21T14:10:58.516578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.043396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.043396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516438+0000","last_change":"2024-03-21T14:08:23.436895+0000","last_active":"2024-03-21T14:10:58.516438+0000","last_peered":"2024-03-21T14:10:58.516438+0000","last_clean":"2024-03-21T14:10:58.516438+0000","last_became_active":"2024-03-21T14:02:33.440512+0000","last_became_peered":"2024-03-21T14:02:33.440512+0000","last_unstale":"2024-03-21T14:10:58.516438+0000","last_undegraded":"2024-03-21T14:10:58.516438+0000","last_fullsized":"2024-03-21T14:10:58.516438+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:23.436768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:08:23.436768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"72'9","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510524+0000","last_change":"2024-03-21T14:10:46.814006+0000","last_active":"2024-03-21T14:10:58.510524+0000","last_peered":"2024-03-21T14:10:58.510524+0000","last_clean":"2024-03-21T14:10:58.510524+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:10:58.510524+0000","last_undegraded":"2024-03-21T14:10:58.510524+0000","last_fullsized":"2024-03-21T14:10:58.510524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:10:46.813854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:46.813854+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:56.137318+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515865+0000","last_change":"2024-03-21T14:10:51.000232+0000","last_active":"2024-03-21T14:10:58.515865+0000","last_peered":"2024-03-21T14:10:58.515865+0000","last_clean":"2024-03-21T14:10:58.515865+0000","last_became_active":"2024-03-21T14:07:23.350761+0000","last_became_peered":"2024-03-21T14:07:23.350761+0000","last_unstale":"2024-03-21T14:10:58.515865+0000","last_undegraded":"2024-03-21T14:10:58.515865+0000","last_fullsized":"2024-03-21T14:10:58.515865+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:51.000141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:51.000141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:18.201136+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":274,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.520700+0000","last_change":"2024-03-21T14:09:25.498525+0000","last_active":"2024-03-21T14:10:58.520700+0000","last_peered":"2024-03-21T14:10:58.520700+0000","last_clean":"2024-03-21T14:10:58.520700+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:10:58.520700+0000","last_undegraded":"2024-03-21T14:10:58.520700+0000","last_fullsized":"2024-03-21T14:10:58.520700+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.498388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.498388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":289,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516359+0000","last_change":"2024-03-21T14:09:58.390279+0000","last_active":"2024-03-21T14:10:58.516359+0000","last_peered":"2024-03-21T14:10:58.516359+0000","last_clean":"2024-03-21T14:10:58.516359+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:10:58.516359+0000","last_undegraded":"2024-03-21T14:10:58.516359+0000","last_fullsized":"2024-03-21T14:10:58.516359+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:58.390161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.390161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.872420+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"72'5","reported_seq":287,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511766+0000","last_change":"2024-03-21T14:09:26.168563+0000","last_active":"2024-03-21T14:10:58.511766+0000","last_peered":"2024-03-21T14:10:58.511766+0000","last_clean":"2024-03-21T14:10:58.511766+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:10:58.511766+0000","last_undegraded":"2024-03-21T14:10:58.511766+0000","last_fullsized":"2024-03-21T14:10:58.511766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'5","last_scrub_stamp":"2024-03-21T14:09:26.168400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.168400+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515666+0000","last_change":"2024-03-21T14:10:11.357570+0000","last_active":"2024-03-21T14:10:58.515666+0000","last_peered":"2024-03-21T14:10:58.515666+0000","last_clean":"2024-03-21T14:10:58.515666+0000","last_became_active":"2024-03-21T14:02:39.516011+0000","last_became_peered":"2024-03-21T14:02:39.516011+0000","last_unstale":"2024-03-21T14:10:58.515666+0000","last_undegraded":"2024-03-21T14:10:58.515666+0000","last_fullsized":"2024-03-21T14:10:58.515666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.357287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.357287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.442137+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.529846+0000","last_change":"2024-03-21T14:10:03.746595+0000","last_active":"2024-03-21T14:10:58.529846+0000","last_peered":"2024-03-21T14:10:58.529846+0000","last_clean":"2024-03-21T14:10:58.529846+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:10:58.529846+0000","last_undegraded":"2024-03-21T14:10:58.529846+0000","last_fullsized":"2024-03-21T14:10:58.529846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.746480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.746480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.495837+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":134,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511682+0000","last_change":"2024-03-21T14:09:19.179886+0000","last_active":"2024-03-21T14:10:58.511682+0000","last_peered":"2024-03-21T14:10:58.511682+0000","last_clean":"2024-03-21T14:10:58.511682+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:10:58.511682+0000","last_undegraded":"2024-03-21T14:10:58.511682+0000","last_fullsized":"2024-03-21T14:10:58.511682+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:19.179692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:19.179692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"72'15","reported_seq":327,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.527358+0000","last_change":"2024-03-21T14:10:47.604495+0000","last_active":"2024-03-21T14:10:58.527358+0000","last_peered":"2024-03-21T14:10:58.527358+0000","last_clean":"2024-03-21T14:10:58.527358+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:10:58.527358+0000","last_undegraded":"2024-03-21T14:10:58.527358+0000","last_fullsized":"2024-03-21T14:10:58.527358+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:10:47.604355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:47.604355+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:55.255932+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":279,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.513919+0000","last_change":"2024-03-21T14:10:55.043514+0000","last_active":"2024-03-21T14:10:58.513919+0000","last_peered":"2024-03-21T14:10:58.513919+0000","last_clean":"2024-03-21T14:10:58.513919+0000","last_became_active":"2024-03-21T14:02:39.517059+0000","last_became_peered":"2024-03-21T14:02:39.517059+0000","last_unstale":"2024-03-21T14:10:58.513919+0000","last_undegraded":"2024-03-21T14:10:58.513919+0000","last_fullsized":"2024-03-21T14:10:58.513919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:55.043404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.043404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:24.594672+0000","scrub_duration":121,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.512132+0000","last_change":"2024-03-21T14:10:04.249646+0000","last_active":"2024-03-21T14:10:58.512132+0000","last_peered":"2024-03-21T14:10:58.512132+0000","last_clean":"2024-03-21T14:10:58.512132+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:10:58.512132+0000","last_undegraded":"2024-03-21T14:10:58.512132+0000","last_fullsized":"2024-03-21T14:10:58.512132+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:04.249513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:04.249513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:18.620999+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":289,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510589+0000","last_change":"2024-03-21T14:10:49.867066+0000","last_active":"2024-03-21T14:10:58.510589+0000","last_peered":"2024-03-21T14:10:58.510589+0000","last_clean":"2024-03-21T14:10:58.510589+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:10:58.510589+0000","last_undegraded":"2024-03-21T14:10:58.510589+0000","last_fullsized":"2024-03-21T14:10:58.510589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:49.866989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.866989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:15.751214+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"72'12","reported_seq":309,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:00.963590+0000","last_change":"2024-03-21T14:11:00.963645+0000","last_active":"2024-03-21T14:11:00.963590+0000","last_peered":"2024-03-21T14:11:00.963590+0000","last_clean":"2024-03-21T14:11:00.963590+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:11:00.963590+0000","last_undegraded":"2024-03-21T14:11:00.963590+0000","last_fullsized":"2024-03-21T14:11:00.963590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:11:00.963504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:11:00.963504+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:11.556738+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":288,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.054248+0000","last_change":"2024-03-21T14:11:01.054286+0000","last_active":"2024-03-21T14:11:01.054248+0000","last_peered":"2024-03-21T14:11:01.054248+0000","last_clean":"2024-03-21T14:11:01.054248+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:11:01.054248+0000","last_undegraded":"2024-03-21T14:11:01.054248+0000","last_fullsized":"2024-03-21T14:11:01.054248+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:11:01.054164+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:09:24.494516+0000","last_clean_scrub_stamp":"2024-03-21T14:11:01.054164+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:15.630047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":398,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514195+0000","last_change":"2024-03-21T14:09:58.786565+0000","last_active":"2024-03-21T14:10:58.514195+0000","last_peered":"2024-03-21T14:10:58.514195+0000","last_clean":"2024-03-21T14:10:58.514195+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:10:58.514195+0000","last_undegraded":"2024-03-21T14:10:58.514195+0000","last_fullsized":"2024-03-21T14:10:58.514195+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:58.786336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.786336+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.024325+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":298,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522346+0000","last_change":"2024-03-21T14:09:09.213981+0000","last_active":"2024-03-21T14:10:58.522346+0000","last_peered":"2024-03-21T14:10:58.522346+0000","last_clean":"2024-03-21T14:10:58.522346+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:10:58.522346+0000","last_undegraded":"2024-03-21T14:10:58.522346+0000","last_fullsized":"2024-03-21T14:10:58.522346+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:09:09.213878+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.213878+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"72'10","reported_seq":346,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530598+0000","last_change":"2024-03-21T14:10:44.049053+0000","last_active":"2024-03-21T14:10:58.530598+0000","last_peered":"2024-03-21T14:10:58.530598+0000","last_clean":"2024-03-21T14:10:58.530598+0000","last_became_active":"2024-03-21T14:08:17.144672+0000","last_became_peered":"2024-03-21T14:08:17.144672+0000","last_unstale":"2024-03-21T14:10:58.530598+0000","last_undegraded":"2024-03-21T14:10:58.530598+0000","last_fullsized":"2024-03-21T14:10:58.530598+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'10","last_scrub_stamp":"2024-03-21T14:10:44.048668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:44.048668+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:11.435143+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.529564+0000","last_change":"2024-03-21T14:09:59.790745+0000","last_active":"2024-03-21T14:10:58.529564+0000","last_peered":"2024-03-21T14:10:58.529564+0000","last_clean":"2024-03-21T14:10:58.529564+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:10:58.529564+0000","last_undegraded":"2024-03-21T14:10:58.529564+0000","last_fullsized":"2024-03-21T14:10:58.529564+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.790627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.790627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.885224+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.531006+0000","last_change":"2024-03-21T14:10:07.843704+0000","last_active":"2024-03-21T14:10:58.531006+0000","last_peered":"2024-03-21T14:10:58.531006+0000","last_clean":"2024-03-21T14:10:58.531006+0000","last_became_active":"2024-03-21T14:02:37.463015+0000","last_became_peered":"2024-03-21T14:02:37.463015+0000","last_unstale":"2024-03-21T14:10:58.531006+0000","last_undegraded":"2024-03-21T14:10:58.531006+0000","last_fullsized":"2024-03-21T14:10:58.531006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.839111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.839111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:08.766204+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514068+0000","last_change":"2024-03-21T14:10:05.652480+0000","last_active":"2024-03-21T14:10:58.514068+0000","last_peered":"2024-03-21T14:10:58.514068+0000","last_clean":"2024-03-21T14:10:58.514068+0000","last_became_active":"2024-03-21T14:07:23.364680+0000","last_became_peered":"2024-03-21T14:07:23.364680+0000","last_unstale":"2024-03-21T14:10:58.514068+0000","last_undegraded":"2024-03-21T14:10:58.514068+0000","last_fullsized":"2024-03-21T14:10:58.514068+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:05.652409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:10:05.652409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.074656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"127'16","reported_seq":318,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514071+0000","last_change":"2024-03-21T14:10:52.061645+0000","last_active":"2024-03-21T14:10:58.514071+0000","last_peered":"2024-03-21T14:10:58.514071+0000","last_clean":"2024-03-21T14:10:58.514071+0000","last_became_active":"2024-03-21T14:02:35.449129+0000","last_became_peered":"2024-03-21T14:02:35.449129+0000","last_unstale":"2024-03-21T14:10:58.514071+0000","last_undegraded":"2024-03-21T14:10:58.514071+0000","last_fullsized":"2024-03-21T14:10:58.514071+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"127'16","last_scrub_stamp":"2024-03-21T14:10:52.061506+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:10:52.061506+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:52.804797+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.520600+0000","last_change":"2024-03-21T14:10:07.201542+0000","last_active":"2024-03-21T14:10:58.520600+0000","last_peered":"2024-03-21T14:10:58.520600+0000","last_clean":"2024-03-21T14:10:58.520600+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:10:58.520600+0000","last_undegraded":"2024-03-21T14:10:58.520600+0000","last_fullsized":"2024-03-21T14:10:58.520600+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.201435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.201435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:12.168125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.517525+0000","last_change":"2024-03-21T14:10:07.788804+0000","last_active":"2024-03-21T14:10:58.517525+0000","last_peered":"2024-03-21T14:10:58.517525+0000","last_clean":"2024-03-21T14:10:58.517525+0000","last_became_active":"2024-03-21T14:02:37.454240+0000","last_became_peered":"2024-03-21T14:02:37.454240+0000","last_unstale":"2024-03-21T14:10:58.517525+0000","last_undegraded":"2024-03-21T14:10:58.517525+0000","last_fullsized":"2024-03-21T14:10:58.517525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.788680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.788680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:24.873673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":298,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.531002+0000","last_change":"2024-03-21T14:10:12.855709+0000","last_active":"2024-03-21T14:10:58.531002+0000","last_peered":"2024-03-21T14:10:58.531002+0000","last_clean":"2024-03-21T14:10:58.531002+0000","last_became_active":"2024-03-21T14:02:33.442641+0000","last_became_peered":"2024-03-21T14:02:33.442641+0000","last_unstale":"2024-03-21T14:10:58.531002+0000","last_undegraded":"2024-03-21T14:10:58.531002+0000","last_fullsized":"2024-03-21T14:10:58.531002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:12.855582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.855582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:39.565947+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"72'4","reported_seq":292,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514161+0000","last_change":"2024-03-21T14:10:07.737599+0000","last_active":"2024-03-21T14:10:58.514161+0000","last_peered":"2024-03-21T14:10:58.514161+0000","last_clean":"2024-03-21T14:10:58.514161+0000","last_became_active":"2024-03-21T14:02:35.448435+0000","last_became_peered":"2024-03-21T14:02:35.448435+0000","last_unstale":"2024-03-21T14:10:58.514161+0000","last_undegraded":"2024-03-21T14:10:58.514161+0000","last_fullsized":"2024-03-21T14:10:58.514161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:10:07.737472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.737472+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:10.501380+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510882+0000","last_change":"2024-03-21T14:10:13.841129+0000","last_active":"2024-03-21T14:10:58.510882+0000","last_peered":"2024-03-21T14:10:58.510882+0000","last_clean":"2024-03-21T14:10:58.510882+0000","last_became_active":"2024-03-21T14:08:17.194651+0000","last_became_peered":"2024-03-21T14:08:17.194651+0000","last_unstale":"2024-03-21T14:10:58.510882+0000","last_undegraded":"2024-03-21T14:10:58.510882+0000","last_fullsized":"2024-03-21T14:10:58.510882+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:10:13.840591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:14.102381+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521954+0000","last_change":"2024-03-21T14:10:58.117415+0000","last_active":"2024-03-21T14:10:58.521954+0000","last_peered":"2024-03-21T14:10:58.521954+0000","last_clean":"2024-03-21T14:10:58.521954+0000","last_became_active":"2024-03-21T14:02:37.455621+0000","last_became_peered":"2024-03-21T14:02:37.455621+0000","last_unstale":"2024-03-21T14:10:58.521954+0000","last_undegraded":"2024-03-21T14:10:58.521954+0000","last_fullsized":"2024-03-21T14:10:58.521954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:58.117216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:58.117216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:05.790874+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":295,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515743+0000","last_change":"2024-03-21T14:09:55.427714+0000","last_active":"2024-03-21T14:10:58.515743+0000","last_peered":"2024-03-21T14:10:58.515743+0000","last_clean":"2024-03-21T14:10:58.515743+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:10:58.515743+0000","last_undegraded":"2024-03-21T14:10:58.515743+0000","last_fullsized":"2024-03-21T14:10:58.515743+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:55.427563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.427563+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.673618+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"72'11","reported_seq":305,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.509729+0000","last_change":"2024-03-21T14:09:56.592384+0000","last_active":"2024-03-21T14:10:58.509729+0000","last_peered":"2024-03-21T14:10:58.509729+0000","last_clean":"2024-03-21T14:10:58.509729+0000","last_became_active":"2024-03-21T14:02:35.773661+0000","last_became_peered":"2024-03-21T14:02:35.773661+0000","last_unstale":"2024-03-21T14:10:58.509729+0000","last_undegraded":"2024-03-21T14:10:58.509729+0000","last_fullsized":"2024-03-21T14:10:58.509729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:56.592242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.592242+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.889308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.528655+0000","last_change":"2024-03-21T14:10:50.076749+0000","last_active":"2024-03-21T14:10:58.528655+0000","last_peered":"2024-03-21T14:10:58.528655+0000","last_clean":"2024-03-21T14:10:58.528655+0000","last_became_active":"2024-03-21T14:02:39.470684+0000","last_became_peered":"2024-03-21T14:02:39.470684+0000","last_unstale":"2024-03-21T14:10:58.528655+0000","last_undegraded":"2024-03-21T14:10:58.528655+0000","last_fullsized":"2024-03-21T14:10:58.528655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:50.076608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:50.076608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:59.279329+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510803+0000","last_change":"2024-03-21T14:10:57.941052+0000","last_active":"2024-03-21T14:10:58.510803+0000","last_peered":"2024-03-21T14:10:58.510803+0000","last_clean":"2024-03-21T14:10:58.510803+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:10:58.510803+0000","last_undegraded":"2024-03-21T14:10:58.510803+0000","last_fullsized":"2024-03-21T14:10:58.510803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:57.940907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:10:57.940907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:14.290145+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.524257+0000","last_change":"2024-03-21T14:10:02.141197+0000","last_active":"2024-03-21T14:10:58.524257+0000","last_peered":"2024-03-21T14:10:58.524257+0000","last_clean":"2024-03-21T14:10:58.524257+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:10:58.524257+0000","last_undegraded":"2024-03-21T14:10:58.524257+0000","last_fullsized":"2024-03-21T14:10:58.524257+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.140845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.140845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:32.006676+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"72'9","reported_seq":309,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.528326+0000","last_change":"2024-03-21T14:10:47.963385+0000","last_active":"2024-03-21T14:10:58.528326+0000","last_peered":"2024-03-21T14:10:58.528326+0000","last_clean":"2024-03-21T14:10:58.528326+0000","last_became_active":"2024-03-21T14:02:35.450591+0000","last_became_peered":"2024-03-21T14:02:35.450591+0000","last_unstale":"2024-03-21T14:10:58.528326+0000","last_undegraded":"2024-03-21T14:10:58.528326+0000","last_fullsized":"2024-03-21T14:10:58.528326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:10:47.963086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:47.963086+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:02.758574+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":143,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.540140+0000","last_change":"2024-03-21T14:10:10.915236+0000","last_active":"2024-03-21T14:10:58.540140+0000","last_peered":"2024-03-21T14:10:58.540140+0000","last_clean":"2024-03-21T14:10:58.540140+0000","last_became_active":"2024-03-21T14:07:23.360741+0000","last_became_peered":"2024-03-21T14:07:23.360741+0000","last_unstale":"2024-03-21T14:10:58.540140+0000","last_undegraded":"2024-03-21T14:10:58.540140+0000","last_fullsized":"2024-03-21T14:10:58.540140+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:10.915115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:10.915115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:39.408472+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":287,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.528130+0000","last_change":"2024-03-21T14:09:26.436609+0000","last_active":"2024-03-21T14:10:58.528130+0000","last_peered":"2024-03-21T14:10:58.528130+0000","last_clean":"2024-03-21T14:10:58.528130+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:10:58.528130+0000","last_undegraded":"2024-03-21T14:10:58.528130+0000","last_fullsized":"2024-03-21T14:10:58.528130+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:26.436326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.436326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511930+0000","last_change":"2024-03-21T14:10:42.706283+0000","last_active":"2024-03-21T14:10:58.511930+0000","last_peered":"2024-03-21T14:10:58.511930+0000","last_clean":"2024-03-21T14:10:58.511930+0000","last_became_active":"2024-03-21T14:02:33.444582+0000","last_became_peered":"2024-03-21T14:02:33.444582+0000","last_unstale":"2024-03-21T14:10:58.511930+0000","last_undegraded":"2024-03-21T14:10:58.511930+0000","last_fullsized":"2024-03-21T14:10:58.511930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:42.706023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:42.706023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:09.226013+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"153'16","reported_seq":327,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511861+0000","last_change":"2024-03-21T14:10:54.800377+0000","last_active":"2024-03-21T14:10:58.511861+0000","last_peered":"2024-03-21T14:10:58.511861+0000","last_clean":"2024-03-21T14:10:58.511861+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:10:58.511861+0000","last_undegraded":"2024-03-21T14:10:58.511861+0000","last_fullsized":"2024-03-21T14:10:58.511861+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-03-21T14:10:54.800107+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:10:54.800107+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:17.303709+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":135,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522272+0000","last_change":"2024-03-21T14:09:10.489473+0000","last_active":"2024-03-21T14:10:58.522272+0000","last_peered":"2024-03-21T14:10:58.522272+0000","last_clean":"2024-03-21T14:10:58.522272+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:10:58.522272+0000","last_undegraded":"2024-03-21T14:10:58.522272+0000","last_fullsized":"2024-03-21T14:10:58.522272+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:10.489356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:10.489356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":399,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516578+0000","last_change":"2024-03-21T14:10:43.808848+0000","last_active":"2024-03-21T14:10:58.516578+0000","last_peered":"2024-03-21T14:10:58.516578+0000","last_clean":"2024-03-21T14:10:58.516578+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:10:58.516578+0000","last_undegraded":"2024-03-21T14:10:58.516578+0000","last_fullsized":"2024-03-21T14:10:58.516578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:10:43.808733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:43.808733+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:07.324451+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.520709+0000","last_change":"2024-03-21T14:10:52.119201+0000","last_active":"2024-03-21T14:10:58.520709+0000","last_peered":"2024-03-21T14:10:58.520709+0000","last_clean":"2024-03-21T14:10:58.520709+0000","last_became_active":"2024-03-21T14:08:17.142668+0000","last_became_peered":"2024-03-21T14:08:17.142668+0000","last_unstale":"2024-03-21T14:10:58.520709+0000","last_undegraded":"2024-03-21T14:10:58.520709+0000","last_fullsized":"2024-03-21T14:10:58.520709+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:52.119077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:10:52.119077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:52.796940+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"180'15","reported_seq":298,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511325+0000","last_change":"2024-03-21T14:10:09.765207+0000","last_active":"2024-03-21T14:10:58.511325+0000","last_peered":"2024-03-21T14:10:58.511325+0000","last_clean":"2024-03-21T14:10:58.511325+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:10:58.511325+0000","last_undegraded":"2024-03-21T14:10:58.511325+0000","last_fullsized":"2024-03-21T14:10:58.511325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'15","last_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:10:09.765058+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.504477+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511875+0000","last_change":"2024-03-21T14:09:17.192133+0000","last_active":"2024-03-21T14:10:58.511875+0000","last_peered":"2024-03-21T14:10:58.511875+0000","last_clean":"2024-03-21T14:10:58.511875+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:10:58.511875+0000","last_undegraded":"2024-03-21T14:10:58.511875+0000","last_fullsized":"2024-03-21T14:10:58.511875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:17.192018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.192018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"130'9","reported_seq":216,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521762+0000","last_change":"2024-03-21T14:10:00.156687+0000","last_active":"2024-03-21T14:10:58.521762+0000","last_peered":"2024-03-21T14:10:58.521762+0000","last_clean":"2024-03-21T14:10:58.521762+0000","last_became_active":"2024-03-21T14:07:23.354945+0000","last_became_peered":"2024-03-21T14:07:23.354945+0000","last_unstale":"2024-03-21T14:10:58.521762+0000","last_undegraded":"2024-03-21T14:10:58.521762+0000","last_fullsized":"2024-03-21T14:10:58.521762+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:10:00.156602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:00.156602+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.646196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":304,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.531709+0000","last_change":"2024-03-21T14:10:58.048691+0000","last_active":"2024-03-21T14:10:58.531709+0000","last_peered":"2024-03-21T14:10:58.531709+0000","last_clean":"2024-03-21T14:10:58.531709+0000","last_became_active":"2024-03-21T14:07:23.358188+0000","last_became_peered":"2024-03-21T14:07:23.358188+0000","last_unstale":"2024-03-21T14:10:58.531709+0000","last_undegraded":"2024-03-21T14:10:58.531709+0000","last_fullsized":"2024-03-21T14:10:58.531709+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:58.048573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:58.048573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:03.729233+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"72'9","reported_seq":293,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516891+0000","last_change":"2024-03-21T14:09:11.022485+0000","last_active":"2024-03-21T14:10:58.516891+0000","last_peered":"2024-03-21T14:10:58.516891+0000","last_clean":"2024-03-21T14:10:58.516891+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:10:58.516891+0000","last_undegraded":"2024-03-21T14:10:58.516891+0000","last_fullsized":"2024-03-21T14:10:58.516891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:11.022384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.022384+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":286,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.527655+0000","last_change":"2024-03-21T14:10:46.580338+0000","last_active":"2024-03-21T14:10:58.527655+0000","last_peered":"2024-03-21T14:10:58.527655+0000","last_clean":"2024-03-21T14:10:58.527655+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:10:58.527655+0000","last_undegraded":"2024-03-21T14:10:58.527655+0000","last_fullsized":"2024-03-21T14:10:58.527655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:46.580264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:46.580264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:05.149663+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":134,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.534515+0000","last_change":"2024-03-21T14:09:11.232306+0000","last_active":"2024-03-21T14:10:58.534515+0000","last_peered":"2024-03-21T14:10:58.534515+0000","last_clean":"2024-03-21T14:10:58.534515+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:10:58.534515+0000","last_undegraded":"2024-03-21T14:10:58.534515+0000","last_fullsized":"2024-03-21T14:10:58.534515+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.232197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.232197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521377+0000","last_change":"2024-03-21T14:09:55.186771+0000","last_active":"2024-03-21T14:10:58.521377+0000","last_peered":"2024-03-21T14:10:58.521377+0000","last_clean":"2024-03-21T14:10:58.521377+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:10:58.521377+0000","last_undegraded":"2024-03-21T14:10:58.521377+0000","last_fullsized":"2024-03-21T14:10:58.521377+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:55.186641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.186641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"72'9","reported_seq":310,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530708+0000","last_change":"2024-03-21T14:10:49.055663+0000","last_active":"2024-03-21T14:10:58.530708+0000","last_peered":"2024-03-21T14:10:58.530708+0000","last_clean":"2024-03-21T14:10:58.530708+0000","last_became_active":"2024-03-21T14:02:35.774637+0000","last_became_peered":"2024-03-21T14:02:35.774637+0000","last_unstale":"2024-03-21T14:10:58.530708+0000","last_undegraded":"2024-03-21T14:10:58.530708+0000","last_fullsized":"2024-03-21T14:10:58.530708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:10:49.055508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.055508+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:00.544288+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.524138+0000","last_change":"2024-03-21T14:10:51.156733+0000","last_active":"2024-03-21T14:10:58.524138+0000","last_peered":"2024-03-21T14:10:58.524138+0000","last_clean":"2024-03-21T14:10:58.524138+0000","last_became_active":"2024-03-21T14:02:39.517582+0000","last_became_peered":"2024-03-21T14:02:39.517582+0000","last_unstale":"2024-03-21T14:10:58.524138+0000","last_undegraded":"2024-03-21T14:10:58.524138+0000","last_fullsized":"2024-03-21T14:10:58.524138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:51.156629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:51.156629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:02.484084+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515640+0000","last_change":"2024-03-21T14:10:09.507069+0000","last_active":"2024-03-21T14:10:58.515640+0000","last_peered":"2024-03-21T14:10:58.515640+0000","last_clean":"2024-03-21T14:10:58.515640+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:10:58.515640+0000","last_undegraded":"2024-03-21T14:10:58.515640+0000","last_fullsized":"2024-03-21T14:10:58.515640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:10:09.506940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.590201+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":303,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.513283+0000","last_change":"2024-03-21T14:10:48.999582+0000","last_active":"2024-03-21T14:10:58.513283+0000","last_peered":"2024-03-21T14:10:58.513283+0000","last_clean":"2024-03-21T14:10:58.513283+0000","last_became_active":"2024-03-21T14:07:23.354457+0000","last_became_peered":"2024-03-21T14:07:23.354457+0000","last_unstale":"2024-03-21T14:10:58.513283+0000","last_undegraded":"2024-03-21T14:10:58.513283+0000","last_fullsized":"2024-03-21T14:10:58.513283+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:48.999469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:48.999469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:08.786374+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"72'6","reported_seq":179,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:59.133643+0000","last_change":"2024-03-21T14:10:59.133691+0000","last_active":"2024-03-21T14:10:59.133643+0000","last_peered":"2024-03-21T14:10:59.133643+0000","last_clean":"2024-03-21T14:10:59.133643+0000","last_became_active":"2024-03-21T14:07:23.365278+0000","last_became_peered":"2024-03-21T14:07:23.365278+0000","last_unstale":"2024-03-21T14:10:59.133643+0000","last_undegraded":"2024-03-21T14:10:59.133643+0000","last_fullsized":"2024-03-21T14:10:59.133643+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'6","last_scrub_stamp":"2024-03-21T14:10:59.133557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:59.133557+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:11.194763+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"187'9448","reported_seq":9507,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.377297+0000","last_change":"2024-03-21T14:09:01.659122+0000","last_active":"2024-03-21T14:11:01.377297+0000","last_peered":"2024-03-21T14:11:01.377297+0000","last_clean":"2024-03-21T14:11:01.377297+0000","last_became_active":"2024-03-21T14:09:01.658623+0000","last_became_peered":"2024-03-21T14:09:01.658623+0000","last_unstale":"2024-03-21T14:11:01.377297+0000","last_undegraded":"2024-03-21T14:11:01.377297+0000","last_fullsized":"2024-03-21T14:11:01.377297+0000","mapping_epoch":162,"log_start":"171'4400","ondisk_log_start":"171'4400","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5048,"log_dups_size":0,"ondisk_log_size":5048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454950912,"num_objects":6942,"num_object_clones":0,"num_object_copies":20826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9448,"num_write_kb":524480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:59.984190+0000","last_change":"2024-03-21T14:10:59.984215+0000","last_active":"2024-03-21T14:10:59.984190+0000","last_peered":"2024-03-21T14:10:59.984190+0000","last_clean":"2024-03-21T14:10:59.984190+0000","last_became_active":"2024-03-21T14:02:39.516934+0000","last_became_peered":"2024-03-21T14:02:39.516934+0000","last_unstale":"2024-03-21T14:10:59.984190+0000","last_undegraded":"2024-03-21T14:10:59.984190+0000","last_fullsized":"2024-03-21T14:10:59.984190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:59.984131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:10:59.984131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:10.227256+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":314,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522204+0000","last_change":"2024-03-21T14:10:08.205353+0000","last_active":"2024-03-21T14:10:58.522204+0000","last_peered":"2024-03-21T14:10:58.522204+0000","last_clean":"2024-03-21T14:10:58.522204+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:10:58.522204+0000","last_undegraded":"2024-03-21T14:10:58.522204+0000","last_fullsized":"2024-03-21T14:10:58.522204+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:10:08.205222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:08.205222+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:25.723255+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"72'15","reported_seq":310,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515621+0000","last_change":"2024-03-21T14:10:11.474876+0000","last_active":"2024-03-21T14:10:58.515621+0000","last_peered":"2024-03-21T14:10:58.515621+0000","last_clean":"2024-03-21T14:10:58.515621+0000","last_became_active":"2024-03-21T14:02:35.775105+0000","last_became_peered":"2024-03-21T14:02:35.775105+0000","last_unstale":"2024-03-21T14:10:58.515621+0000","last_undegraded":"2024-03-21T14:10:58.515621+0000","last_fullsized":"2024-03-21T14:10:58.515621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:10:11.474732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.474732+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.236235+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"72'8","reported_seq":392,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:00.080816+0000","last_change":"2024-03-21T14:11:00.080855+0000","last_active":"2024-03-21T14:11:00.080816+0000","last_peered":"2024-03-21T14:11:00.080816+0000","last_clean":"2024-03-21T14:11:00.080816+0000","last_became_active":"2024-03-21T14:02:37.454610+0000","last_became_peered":"2024-03-21T14:02:37.454610+0000","last_unstale":"2024-03-21T14:11:00.080816+0000","last_undegraded":"2024-03-21T14:11:00.080816+0000","last_fullsized":"2024-03-21T14:11:00.080816+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:11:00.080732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:11:00.080732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:29.659972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"187'9321","reported_seq":9450,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.373297+0000","last_change":"2024-03-21T14:10:12.650114+0000","last_active":"2024-03-21T14:11:01.373297+0000","last_peered":"2024-03-21T14:11:01.373297+0000","last_clean":"2024-03-21T14:11:01.373297+0000","last_became_active":"2024-03-21T14:09:01.656279+0000","last_became_peered":"2024-03-21T14:09:01.656279+0000","last_unstale":"2024-03-21T14:11:01.373297+0000","last_undegraded":"2024-03-21T14:11:01.373297+0000","last_fullsized":"2024-03-21T14:11:01.373297+0000","mapping_epoch":162,"log_start":"169'3700","ondisk_log_start":"169'3700","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'5980","last_scrub_stamp":"2024-03-21T14:10:12.649926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.649926+0000","objects_scrubbed":5922,"log_size":5621,"log_dups_size":0,"ondisk_log_size":5621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:13.781877+0000","scrub_duration":1885,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447938560,"num_objects":6835,"num_object_clones":0,"num_object_copies":20505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9321,"num_write_kb":516992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":13,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510199+0000","last_change":"2024-03-21T14:10:53.017508+0000","last_active":"2024-03-21T14:10:58.510199+0000","last_peered":"2024-03-21T14:10:58.510199+0000","last_clean":"2024-03-21T14:10:58.510199+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:10:58.510199+0000","last_undegraded":"2024-03-21T14:10:58.510199+0000","last_fullsized":"2024-03-21T14:10:58.510199+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.896413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.896413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:16.646828+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":298,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522417+0000","last_change":"2024-03-21T14:10:57.100243+0000","last_active":"2024-03-21T14:10:58.522417+0000","last_peered":"2024-03-21T14:10:58.522417+0000","last_clean":"2024-03-21T14:10:58.522417+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:10:58.522417+0000","last_undegraded":"2024-03-21T14:10:58.522417+0000","last_fullsized":"2024-03-21T14:10:58.522417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:57.100129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:57.100129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:04.194645+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"72'12","reported_seq":318,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511210+0000","last_change":"2024-03-21T14:09:20.135457+0000","last_active":"2024-03-21T14:10:58.511210+0000","last_peered":"2024-03-21T14:10:58.511210+0000","last_clean":"2024-03-21T14:10:58.511210+0000","last_became_active":"2024-03-21T14:07:23.352151+0000","last_became_peered":"2024-03-21T14:07:23.352151+0000","last_unstale":"2024-03-21T14:10:58.511210+0000","last_undegraded":"2024-03-21T14:10:58.511210+0000","last_fullsized":"2024-03-21T14:10:58.511210+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:09:20.135288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:20.135288+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":298,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.524828+0000","last_change":"2024-03-21T14:10:56.560624+0000","last_active":"2024-03-21T14:10:58.524828+0000","last_peered":"2024-03-21T14:10:58.524828+0000","last_clean":"2024-03-21T14:10:58.524828+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:10:58.524828+0000","last_undegraded":"2024-03-21T14:10:58.524828+0000","last_fullsized":"2024-03-21T14:10:58.524828+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:56.560513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:10:56.560513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:06.873055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"187'9370","reported_seq":9518,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.917140+0000","last_change":"2024-03-21T14:10:45.468374+0000","last_active":"2024-03-21T14:11:01.917140+0000","last_peered":"2024-03-21T14:11:01.917140+0000","last_clean":"2024-03-21T14:11:01.917140+0000","last_became_active":"2024-03-21T14:09:01.656858+0000","last_became_peered":"2024-03-21T14:09:01.656858+0000","last_unstale":"2024-03-21T14:11:01.917140+0000","last_undegraded":"2024-03-21T14:11:01.917140+0000","last_fullsized":"2024-03-21T14:11:01.917140+0000","mapping_epoch":162,"log_start":"167'1200","ondisk_log_start":"167'1200","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'8260","last_scrub_stamp":"2024-03-21T14:10:45.467897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:45.467897+0000","objects_scrubbed":8040,"log_size":8170,"log_dups_size":0,"ondisk_log_size":8170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:06.753611+0000","scrub_duration":3499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447741952,"num_objects":6832,"num_object_clones":0,"num_object_copies":20496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9370,"num_write_kb":518464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.520937+0000","last_change":"2024-03-21T14:10:11.210321+0000","last_active":"2024-03-21T14:10:58.520937+0000","last_peered":"2024-03-21T14:10:58.520937+0000","last_clean":"2024-03-21T14:10:58.520937+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:10:58.520937+0000","last_undegraded":"2024-03-21T14:10:58.520937+0000","last_fullsized":"2024-03-21T14:10:58.520937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.210194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.210194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:23.155153+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.524720+0000","last_change":"2024-03-21T14:10:44.527517+0000","last_active":"2024-03-21T14:10:58.524720+0000","last_peered":"2024-03-21T14:10:58.524720+0000","last_clean":"2024-03-21T14:10:58.524720+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:10:58.524720+0000","last_undegraded":"2024-03-21T14:10:58.524720+0000","last_fullsized":"2024-03-21T14:10:58.524720+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:44.527395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:44.527395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:55.312092+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.524407+0000","last_change":"2024-03-21T14:10:54.060704+0000","last_active":"2024-03-21T14:10:58.524407+0000","last_peered":"2024-03-21T14:10:58.524407+0000","last_clean":"2024-03-21T14:10:58.524407+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:10:58.524407+0000","last_undegraded":"2024-03-21T14:10:58.524407+0000","last_fullsized":"2024-03-21T14:10:58.524407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:54.060588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:54.060588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:05.429955+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"151'28","reported_seq":325,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.513695+0000","last_change":"2024-03-21T14:10:03.820489+0000","last_active":"2024-03-21T14:10:58.513695+0000","last_peered":"2024-03-21T14:10:58.513695+0000","last_clean":"2024-03-21T14:10:58.513695+0000","last_became_active":"2024-03-21T14:02:35.773356+0000","last_became_peered":"2024-03-21T14:02:35.773356+0000","last_unstale":"2024-03-21T14:10:58.513695+0000","last_undegraded":"2024-03-21T14:10:58.513695+0000","last_fullsized":"2024-03-21T14:10:58.513695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'28","last_scrub_stamp":"2024-03-21T14:10:03.820339+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:08:22.013464+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.820339+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:10.338619+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"187'9395","reported_seq":9455,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.788743+0000","last_change":"2024-03-21T14:09:01.452705+0000","last_active":"2024-03-21T14:11:01.788743+0000","last_peered":"2024-03-21T14:11:01.788743+0000","last_clean":"2024-03-21T14:11:01.788743+0000","last_became_active":"2024-03-21T14:09:01.452393+0000","last_became_peered":"2024-03-21T14:09:01.452393+0000","last_unstale":"2024-03-21T14:11:01.788743+0000","last_undegraded":"2024-03-21T14:11:01.788743+0000","last_fullsized":"2024-03-21T14:11:01.788743+0000","mapping_epoch":162,"log_start":"174'5000","ondisk_log_start":"174'5000","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4395,"log_dups_size":0,"ondisk_log_size":4395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447938560,"num_objects":6835,"num_object_clones":0,"num_object_copies":20505,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6835,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9395,"num_write_kb":519360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":135,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521167+0000","last_change":"2024-03-21T14:09:28.472395+0000","last_active":"2024-03-21T14:10:58.521167+0000","last_peered":"2024-03-21T14:10:58.521167+0000","last_clean":"2024-03-21T14:10:58.521167+0000","last_became_active":"2024-03-21T14:07:23.354443+0000","last_became_peered":"2024-03-21T14:07:23.354443+0000","last_unstale":"2024-03-21T14:10:58.521167+0000","last_undegraded":"2024-03-21T14:10:58.521167+0000","last_fullsized":"2024-03-21T14:10:58.521167+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.472282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.472282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":298,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.526627+0000","last_change":"2024-03-21T14:10:50.527648+0000","last_active":"2024-03-21T14:10:58.526627+0000","last_peered":"2024-03-21T14:10:58.526627+0000","last_clean":"2024-03-21T14:10:58.526627+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:10:58.526627+0000","last_undegraded":"2024-03-21T14:10:58.526627+0000","last_fullsized":"2024-03-21T14:10:58.526627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:50.525697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:10:50.525697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:19.914092+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":289,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516455+0000","last_change":"2024-03-21T14:10:12.496525+0000","last_active":"2024-03-21T14:10:58.516455+0000","last_peered":"2024-03-21T14:10:58.516455+0000","last_clean":"2024-03-21T14:10:58.516455+0000","last_became_active":"2024-03-21T14:02:33.442334+0000","last_became_peered":"2024-03-21T14:02:33.442334+0000","last_unstale":"2024-03-21T14:10:58.516455+0000","last_undegraded":"2024-03-21T14:10:58.516455+0000","last_fullsized":"2024-03-21T14:10:58.516455+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:12.496111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.496111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:41.002562+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"72'9","reported_seq":318,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530709+0000","last_change":"2024-03-21T14:09:55.859340+0000","last_active":"2024-03-21T14:10:58.530709+0000","last_peered":"2024-03-21T14:10:58.530709+0000","last_clean":"2024-03-21T14:10:58.530709+0000","last_became_active":"2024-03-21T14:02:58.505855+0000","last_became_peered":"2024-03-21T14:02:58.505855+0000","last_unstale":"2024-03-21T14:10:58.530709+0000","last_undegraded":"2024-03-21T14:10:58.530709+0000","last_fullsized":"2024-03-21T14:10:58.530709+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:09:55.859206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:55.859206+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.961510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"187'9523","reported_seq":9583,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.410252+0000","last_change":"2024-03-21T14:09:01.657675+0000","last_active":"2024-03-21T14:11:01.410252+0000","last_peered":"2024-03-21T14:11:01.410252+0000","last_clean":"2024-03-21T14:11:01.410252+0000","last_became_active":"2024-03-21T14:09:01.657373+0000","last_became_peered":"2024-03-21T14:09:01.657373+0000","last_unstale":"2024-03-21T14:11:01.410252+0000","last_undegraded":"2024-03-21T14:11:01.410252+0000","last_fullsized":"2024-03-21T14:11:01.410252+0000","mapping_epoch":162,"log_start":"171'4500","ondisk_log_start":"171'4500","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5023,"log_dups_size":0,"ondisk_log_size":5023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464453632,"num_objects":7087,"num_object_clones":0,"num_object_copies":21261,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7087,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9523,"num_write_kb":531520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514193+0000","last_change":"2024-03-21T14:10:04.683158+0000","last_active":"2024-03-21T14:10:58.514193+0000","last_peered":"2024-03-21T14:10:58.514193+0000","last_clean":"2024-03-21T14:10:58.514193+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:10:58.514193+0000","last_undegraded":"2024-03-21T14:10:58.514193+0000","last_fullsized":"2024-03-21T14:10:58.514193+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:04.683006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:04.683006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:15.743596+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":301,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515964+0000","last_change":"2024-03-21T14:09:17.993826+0000","last_active":"2024-03-21T14:10:58.515964+0000","last_peered":"2024-03-21T14:10:58.515964+0000","last_clean":"2024-03-21T14:10:58.515964+0000","last_became_active":"2024-03-21T14:02:32.437876+0000","last_became_peered":"2024-03-21T14:02:32.437876+0000","last_unstale":"2024-03-21T14:10:58.515964+0000","last_undegraded":"2024-03-21T14:10:58.515964+0000","last_fullsized":"2024-03-21T14:10:58.515964+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:09:17.993708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:17.993708+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511534+0000","last_change":"2024-03-21T14:10:01.249279+0000","last_active":"2024-03-21T14:10:58.511534+0000","last_peered":"2024-03-21T14:10:58.511534+0000","last_clean":"2024-03-21T14:10:58.511534+0000","last_became_active":"2024-03-21T14:02:33.440949+0000","last_became_peered":"2024-03-21T14:02:33.440949+0000","last_unstale":"2024-03-21T14:10:58.511534+0000","last_undegraded":"2024-03-21T14:10:58.511534+0000","last_fullsized":"2024-03-21T14:10:58.511534+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:01.249165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.249165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:27.783002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"128'39","reported_seq":399,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516196+0000","last_change":"2024-03-21T14:10:03.405443+0000","last_active":"2024-03-21T14:10:58.516196+0000","last_peered":"2024-03-21T14:10:58.516196+0000","last_clean":"2024-03-21T14:10:58.516196+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:10:58.516196+0000","last_undegraded":"2024-03-21T14:10:58.516196+0000","last_fullsized":"2024-03-21T14:10:58.516196+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'39","last_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.405289+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:11.513185+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":143,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.534038+0000","last_change":"2024-03-21T14:10:07.847117+0000","last_active":"2024-03-21T14:10:58.534038+0000","last_peered":"2024-03-21T14:10:58.534038+0000","last_clean":"2024-03-21T14:10:58.534038+0000","last_became_active":"2024-03-21T14:07:23.357608+0000","last_became_peered":"2024-03-21T14:07:23.357608+0000","last_unstale":"2024-03-21T14:10:58.534038+0000","last_undegraded":"2024-03-21T14:10:58.534038+0000","last_fullsized":"2024-03-21T14:10:58.534038+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:07.847006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:07.847006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:09.406528+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.b","version":"187'9315","reported_seq":9375,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.278912+0000","last_change":"2024-03-21T14:09:01.656314+0000","last_active":"2024-03-21T14:11:01.278912+0000","last_peered":"2024-03-21T14:11:01.278912+0000","last_clean":"2024-03-21T14:11:01.278912+0000","last_became_active":"2024-03-21T14:09:01.655845+0000","last_became_peered":"2024-03-21T14:09:01.655845+0000","last_unstale":"2024-03-21T14:11:01.278912+0000","last_undegraded":"2024-03-21T14:11:01.278912+0000","last_fullsized":"2024-03-21T14:11:01.278912+0000","mapping_epoch":162,"log_start":"172'4600","ondisk_log_start":"172'4600","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4715,"log_dups_size":0,"ondisk_log_size":4715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454230016,"num_objects":6931,"num_object_clones":0,"num_object_copies":20793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9315,"num_write_kb":519872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":279,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.524537+0000","last_change":"2024-03-21T14:10:49.206105+0000","last_active":"2024-03-21T14:10:58.524537+0000","last_peered":"2024-03-21T14:10:58.524537+0000","last_clean":"2024-03-21T14:10:58.524537+0000","last_became_active":"2024-03-21T14:02:39.515758+0000","last_became_peered":"2024-03-21T14:02:39.515758+0000","last_unstale":"2024-03-21T14:10:58.524537+0000","last_undegraded":"2024-03-21T14:10:58.524537+0000","last_fullsized":"2024-03-21T14:10:58.524537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:49.205969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.205969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:50.394238+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530053+0000","last_change":"2024-03-21T14:09:28.482765+0000","last_active":"2024-03-21T14:10:58.530053+0000","last_peered":"2024-03-21T14:10:58.530053+0000","last_clean":"2024-03-21T14:10:58.530053+0000","last_became_active":"2024-03-21T14:02:33.442141+0000","last_became_peered":"2024-03-21T14:02:33.442141+0000","last_unstale":"2024-03-21T14:10:58.530053+0000","last_undegraded":"2024-03-21T14:10:58.530053+0000","last_fullsized":"2024-03-21T14:10:58.530053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:28.480168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:09:28.480168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":308,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.525241+0000","last_change":"2024-03-21T14:09:58.644132+0000","last_active":"2024-03-21T14:10:58.525241+0000","last_peered":"2024-03-21T14:10:58.525241+0000","last_clean":"2024-03-21T14:10:58.525241+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:10:58.525241+0000","last_undegraded":"2024-03-21T14:10:58.525241+0000","last_fullsized":"2024-03-21T14:10:58.525241+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:58.643445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.643445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:06.955399+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":343,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.534329+0000","last_change":"2024-03-21T14:10:42.962712+0000","last_active":"2024-03-21T14:10:58.534329+0000","last_peered":"2024-03-21T14:10:58.534329+0000","last_clean":"2024-03-21T14:10:58.534329+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:10:58.534329+0000","last_undegraded":"2024-03-21T14:10:58.534329+0000","last_fullsized":"2024-03-21T14:10:58.534329+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-21T14:10:42.962351+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:10:42.962351+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:51.210815+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.525898+0000","last_change":"2024-03-21T14:10:08.638321+0000","last_active":"2024-03-21T14:10:58.525898+0000","last_peered":"2024-03-21T14:10:58.525898+0000","last_clean":"2024-03-21T14:10:58.525898+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:10:58.525898+0000","last_undegraded":"2024-03-21T14:10:58.525898+0000","last_fullsized":"2024-03-21T14:10:58.525898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:08.638191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:08.638191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:33.293386+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"187'9368","reported_seq":9512,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.357187+0000","last_change":"2024-03-21T14:10:54.986925+0000","last_active":"2024-03-21T14:11:01.357187+0000","last_peered":"2024-03-21T14:11:01.357187+0000","last_clean":"2024-03-21T14:11:01.357187+0000","last_became_active":"2024-03-21T14:09:01.451272+0000","last_became_peered":"2024-03-21T14:09:01.451272+0000","last_unstale":"2024-03-21T14:11:01.357187+0000","last_undegraded":"2024-03-21T14:11:01.357187+0000","last_fullsized":"2024-03-21T14:11:01.357187+0000","mapping_epoch":162,"log_start":"169'3800","ondisk_log_start":"169'3800","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"186'8908","last_scrub_stamp":"2024-03-21T14:10:54.986820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:54.986820+0000","objects_scrubbed":7390,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:02.823571+0000","scrub_duration":1965,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450625536,"num_objects":6876,"num_object_clones":0,"num_object_copies":20628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9368,"num_write_kb":519808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516078+0000","last_change":"2024-03-21T14:09:04.091898+0000","last_active":"2024-03-21T14:10:58.516078+0000","last_peered":"2024-03-21T14:10:58.516078+0000","last_clean":"2024-03-21T14:10:58.516078+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:10:58.516078+0000","last_undegraded":"2024-03-21T14:10:58.516078+0000","last_fullsized":"2024-03-21T14:10:58.516078+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:04.091797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.091797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":562,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521657+0000","last_change":"2024-03-21T14:09:26.513570+0000","last_active":"2024-03-21T14:10:58.521657+0000","last_peered":"2024-03-21T14:10:58.521657+0000","last_clean":"2024-03-21T14:10:58.521657+0000","last_became_active":"2024-03-21T14:07:25.440544+0000","last_became_peered":"2024-03-21T14:07:25.440544+0000","last_unstale":"2024-03-21T14:10:58.521657+0000","last_undegraded":"2024-03-21T14:10:58.521657+0000","last_fullsized":"2024-03-21T14:10:58.521657+0000","mapping_epoch":132,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":133,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:09:26.512762+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":291,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.509977+0000","last_change":"2024-03-21T14:10:11.760727+0000","last_active":"2024-03-21T14:10:58.509977+0000","last_peered":"2024-03-21T14:10:58.509977+0000","last_clean":"2024-03-21T14:10:58.509977+0000","last_became_active":"2024-03-21T14:02:33.443593+0000","last_became_peered":"2024-03-21T14:02:33.443593+0000","last_unstale":"2024-03-21T14:10:58.509977+0000","last_undegraded":"2024-03-21T14:10:58.509977+0000","last_fullsized":"2024-03-21T14:10:58.509977+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.760585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.760585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:31.868157+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"72'16","reported_seq":418,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.591556+0000","last_change":"2024-03-21T14:10:55.042214+0000","last_active":"2024-03-21T14:10:58.591556+0000","last_peered":"2024-03-21T14:10:58.591556+0000","last_clean":"2024-03-21T14:10:58.591556+0000","last_became_active":"2024-03-21T14:02:35.773685+0000","last_became_peered":"2024-03-21T14:02:35.773685+0000","last_unstale":"2024-03-21T14:10:58.591556+0000","last_undegraded":"2024-03-21T14:10:58.591556+0000","last_fullsized":"2024-03-21T14:10:58.591556+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:10:55.042070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.042070+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:03.601196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":283,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522018+0000","last_change":"2024-03-21T14:10:10.232175+0000","last_active":"2024-03-21T14:10:58.522018+0000","last_peered":"2024-03-21T14:10:58.522018+0000","last_clean":"2024-03-21T14:10:58.522018+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:10:58.522018+0000","last_undegraded":"2024-03-21T14:10:58.522018+0000","last_fullsized":"2024-03-21T14:10:58.522018+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:10.231809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:10.231809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:25.178394+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"187'9282","reported_seq":9342,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.788297+0000","last_change":"2024-03-21T14:09:01.656348+0000","last_active":"2024-03-21T14:11:01.788297+0000","last_peered":"2024-03-21T14:11:01.788297+0000","last_clean":"2024-03-21T14:11:01.788297+0000","last_became_active":"2024-03-21T14:09:01.656033+0000","last_became_peered":"2024-03-21T14:09:01.656033+0000","last_unstale":"2024-03-21T14:11:01.788297+0000","last_undegraded":"2024-03-21T14:11:01.788297+0000","last_fullsized":"2024-03-21T14:11:01.788297+0000","mapping_epoch":162,"log_start":"174'4900","ondisk_log_start":"174'4900","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4382,"log_dups_size":0,"ondisk_log_size":4382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447479808,"num_objects":6828,"num_object_clones":0,"num_object_copies":20484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9282,"num_write_kb":515520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":277,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.529638+0000","last_change":"2024-03-21T14:10:00.794766+0000","last_active":"2024-03-21T14:10:58.529638+0000","last_peered":"2024-03-21T14:10:58.529638+0000","last_clean":"2024-03-21T14:10:58.529638+0000","last_became_active":"2024-03-21T14:02:39.471436+0000","last_became_peered":"2024-03-21T14:02:39.471436+0000","last_unstale":"2024-03-21T14:10:58.529638+0000","last_undegraded":"2024-03-21T14:10:58.529638+0000","last_fullsized":"2024-03-21T14:10:58.529638+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:00.794637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:00.794637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.227279+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.540643+0000","last_change":"2024-03-21T14:09:59.812788+0000","last_active":"2024-03-21T14:10:58.540643+0000","last_peered":"2024-03-21T14:10:58.540643+0000","last_clean":"2024-03-21T14:10:58.540643+0000","last_became_active":"2024-03-21T14:07:23.362137+0000","last_became_peered":"2024-03-21T14:07:23.362137+0000","last_unstale":"2024-03-21T14:10:58.540643+0000","last_undegraded":"2024-03-21T14:10:58.540643+0000","last_fullsized":"2024-03-21T14:10:58.540643+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:59.812533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:59.812533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:07.401150+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":804,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:00.149240+0000","last_change":"2024-03-21T14:10:06.216315+0000","last_active":"2024-03-21T14:11:00.149240+0000","last_peered":"2024-03-21T14:11:00.149240+0000","last_clean":"2024-03-21T14:11:00.149240+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:11:00.149240+0000","last_undegraded":"2024-03-21T14:11:00.149240+0000","last_fullsized":"2024-03-21T14:11:00.149240+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:10:06.216180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:10:06.216180+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:32.696552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":501,"num_read_kb":496,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"72'12","reported_seq":193,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.533538+0000","last_change":"2024-03-21T14:10:12.821439+0000","last_active":"2024-03-21T14:10:58.533538+0000","last_peered":"2024-03-21T14:10:58.533538+0000","last_clean":"2024-03-21T14:10:58.533538+0000","last_became_active":"2024-03-21T14:07:23.363490+0000","last_became_peered":"2024-03-21T14:07:23.363490+0000","last_unstale":"2024-03-21T14:10:58.533538+0000","last_undegraded":"2024-03-21T14:10:58.533538+0000","last_fullsized":"2024-03-21T14:10:58.533538+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'12","last_scrub_stamp":"2024-03-21T14:10:12.821292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:12.821292+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:39.357062+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516513+0000","last_change":"2024-03-21T14:09:12.000023+0000","last_active":"2024-03-21T14:10:58.516513+0000","last_peered":"2024-03-21T14:10:58.516513+0000","last_clean":"2024-03-21T14:10:58.516513+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:10:58.516513+0000","last_undegraded":"2024-03-21T14:10:58.516513+0000","last_fullsized":"2024-03-21T14:10:58.516513+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:11.999855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:11.999855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"187'9362","reported_seq":9507,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.833625+0000","last_change":"2024-03-21T14:10:50.488062+0000","last_active":"2024-03-21T14:11:01.833625+0000","last_peered":"2024-03-21T14:11:01.833625+0000","last_clean":"2024-03-21T14:11:01.833625+0000","last_became_active":"2024-03-21T14:09:01.656838+0000","last_became_peered":"2024-03-21T14:09:01.656838+0000","last_unstale":"2024-03-21T14:11:01.833625+0000","last_undegraded":"2024-03-21T14:11:01.833625+0000","last_fullsized":"2024-03-21T14:11:01.833625+0000","mapping_epoch":162,"log_start":"168'2893","ondisk_log_start":"168'2893","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'8591","last_scrub_stamp":"2024-03-21T14:10:50.487498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:50.487498+0000","objects_scrubbed":7720,"log_size":6469,"log_dups_size":0,"ondisk_log_size":6469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:06.026596+0000","scrub_duration":3568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448397312,"num_objects":6842,"num_object_clones":0,"num_object_copies":20526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9362,"num_write_kb":518528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":278,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.522572+0000","last_change":"2024-03-21T14:09:25.416918+0000","last_active":"2024-03-21T14:10:58.522572+0000","last_peered":"2024-03-21T14:10:58.522572+0000","last_clean":"2024-03-21T14:10:58.522572+0000","last_became_active":"2024-03-21T14:02:39.516586+0000","last_became_peered":"2024-03-21T14:02:39.516586+0000","last_unstale":"2024-03-21T14:10:58.522572+0000","last_undegraded":"2024-03-21T14:10:58.522572+0000","last_fullsized":"2024-03-21T14:10:58.522572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:25.416532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:25.416532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530405+0000","last_change":"2024-03-21T14:09:23.381952+0000","last_active":"2024-03-21T14:10:58.530405+0000","last_peered":"2024-03-21T14:10:58.530405+0000","last_clean":"2024-03-21T14:10:58.530405+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:10:58.530405+0000","last_undegraded":"2024-03-21T14:10:58.530405+0000","last_fullsized":"2024-03-21T14:10:58.530405+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:09:23.381813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"72'19","reported_seq":331,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530611+0000","last_change":"2024-03-21T14:10:01.815306+0000","last_active":"2024-03-21T14:10:58.530611+0000","last_peered":"2024-03-21T14:10:58.530611+0000","last_clean":"2024-03-21T14:10:58.530611+0000","last_became_active":"2024-03-21T14:02:35.773349+0000","last_became_peered":"2024-03-21T14:02:35.773349+0000","last_unstale":"2024-03-21T14:10:58.530611+0000","last_undegraded":"2024-03-21T14:10:58.530611+0000","last_fullsized":"2024-03-21T14:10:58.530611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'19","last_scrub_stamp":"2024-03-21T14:10:01.815146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.815146+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:13.735644+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514361+0000","last_change":"2024-03-21T14:10:13.768757+0000","last_active":"2024-03-21T14:10:58.514361+0000","last_peered":"2024-03-21T14:10:58.514361+0000","last_clean":"2024-03-21T14:10:58.514361+0000","last_became_active":"2024-03-21T14:02:37.454286+0000","last_became_peered":"2024-03-21T14:02:37.454286+0000","last_unstale":"2024-03-21T14:10:58.514361+0000","last_undegraded":"2024-03-21T14:10:58.514361+0000","last_fullsized":"2024-03-21T14:10:58.514361+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:13.768639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:10:13.768639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:21.476557+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"187'9667","reported_seq":9817,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.869637+0000","last_change":"2024-03-21T14:10:49.938555+0000","last_active":"2024-03-21T14:11:01.869637+0000","last_peered":"2024-03-21T14:11:01.869637+0000","last_clean":"2024-03-21T14:11:01.869637+0000","last_became_active":"2024-03-21T14:09:01.657572+0000","last_became_peered":"2024-03-21T14:09:01.657572+0000","last_unstale":"2024-03-21T14:11:01.869637+0000","last_undegraded":"2024-03-21T14:11:01.869637+0000","last_fullsized":"2024-03-21T14:11:01.869637+0000","mapping_epoch":162,"log_start":"167'1500","ondisk_log_start":"167'1500","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'8805","last_scrub_stamp":"2024-03-21T14:10:49.938238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.938238+0000","objects_scrubbed":8007,"log_size":8167,"log_dups_size":0,"ondisk_log_size":8167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:19.794253+0000","scrub_duration":3925,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460783616,"num_objects":7031,"num_object_clones":0,"num_object_copies":21093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9667,"num_write_kb":534336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":285,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.531158+0000","last_change":"2024-03-21T14:09:54.876245+0000","last_active":"2024-03-21T14:10:58.531158+0000","last_peered":"2024-03-21T14:10:58.531158+0000","last_clean":"2024-03-21T14:10:58.531158+0000","last_became_active":"2024-03-21T14:08:17.136362+0000","last_became_peered":"2024-03-21T14:08:17.136362+0000","last_unstale":"2024-03-21T14:10:58.531158+0000","last_undegraded":"2024-03-21T14:10:58.531158+0000","last_fullsized":"2024-03-21T14:10:58.531158+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:54.875993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:54.875993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:08.671071+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":282,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516673+0000","last_change":"2024-03-21T14:09:05.122336+0000","last_active":"2024-03-21T14:10:58.516673+0000","last_peered":"2024-03-21T14:10:58.516673+0000","last_clean":"2024-03-21T14:10:58.516673+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:10:58.516673+0000","last_undegraded":"2024-03-21T14:10:58.516673+0000","last_fullsized":"2024-03-21T14:10:58.516673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:05.122232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:05.122232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"152'15","reported_seq":314,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.528225+0000","last_change":"2024-03-21T14:10:01.771055+0000","last_active":"2024-03-21T14:10:58.528225+0000","last_peered":"2024-03-21T14:10:58.528225+0000","last_clean":"2024-03-21T14:10:58.528225+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:10:58.528225+0000","last_undegraded":"2024-03-21T14:10:58.528225+0000","last_fullsized":"2024-03-21T14:10:58.528225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'15","last_scrub_stamp":"2024-03-21T14:10:01.770915+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:10:01.770915+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:17.570180+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"72'8","reported_seq":393,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.531464+0000","last_change":"2024-03-21T14:09:56.813911+0000","last_active":"2024-03-21T14:10:58.531464+0000","last_peered":"2024-03-21T14:10:58.531464+0000","last_clean":"2024-03-21T14:10:58.531464+0000","last_became_active":"2024-03-21T14:02:37.459602+0000","last_became_peered":"2024-03-21T14:02:37.459602+0000","last_unstale":"2024-03-21T14:10:58.531464+0000","last_undegraded":"2024-03-21T14:10:58.531464+0000","last_fullsized":"2024-03-21T14:10:58.531464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:09:56.813768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:56.813768+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.384070+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"187'9434","reported_seq":9495,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.403086+0000","last_change":"2024-03-21T14:09:01.658726+0000","last_active":"2024-03-21T14:11:01.403086+0000","last_peered":"2024-03-21T14:11:01.403086+0000","last_clean":"2024-03-21T14:11:01.403086+0000","last_became_active":"2024-03-21T14:09:01.656735+0000","last_became_peered":"2024-03-21T14:09:01.656735+0000","last_unstale":"2024-03-21T14:11:01.403086+0000","last_undegraded":"2024-03-21T14:11:01.403086+0000","last_fullsized":"2024-03-21T14:11:01.403086+0000","mapping_epoch":162,"log_start":"171'4400","ondisk_log_start":"171'4400","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":5034,"log_dups_size":0,"ondisk_log_size":5034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":457506840,"num_objects":6982,"num_object_clones":0,"num_object_copies":20946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6982,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9434,"num_write_kb":525249,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":271,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521002+0000","last_change":"2024-03-21T14:09:03.463520+0000","last_active":"2024-03-21T14:10:58.521002+0000","last_peered":"2024-03-21T14:10:58.521002+0000","last_clean":"2024-03-21T14:10:58.521002+0000","last_became_active":"2024-03-21T14:02:39.516947+0000","last_became_peered":"2024-03-21T14:02:39.516947+0000","last_unstale":"2024-03-21T14:10:58.521002+0000","last_undegraded":"2024-03-21T14:10:58.521002+0000","last_fullsized":"2024-03-21T14:10:58.521002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:03.463400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.463400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"65'1","reported_seq":146,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530516+0000","last_change":"2024-03-21T14:09:24.419269+0000","last_active":"2024-03-21T14:10:58.530516+0000","last_peered":"2024-03-21T14:10:58.530516+0000","last_clean":"2024-03-21T14:10:58.530516+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:10:58.530516+0000","last_undegraded":"2024-03-21T14:10:58.530516+0000","last_fullsized":"2024-03-21T14:10:58.530516+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:09:24.418831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:24.418831+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"72'14","reported_seq":299,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.511015+0000","last_change":"2024-03-21T14:08:32.898925+0000","last_active":"2024-03-21T14:10:58.511015+0000","last_peered":"2024-03-21T14:10:58.511015+0000","last_clean":"2024-03-21T14:10:58.511015+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:10:58.511015+0000","last_undegraded":"2024-03-21T14:10:58.511015+0000","last_fullsized":"2024-03-21T14:10:58.511015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:08:32.898753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:08:32.898753+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":393,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514302+0000","last_change":"2024-03-21T14:10:48.841874+0000","last_active":"2024-03-21T14:10:58.514302+0000","last_peered":"2024-03-21T14:10:58.514302+0000","last_clean":"2024-03-21T14:10:58.514302+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:10:58.514302+0000","last_undegraded":"2024-03-21T14:10:58.514302+0000","last_fullsized":"2024-03-21T14:10:58.514302+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:10:48.841753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:48.841753+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:09.889695+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"187'9538","reported_seq":9688,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.829865+0000","last_change":"2024-03-21T14:10:47.713790+0000","last_active":"2024-03-21T14:11:01.829865+0000","last_peered":"2024-03-21T14:11:01.829865+0000","last_clean":"2024-03-21T14:11:01.829865+0000","last_became_active":"2024-03-21T14:09:01.657075+0000","last_became_peered":"2024-03-21T14:09:01.657075+0000","last_unstale":"2024-03-21T14:11:01.829865+0000","last_undegraded":"2024-03-21T14:11:01.829865+0000","last_fullsized":"2024-03-21T14:11:01.829865+0000","mapping_epoch":162,"log_start":"168'3127","ondisk_log_start":"168'3127","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'8531","last_scrub_stamp":"2024-03-21T14:10:47.713449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:47.713449+0000","objects_scrubbed":8038,"log_size":6411,"log_dups_size":0,"ondisk_log_size":6411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:53.134243+0000","scrub_duration":2897,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455344128,"num_objects":6948,"num_object_clones":0,"num_object_copies":20844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9538,"num_write_kb":527552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":276,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510537+0000","last_change":"2024-03-21T14:08:31.865055+0000","last_active":"2024-03-21T14:10:58.510537+0000","last_peered":"2024-03-21T14:10:58.510537+0000","last_clean":"2024-03-21T14:10:58.510537+0000","last_became_active":"2024-03-21T14:07:23.364265+0000","last_became_peered":"2024-03-21T14:07:23.364265+0000","last_unstale":"2024-03-21T14:10:58.510537+0000","last_undegraded":"2024-03-21T14:10:58.510537+0000","last_fullsized":"2024-03-21T14:10:58.510537+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:31.864937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:08:31.864937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":290,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514119+0000","last_change":"2024-03-21T14:10:10.780682+0000","last_active":"2024-03-21T14:10:58.514119+0000","last_peered":"2024-03-21T14:10:58.514119+0000","last_clean":"2024-03-21T14:10:58.514119+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:10:58.514119+0000","last_undegraded":"2024-03-21T14:10:58.514119+0000","last_fullsized":"2024-03-21T14:10:58.514119+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:10:10.780543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:25.376223+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"72'18","reported_seq":244,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.521845+0000","last_change":"2024-03-21T14:10:56.096438+0000","last_active":"2024-03-21T14:10:58.521845+0000","last_peered":"2024-03-21T14:10:58.521845+0000","last_clean":"2024-03-21T14:10:58.521845+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:10:58.521845+0000","last_undegraded":"2024-03-21T14:10:58.521845+0000","last_fullsized":"2024-03-21T14:10:58.521845+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'18","last_scrub_stamp":"2024-03-21T14:10:56.096183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:56.096183+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:11.124685+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":266,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510820+0000","last_change":"2024-03-21T14:08:16.845392+0000","last_active":"2024-03-21T14:10:58.510820+0000","last_peered":"2024-03-21T14:10:58.510820+0000","last_clean":"2024-03-21T14:10:58.510820+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:10:58.510820+0000","last_undegraded":"2024-03-21T14:10:58.510820+0000","last_fullsized":"2024-03-21T14:10:58.510820+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:08:16.845286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:08:16.845286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"187'9436","reported_seq":9496,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.720869+0000","last_change":"2024-03-21T14:09:01.452701+0000","last_active":"2024-03-21T14:11:01.720869+0000","last_peered":"2024-03-21T14:11:01.720869+0000","last_clean":"2024-03-21T14:11:01.720869+0000","last_became_active":"2024-03-21T14:09:01.452387+0000","last_became_peered":"2024-03-21T14:09:01.452387+0000","last_unstale":"2024-03-21T14:11:01.720869+0000","last_undegraded":"2024-03-21T14:11:01.720869+0000","last_fullsized":"2024-03-21T14:11:01.720869+0000","mapping_epoch":162,"log_start":"174'5000","ondisk_log_start":"174'5000","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4436,"log_dups_size":0,"ondisk_log_size":4436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459276288,"num_objects":7008,"num_object_clones":0,"num_object_copies":21024,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7008,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9436,"num_write_kb":526208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":289,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516353+0000","last_change":"2024-03-21T14:09:57.384748+0000","last_active":"2024-03-21T14:10:58.516353+0000","last_peered":"2024-03-21T14:10:58.516353+0000","last_clean":"2024-03-21T14:10:58.516353+0000","last_became_active":"2024-03-21T14:08:17.203307+0000","last_became_peered":"2024-03-21T14:08:17.203307+0000","last_unstale":"2024-03-21T14:10:58.516353+0000","last_undegraded":"2024-03-21T14:10:58.516353+0000","last_fullsized":"2024-03-21T14:10:58.516353+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:57.384552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:09:57.384552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:23.621828+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":391,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515794+0000","last_change":"2024-03-21T14:10:55.804376+0000","last_active":"2024-03-21T14:10:58.515794+0000","last_peered":"2024-03-21T14:10:58.515794+0000","last_clean":"2024-03-21T14:10:58.515794+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:10:58.515794+0000","last_undegraded":"2024-03-21T14:10:58.515794+0000","last_fullsized":"2024-03-21T14:10:58.515794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:10:55.804112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.804112+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:04.730841+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":135,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.532002+0000","last_change":"2024-03-21T14:09:13.247866+0000","last_active":"2024-03-21T14:10:58.532002+0000","last_peered":"2024-03-21T14:10:58.532002+0000","last_clean":"2024-03-21T14:10:58.532002+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:10:58.532002+0000","last_undegraded":"2024-03-21T14:10:58.532002+0000","last_fullsized":"2024-03-21T14:10:58.532002+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:13.247726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:09:13.247726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"72'15","reported_seq":310,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.534424+0000","last_change":"2024-03-21T14:09:58.762917+0000","last_active":"2024-03-21T14:10:58.534424+0000","last_peered":"2024-03-21T14:10:58.534424+0000","last_clean":"2024-03-21T14:10:58.534424+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:10:58.534424+0000","last_undegraded":"2024-03-21T14:10:58.534424+0000","last_fullsized":"2024-03-21T14:10:58.534424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'15","last_scrub_stamp":"2024-03-21T14:09:58.762639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:58.762639+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:01.152458+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"187'9528","reported_seq":9588,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.355581+0000","last_change":"2024-03-21T14:09:01.451050+0000","last_active":"2024-03-21T14:11:01.355581+0000","last_peered":"2024-03-21T14:11:01.355581+0000","last_clean":"2024-03-21T14:11:01.355581+0000","last_became_active":"2024-03-21T14:09:01.450729+0000","last_became_peered":"2024-03-21T14:09:01.450729+0000","last_unstale":"2024-03-21T14:11:01.355581+0000","last_undegraded":"2024-03-21T14:11:01.355581+0000","last_fullsized":"2024-03-21T14:11:01.355581+0000","mapping_epoch":162,"log_start":"172'4800","ondisk_log_start":"172'4800","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463339520,"num_objects":7070,"num_object_clones":0,"num_object_copies":21210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9528,"num_write_kb":531136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":270,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.509665+0000","last_change":"2024-03-21T14:10:03.673038+0000","last_active":"2024-03-21T14:10:58.509665+0000","last_peered":"2024-03-21T14:10:58.509665+0000","last_clean":"2024-03-21T14:10:58.509665+0000","last_became_active":"2024-03-21T14:02:39.515823+0000","last_became_peered":"2024-03-21T14:02:39.515823+0000","last_unstale":"2024-03-21T14:10:58.509665+0000","last_undegraded":"2024-03-21T14:10:58.509665+0000","last_fullsized":"2024-03-21T14:10:58.509665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:03.672903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.672903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:29.596347+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":273,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.516425+0000","last_change":"2024-03-21T14:09:16.047979+0000","last_active":"2024-03-21T14:10:58.516425+0000","last_peered":"2024-03-21T14:10:58.516425+0000","last_clean":"2024-03-21T14:10:58.516425+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:10:58.516425+0000","last_undegraded":"2024-03-21T14:10:58.516425+0000","last_fullsized":"2024-03-21T14:10:58.516425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.047852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.047852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":289,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510866+0000","last_change":"2024-03-21T14:10:02.232967+0000","last_active":"2024-03-21T14:10:58.510866+0000","last_peered":"2024-03-21T14:10:58.510866+0000","last_clean":"2024-03-21T14:10:58.510866+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:10:58.510866+0000","last_undegraded":"2024-03-21T14:10:58.510866+0000","last_fullsized":"2024-03-21T14:10:58.510866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:02.232818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:02.232818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:05.637769+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"72'11","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515566+0000","last_change":"2024-03-21T14:09:03.147020+0000","last_active":"2024-03-21T14:10:58.515566+0000","last_peered":"2024-03-21T14:10:58.515566+0000","last_clean":"2024-03-21T14:10:58.515566+0000","last_became_active":"2024-03-21T14:02:35.448362+0000","last_became_peered":"2024-03-21T14:02:35.448362+0000","last_unstale":"2024-03-21T14:10:58.515566+0000","last_undegraded":"2024-03-21T14:10:58.515566+0000","last_fullsized":"2024-03-21T14:10:58.515566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:09:03.146877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:03.146877+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"187'9433","reported_seq":9551,"reported_epoch":187,"state":"active+clean+scrubbing","last_fresh":"2024-03-21T14:11:01.765465+0000","last_change":"2024-03-21T14:10:59.903673+0000","last_active":"2024-03-21T14:11:01.765465+0000","last_peered":"2024-03-21T14:11:01.765465+0000","last_clean":"2024-03-21T14:11:01.765465+0000","last_became_active":"2024-03-21T14:09:01.656962+0000","last_became_peered":"2024-03-21T14:09:01.656962+0000","last_unstale":"2024-03-21T14:11:01.765465+0000","last_undegraded":"2024-03-21T14:11:01.765465+0000","last_fullsized":"2024-03-21T14:11:01.765465+0000","mapping_epoch":162,"log_start":"175'5000","ondisk_log_start":"175'5000","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:09:00.432428+0000","objects_scrubbed":5599,"log_size":4433,"log_dups_size":0,"ondisk_log_size":4433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":452788224,"num_objects":6909,"num_object_clones":0,"num_object_copies":20727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9433,"num_write_kb":522944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":287,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.525473+0000","last_change":"2024-03-21T14:10:03.740909+0000","last_active":"2024-03-21T14:10:58.525473+0000","last_peered":"2024-03-21T14:10:58.525473+0000","last_clean":"2024-03-21T14:10:58.525473+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:10:58.525473+0000","last_undegraded":"2024-03-21T14:10:58.525473+0000","last_fullsized":"2024-03-21T14:10:58.525473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:10:03.740783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:03.740783+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:12.932320+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":281,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.530500+0000","last_change":"2024-03-21T14:09:16.366776+0000","last_active":"2024-03-21T14:10:58.530500+0000","last_peered":"2024-03-21T14:10:58.530500+0000","last_clean":"2024-03-21T14:10:58.530500+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:10:58.530500+0000","last_undegraded":"2024-03-21T14:10:58.530500+0000","last_fullsized":"2024-03-21T14:10:58.530500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:16.366566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:09:16.366566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":289,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.514430+0000","last_change":"2024-03-21T14:10:08.767362+0000","last_active":"2024-03-21T14:10:58.514430+0000","last_peered":"2024-03-21T14:10:58.514430+0000","last_clean":"2024-03-21T14:10:58.514430+0000","last_became_active":"2024-03-21T14:02:33.442184+0000","last_became_peered":"2024-03-21T14:02:33.442184+0000","last_unstale":"2024-03-21T14:10:58.514430+0000","last_undegraded":"2024-03-21T14:10:58.514430+0000","last_fullsized":"2024-03-21T14:10:58.514430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:08.767256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:08.767256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:11.594705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"72'17","reported_seq":305,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.512579+0000","last_change":"2024-03-21T14:09:04.171171+0000","last_active":"2024-03-21T14:10:58.512579+0000","last_peered":"2024-03-21T14:10:58.512579+0000","last_clean":"2024-03-21T14:10:58.512579+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:10:58.512579+0000","last_undegraded":"2024-03-21T14:10:58.512579+0000","last_fullsized":"2024-03-21T14:10:58.512579+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'17","last_scrub_stamp":"2024-03-21T14:09:04.171069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:09:04.171069+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"187'9510","reported_seq":9664,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:11:01.836158+0000","last_change":"2024-03-21T14:10:45.929081+0000","last_active":"2024-03-21T14:11:01.836158+0000","last_peered":"2024-03-21T14:11:01.836158+0000","last_clean":"2024-03-21T14:11:01.836158+0000","last_became_active":"2024-03-21T14:09:01.451020+0000","last_became_peered":"2024-03-21T14:09:01.451020+0000","last_unstale":"2024-03-21T14:11:01.836158+0000","last_undegraded":"2024-03-21T14:11:01.836158+0000","last_fullsized":"2024-03-21T14:11:01.836158+0000","mapping_epoch":162,"log_start":"168'3095","ondisk_log_start":"168'3095","created":162,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"184'8393","last_scrub_stamp":"2024-03-21T14:10:45.928806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:00.432428+0000","last_clean_scrub_stamp":"2024-03-21T14:10:45.928806+0000","objects_scrubbed":8153,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:11.570297+0000","scrub_duration":3172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455081984,"num_objects":6944,"num_object_clones":0,"num_object_copies":20832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9510,"num_write_kb":526528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":269,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.515848+0000","last_change":"2024-03-21T14:09:09.117631+0000","last_active":"2024-03-21T14:10:58.515848+0000","last_peered":"2024-03-21T14:10:58.515848+0000","last_clean":"2024-03-21T14:10:58.515848+0000","last_became_active":"2024-03-21T14:02:39.458693+0000","last_became_peered":"2024-03-21T14:02:39.458693+0000","last_unstale":"2024-03-21T14:10:58.515848+0000","last_undegraded":"2024-03-21T14:10:58.515848+0000","last_fullsized":"2024-03-21T14:10:58.515848+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:09:09.117496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:09:09.117496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":401,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.525053+0000","last_change":"2024-03-21T14:10:45.540690+0000","last_active":"2024-03-21T14:10:58.525053+0000","last_peered":"2024-03-21T14:10:58.525053+0000","last_clean":"2024-03-21T14:10:58.525053+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:10:58.525053+0000","last_undegraded":"2024-03-21T14:10:58.525053+0000","last_fullsized":"2024-03-21T14:10:58.525053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:10:45.540556+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:10:45.540556+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:11:55.895451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":296,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.510720+0000","last_change":"2024-03-21T14:10:50.837805+0000","last_active":"2024-03-21T14:10:58.510720+0000","last_peered":"2024-03-21T14:10:58.510720+0000","last_clean":"2024-03-21T14:10:58.510720+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:10:58.510720+0000","last_undegraded":"2024-03-21T14:10:58.510720+0000","last_fullsized":"2024-03-21T14:10:58.510720+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:50.837728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:50.837728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:12:17.050109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"167'15","reported_seq":297,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-03-21T14:10:58.512053+0000","last_change":"2024-03-21T14:09:22.120079+0000","last_active":"2024-03-21T14:10:58.512053+0000","last_peered":"2024-03-21T14:10:58.512053+0000","last_clean":"2024-03-21T14:10:58.512053+0000","last_became_active":"2024-03-21T14:02:35.450025+0000","last_became_peered":"2024-03-21T14:02:35.450025+0000","last_unstale":"2024-03-21T14:10:58.512053+0000","last_undegraded":"2024-03-21T14:10:58.512053+0000","last_fullsized":"2024-03-21T14:10:58.512053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'15","last_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:09:22.119861+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7267876888,"num_objects":110900,"num_object_clones":0,"num_object_copies":332700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":110900,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":150930,"num_write_kb":8378497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21811965952,"data_stored":21811953736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89015,"ondisk_log_size":89015,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":501,"num_read_kb":496,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":1568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":450,"ondisk_log_size":450,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103274,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2492184,"kb_used_data":2231952,"kb_used_omap":1,"kb_used_meta":260222,"kb_avail":91252968,"statfs":{"total":95995035648,"available":93443039232,"internally_reserved":0,"allocated":2285518848,"data_stored":2285123126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":266467786},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430319,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414008,"kb_used_data":2212076,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":91331144,"statfs":{"total":95995035648,"available":93523091456,"internally_reserved":0,"allocated":2265165824,"data_stored":2264804823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":206764486},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3406308,"kb_used_data":3118236,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":90338844,"statfs":{"total":95995035648,"available":92506976256,"internally_reserved":0,"allocated":3193073664,"data_stored":3192665937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2,2,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2522920,"kb_used_data":2234912,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":91222232,"statfs":{"total":95995035648,"available":93411565568,"internally_reserved":0,"allocated":2288549888,"data_stored":2288200737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411460,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3377460,"kb_used_data":3103340,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":90367692,"statfs":{"total":95995035648,"available":92536516608,"internally_reserved":0,"allocated":3177820160,"data_stored":3177458767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182526,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2538692,"kb_used_data":2236796,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":91206460,"statfs":{"total":95995035648,"available":93395415040,"internally_reserved":0,"allocated":2290479104,"data_stored":2290101855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019002,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2433228,"kb_used_data":2243780,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":91311924,"statfs":{"total":95995035648,"available":93503410176,"internally_reserved":0,"allocated":2297630720,"data_stored":2297245433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691957,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4274128,"kb_used_data":4036100,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":89471024,"statfs":{"total":95995035648,"available":91618328576,"internally_reserved":0,"allocated":4132966400,"data_stored":4132581898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":243726789},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2270756864,"data_stored":2270756864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3163947008,"data_stored":3163947008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3172732928,"data_stored":3172728856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2268725248,"data_stored":2268725248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2283601920,"data_stored":2283601920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4114812928,"data_stored":4114808856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2258763776,"data_stored":2258763776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2278625280,"data_stored":2278621208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:11:06.324 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-21T14:11:06.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_1 pg_num 26 2024-03-21T14:11:06.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:06.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[29576]: pgmap v350: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 744 B/s rd, 0 B/s wr, 824 op/s 2024-03-21T14:11:06.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[29576]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:11:06.411 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:11:06.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:11:06.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[40510]: pgmap v350: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 744 B/s rd, 0 B/s wr, 824 op/s 2024-03-21T14:11:06.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[40510]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:11:06.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:11:06.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:11:06.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:06.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.459 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:06.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:06.591 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:06 smithi118 ceph-mon[33862]: pgmap v350: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 744 B/s rd, 0 B/s wr, 824 op/s 2024-03-21T14:11:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:06 smithi118 ceph-mon[33862]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:11:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:06 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:11:06.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:06 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:11:06.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:06.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:11:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:11:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:11:06.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:06.861 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:06.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:06.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:07.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:07.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:07.138 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:07.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:11:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:11:07.209 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:07.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.271 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:07.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:07.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:07.407 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[29576]: 8.4 scrub ok 2024-03-21T14:11:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:11:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:11:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:11:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[40510]: 8.4 scrub ok 2024-03-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:11:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:07 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:11:07.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:07.541 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:07.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:07.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:07.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:07 smithi118 ceph-mon[33862]: 8.4 scrub ok 2024-03-21T14:11:07.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:07 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:11:07.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:07 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:11:07.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:07.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:07 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:11:07.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:07 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:11:07.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:07.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:07.921 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:07.957 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:07.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:08.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:08.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:08.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:08.244 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:08.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:08.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:08.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:08.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:08.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:08.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:08.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:08.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:08.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: 5.7 deep-scrub starts 2024-03-21T14:11:08.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: 5.7 deep-scrub ok 2024-03-21T14:11:08.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: pgmap v351: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-21T14:11:08.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:08.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:08.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:11:08.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:08 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: 5.7 deep-scrub starts 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: 5.7 deep-scrub ok 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: pgmap v351: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:11:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: 5.7 deep-scrub starts 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: 5.7 deep-scrub ok 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: pgmap v351: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:11:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:08 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:11:08.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:08.815 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:08.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:08.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:08.948 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:08.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:08.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:09.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:09.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:09.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:09.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:09.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:09.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:09.339 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.370 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:09.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:09.404 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:09 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:11:09.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:09 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:11:09.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:09 smithi003 ceph-mon[29576]: osdmap e190: 8 total, 8 up, 8 in 2024-03-21T14:11:09.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:09 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:11:09.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:09 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:11:09.405 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:09 smithi003 ceph-mon[40510]: osdmap e190: 8 total, 8 up, 8 in 2024-03-21T14:11:09.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:09.510 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:09.611 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:11:09.611 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:11:09.612 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:11:09.612 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:11:09.612 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:11:09.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:09.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:09.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:09 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:11:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:09 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:11:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:09 smithi118 ceph-mon[33862]: osdmap e190: 8 total, 8 up, 8 in 2024-03-21T14:11:09.751 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:09.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:09.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:09.918 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:09.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:09.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:10.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.052 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:10.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:10.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:10.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:10.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:10.352 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:10.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:10.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.487 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:10.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:10.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:10.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:10.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:10.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:10 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:11:10.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:10 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:11:10.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:10 smithi003 ceph-mon[29576]: pgmap v353: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 724 B/s rd, 0 B/s wr, 792 op/s 2024-03-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:10 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:10 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:11:10.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:10 smithi003 ceph-mon[40510]: pgmap v353: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 724 B/s rd, 0 B/s wr, 792 op/s 2024-03-21T14:11:10.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:10.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:10.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:10.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:10.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:10.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:10 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:11:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:10 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:11:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:10 smithi118 ceph-mon[33862]: pgmap v353: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 724 B/s rd, 0 B/s wr, 792 op/s 2024-03-21T14:11:11.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:11.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:11.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:11.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:11.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:11.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:11.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:11.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:11.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:11.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:11.527 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:11:11.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:11.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:11.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:11:11.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:11.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:11:11.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:11:11.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:11:11.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 4.a deep-scrub starts 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 4.a deep-scrub ok 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2376857674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 4.a deep-scrub starts 2024-03-21T14:11:11.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 4.a deep-scrub ok 2024-03-21T14:11:11.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:11:11.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:11:11.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:11 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2376857674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:11:11.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:11.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 4.a deep-scrub starts 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 4.a deep-scrub ok 2024-03-21T14:11:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:11:11.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:11:11.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:11 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2376857674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:11:11.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:11.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:11.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:12.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:12.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:12.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:12.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:12.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:12.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:12.346 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:12.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:12.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:12.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:12.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:12.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: pgmap v354: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2376857674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[29576]: osdmap e191: 8 total, 8 up, 8 in 2024-03-21T14:11:12.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: pgmap v354: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2376857674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:11:12.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:12 smithi003 ceph-mon[40510]: osdmap e191: 8 total, 8 up, 8 in 2024-03-21T14:11:12.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:12.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:12.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:12.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:12.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: pgmap v354: 148 pgs: 148 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 789 op/s 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2376857674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:11:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:12 smithi118 ceph-mon[33862]: osdmap e191: 8 total, 8 up, 8 in 2024-03-21T14:11:13.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:13.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:13.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:13.191 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:13.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:13.327 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:13.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:13.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.453 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:13.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:13.554 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:13.585 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:13.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:13.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:13.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:13.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:13.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:13.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:14.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:14.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:14.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:14.206 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:11:14.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:11:14.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:14.296 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:11:14.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:14.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:14.335 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:11:14.368 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:11:14.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:11:14.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:14.451 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:11:14.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:14.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:14.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:11:14.497 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:11:14.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:14.589 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:14.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:14.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:14.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:14.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:11:14.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:14.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:14.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:14 smithi003 ceph-mon[29576]: pgmap v356: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-03-21T14:11:14.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2183191436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:14.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:14 smithi003 ceph-mon[40510]: pgmap v356: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-03-21T14:11:14.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:14 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2183191436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:14 smithi118 ceph-mon[33862]: pgmap v356: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-03-21T14:11:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:14 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2183191436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:14.975 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:15.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.001 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:15.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:15.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:15.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:15.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:15.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:15.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:15.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:15.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:15.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:15.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:15.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:15.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:15.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:15.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:15 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:15.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:15 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:15.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:15 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:15.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:15 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:15.803 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:15.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:15.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:15.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:15 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:15 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:15.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.049 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:16.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:16.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:16.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:16.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:16.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:16.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:16.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:16.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:16.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: pgmap v357: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: osdmap e192: 8 total, 8 up, 8 in 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:16.647 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:11:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: pgmap v357: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: osdmap e192: 8 total, 8 up, 8 in 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:16.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:16.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:16.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:16.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:16.815 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:16.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:16.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:11:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: pgmap v357: 148 pgs: 148 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 742 op/s 2024-03-21T14:11:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: osdmap e192: 8 total, 8 up, 8 in 2024-03-21T14:11:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:11:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2024-03-21T14:11:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]: dispatch 2024-03-21T14:11:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:16.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:16.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:16.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:17.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.097 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:17.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:17.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:17.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:17.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:17.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:17.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:17.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.521 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:17.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:17.554 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 8 pg_num to 26 2024-03-21T14:11:17.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:17.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:17.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:17.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:17.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:11:17.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]': finished 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: osdmap e193: 8 total, 8 up, 8 in 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/171849475' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]': finished 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: osdmap e193: 8 total, 8 up, 8 in 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:17.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:11:17.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:11:17.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:17.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:17 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/171849475' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-21T14:11:17.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0]}]': finished 2024-03-21T14:11:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: osdmap e193: 8 total, 8 up, 8 in 2024-03-21T14:11:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:11:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:11:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:17.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:17 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/171849475' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-21T14:11:17.934 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:17.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:17.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:18.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:18.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.070 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:18.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:18.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:18.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:18.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:18.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:18.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:18.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:18.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:18.634 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:18.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:18.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[29576]: pgmap v360: 148 pgs: 1 peering, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.83k op/s 2024-03-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/171849475' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-21T14:11:18.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[29576]: osdmap e194: 8 total, 8 up, 8 in 2024-03-21T14:11:18.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[40510]: pgmap v360: 148 pgs: 1 peering, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.83k op/s 2024-03-21T14:11:18.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:18.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:18.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/171849475' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-21T14:11:18.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:18 smithi003 ceph-mon[40510]: osdmap e194: 8 total, 8 up, 8 in 2024-03-21T14:11:18.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:18.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:18.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:18.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:18 smithi118 ceph-mon[33862]: pgmap v360: 148 pgs: 1 peering, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.83k op/s 2024-03-21T14:11:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:18 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:18 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:18 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/171849475' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-21T14:11:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:18 smithi118 ceph-mon[33862]: osdmap e194: 8 total, 8 up, 8 in 2024-03-21T14:11:19.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:19.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:19.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:19.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.189 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:19.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:19.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:19.324 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:19.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:19.457 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:19.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:19.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:19.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:19.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:19.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:19.870 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:19 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:19.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:19.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:19.998 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:19.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:19 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:20.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:19 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:20.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:20.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:20.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:20.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:20.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:20.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:20.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:20.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:20.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:20.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:20.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:20.638 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:20.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:20.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:20.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:20.879 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:20.913 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:20.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:20.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:20 smithi118 ceph-mon[33862]: pgmap v362: 148 pgs: 1 peering, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 961 op/s 2024-03-21T14:11:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-21T14:11:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:20 smithi118 ceph-mon[33862]: osdmap e195: 8 total, 8 up, 8 in 2024-03-21T14:11:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:20 smithi003 ceph-mon[29576]: pgmap v362: 148 pgs: 1 peering, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 961 op/s 2024-03-21T14:11:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-21T14:11:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:20 smithi003 ceph-mon[29576]: osdmap e195: 8 total, 8 up, 8 in 2024-03-21T14:11:21.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:20 smithi003 ceph-mon[40510]: pgmap v362: 148 pgs: 1 peering, 147 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 961 op/s 2024-03-21T14:11:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-21T14:11:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:20 smithi003 ceph-mon[40510]: osdmap e195: 8 total, 8 up, 8 in 2024-03-21T14:11:21.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:21.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:21.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:21.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.447 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:21.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:21.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:21.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:21.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:21.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:21.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:21.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:21.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:21 smithi003 ceph-mon[29576]: osdmap e196: 8 total, 8 up, 8 in 2024-03-21T14:11:21.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:21 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:21.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:21 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:21.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:21 smithi003 ceph-mon[40510]: osdmap e196: 8 total, 8 up, 8 in 2024-03-21T14:11:21.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:21 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:21.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:21 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:21 smithi118 ceph-mon[33862]: osdmap e196: 8 total, 8 up, 8 in 2024-03-21T14:11:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:21 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:21 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-21T14:11:21.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:21.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:21.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:22.057 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:22.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:22.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:22.184 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:22.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:22.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:22.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.336 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:22.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:22.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:22.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:22.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:22.590 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:22.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:22.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:22.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:22.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:22.732 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:11:22.732 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:11:22.820 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:22.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:22 smithi118 ceph-mon[33862]: pgmap v365: 158 pgs: 9 unknown, 1 peering, 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 795 op/s 2024-03-21T14:11:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:22 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-21T14:11:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:22 smithi118 ceph-mon[33862]: osdmap e197: 8 total, 8 up, 8 in 2024-03-21T14:11:22.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:22.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:22.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:22.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:22 smithi003 ceph-mon[29576]: pgmap v365: 158 pgs: 9 unknown, 1 peering, 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 795 op/s 2024-03-21T14:11:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:22 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-21T14:11:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:22 smithi003 ceph-mon[29576]: osdmap e197: 8 total, 8 up, 8 in 2024-03-21T14:11:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:22 smithi003 ceph-mon[40510]: pgmap v365: 158 pgs: 9 unknown, 1 peering, 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 795 op/s 2024-03-21T14:11:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:22 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-21T14:11:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:22 smithi003 ceph-mon[40510]: osdmap e197: 8 total, 8 up, 8 in 2024-03-21T14:11:23.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:23.102 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:23.184 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:23.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:23.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:23.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:23.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:23.359 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:23.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:23.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:23.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:23.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:23.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:23.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:23.751 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:23.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:23 smithi003 ceph-mon[29576]: osdmap e198: 8 total, 8 up, 8 in 2024-03-21T14:11:23.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:23 smithi003 ceph-mon[40510]: osdmap e198: 8 total, 8 up, 8 in 2024-03-21T14:11:23.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:23.883 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:23.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:23.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:23 smithi118 ceph-mon[33862]: osdmap e198: 8 total, 8 up, 8 in 2024-03-21T14:11:23.986 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.011 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:24.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:24.112 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:24.145 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:24.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:24.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:24.348 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:24.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:24.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:11:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:24.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:24.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:24.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:24.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:24.729 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:24.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:24.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:24.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:24.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: pgmap v368: 158 pgs: 5 unknown, 153 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:11:24.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10911 2024-03-21T14:11:24.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10800 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 196'10776 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 196'10584 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10678 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10810 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.17 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10713 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: osdmap e199: 8 total, 8 up, 8 in 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10800 2024-03-21T14:11:24.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 196'10810 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10713 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[40510]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: pgmap v368: 158 pgs: 5 unknown, 153 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10911 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10800 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 196'10776 2024-03-21T14:11:24.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 196'10584 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10678 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10810 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.17 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10713 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: osdmap e199: 8 total, 8 up, 8 in 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10800 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 196'10810 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10713 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:24 smithi003 ceph-mon[29576]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: pgmap v368: 158 pgs: 5 unknown, 153 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:11:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10911 2024-03-21T14:11:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10800 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 196'10776 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 196'10584 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 196'10678 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10810 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.17 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10713 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: osdmap e199: 8 total, 8 up, 8 in 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10800 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10899 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 196'10810 2024-03-21T14:11:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10713 2024-03-21T14:11:24.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 196'10724 2024-03-21T14:11:24.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:24 smithi118 ceph-mon[33862]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 196'10632 2024-03-21T14:11:24.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:25.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:25.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:25.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:25.186 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:25.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:25.322 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:25.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:25.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:25.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:25.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:25.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:25.713 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:25.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:25.851 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:25.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:25.885 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:25.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:25.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:26.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:26.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:26.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:26.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:26.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:26.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:26.468 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:26.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:26.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:26.609 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:11:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:11:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:11:26.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:26.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.748 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:26.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:26.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:26.882 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:26.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:26 smithi118 ceph-mon[33862]: pgmap v370: 158 pgs: 5 unknown, 153 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:11:26.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:11:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:11:26.984 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:26.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:26 smithi003 ceph-mon[29576]: pgmap v370: 158 pgs: 5 unknown, 153 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:11:26.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:26 smithi003 ceph-mon[40510]: pgmap v370: 158 pgs: 5 unknown, 153 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-21T14:11:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:27.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:27.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:27.118 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:27.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:27.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:27.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:27.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:27.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:27.464 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:27.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:27.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:27.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:27.739 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:27.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:27.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:27.875 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:27.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:27 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:27 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:27.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:27.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:27 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:27.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:27 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:27.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:27.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:27 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:27.940 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:27 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:27.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:28.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:28.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:28.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:28.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:28.221 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:11:28.221 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:11:28.221 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T14:11:28.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T14:11:28.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:28.294 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:28.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:28.428 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:28.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:28.585 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:28.654 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:11:28.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:28.719 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[29576]: pgmap v371: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 48252/257964 objects misplaced (18.705%); 14 MiB/s, 224 objects/s recovering 2024-03-21T14:11:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[29576]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:11:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[29576]: osdmap e200: 8 total, 8 up, 8 in 2024-03-21T14:11:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1271422725' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:11:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:11:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[40510]: pgmap v371: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 48252/257964 objects misplaced (18.705%); 14 MiB/s, 224 objects/s recovering 2024-03-21T14:11:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[40510]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:11:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[40510]: osdmap e200: 8 total, 8 up, 8 in 2024-03-21T14:11:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1271422725' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:11:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:11:28.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:28.856 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:28.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:28 smithi118 ceph-mon[33862]: pgmap v371: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 48252/257964 objects misplaced (18.705%); 14 MiB/s, 224 objects/s recovering 2024-03-21T14:11:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:28 smithi118 ceph-mon[33862]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:11:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:28 smithi118 ceph-mon[33862]: osdmap e200: 8 total, 8 up, 8 in 2024-03-21T14:11:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1271422725' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:11:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:11:28.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:29.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:29.045 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:11:29.088 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:11:29.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:29.139 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:11:29.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:29.184 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:11:29.229 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:11:29.243 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:29.271 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:11:29.273 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:29.314 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:11:29.358 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:11:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:11:29.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:29.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:29.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:29.553 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:29.655 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:29.688 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:29.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:29.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:29.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:29.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:29 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:29 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:11:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:29 smithi118 ceph-mon[33862]: osdmap e201: 8 total, 8 up, 8 in 2024-03-21T14:11:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3377436818' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:29.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:29.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:30.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:30.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:11:30.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[40510]: osdmap e201: 8 total, 8 up, 8 in 2024-03-21T14:11:30.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3377436818' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:30.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:30.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:11:30.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[29576]: osdmap e201: 8 total, 8 up, 8 in 2024-03-21T14:11:30.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3377436818' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:30.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:30.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:30.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:30.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:30.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:30.398 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:30.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:30.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:30.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:30.678 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:30.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:30.814 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:30.860 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:30.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:30 smithi118 ceph-mon[33862]: pgmap v374: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 213 MiB/s wr, 5.05k op/s; 48252/257964 objects misplaced (18.705%); 14 MiB/s, 224 objects/s recovering 2024-03-21T14:11:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:30 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:30 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:30.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:30 smithi003 ceph-mon[29576]: pgmap v374: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 213 MiB/s wr, 5.05k op/s; 48252/257964 objects misplaced (18.705%); 14 MiB/s, 224 objects/s recovering 2024-03-21T14:11:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:30 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:30 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:31.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:30 smithi003 ceph-mon[40510]: pgmap v374: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 213 MiB/s wr, 5.05k op/s; 48252/257964 objects misplaced (18.705%); 14 MiB/s, 224 objects/s recovering 2024-03-21T14:11:31.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:30 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:31.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:30 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:31.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:31.110 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:31.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:31.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:31.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.248 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:31.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:31.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:31.490 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:31.523 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:31.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:31.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:31.658 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:31.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:11:31.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:31.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:31.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:31.926 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:31.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:31.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:31 smithi118 ceph-mon[33862]: osdmap e202: 8 total, 8 up, 8 in 2024-03-21T14:11:31.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:31 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:31.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:32.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:31 smithi003 ceph-mon[29576]: osdmap e202: 8 total, 8 up, 8 in 2024-03-21T14:11:32.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:31 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:32.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:31 smithi003 ceph-mon[40510]: osdmap e202: 8 total, 8 up, 8 in 2024-03-21T14:11:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:31 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:11:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:32.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:32.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:32.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:32.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:32.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:32.352 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:32.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:32.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:11:32.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:32.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:32.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:32.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:32.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:32.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:32 smithi003 ceph-mon[29576]: pgmap v376: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 212 MiB/s wr, 5.20k op/s; 47901/255324 objects misplaced (18.761%); 14 MiB/s, 223 objects/s recovering 2024-03-21T14:11:32.744 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:32 smithi003 ceph-mon[40510]: pgmap v376: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 212 MiB/s wr, 5.20k op/s; 47901/255324 objects misplaced (18.761%); 14 MiB/s, 223 objects/s recovering 2024-03-21T14:11:32.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:32.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:32.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:32.866 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:183060 8.e e202/162 8:739c99aa:::benchmark_data_smithi003_110242_object52143:head v 202'11703, mlcod=202'11701)", 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:32.856843+0000", 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0061685359999999996, 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061983130000000004, 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.856843+0000", 2024-03-21T14:11:32.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.856843+0000", 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.856844+0000", 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7599999999999996e-07 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:32.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.856851+0000", 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.046e-06 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.856864+0000", 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.287e-05 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.856871+0000", 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2099999999999996e-06 2024-03-21T14:11:32.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857704+0000", 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00083304699999999995 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857723+0000", 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8572000000000001e-05 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:183065 8.14 e202/198 8:2d21e8f5:::benchmark_data_smithi003_110242_object52148:head v 202'11101, mlcod=202'11099)", 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:32.857756+0000", 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0052560749999999998, 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0053554140000000002, 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857756+0000", 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857756+0000", 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857757+0000", 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1699999999999997e-07 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857762+0000", 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4330000000000003e-06 2024-03-21T14:11:32.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857772+0000", 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8239999999999995e-06 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.857779+0000", 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8290000000000002e-06 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.858257+0000", 2024-03-21T14:11:32.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000478505 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.858283+0000", 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5426999999999999e-05 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:183068 8.d e202/162 8:b6d66d52:::benchmark_data_smithi003_110242_object52151:head v 202'11471, mlcod=202'11469)", 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:32.861914+0000", 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0010975430000000001, 2024-03-21T14:11:32.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001258686, 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.861914+0000", 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.861914+0000", 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.875 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.861915+0000", 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5000000000000001e-07 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.861923+0000", 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5730000000000002e-06 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.861942+0000", 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9303000000000001e-05 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.876 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.861950+0000", 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5859999999999999e-06 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862738+0000", 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00078794399999999995 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862762+0000", 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3872000000000001e-05 2024-03-21T14:11:32.877 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:183066 8.e e202/162 8:7f7dd0ce:::benchmark_data_smithi003_110242_object52149:head v 202'11704, mlcod=202'11701)", 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:32.862064+0000", 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00094799499999999998, 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001167976, 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862064+0000", 2024-03-21T14:11:32.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862064+0000", 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862065+0000", 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0800000000000004e-07 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862073+0000", 2024-03-21T14:11:32.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7489999999999998e-06 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862083+0000", 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0633e-05 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862091+0000", 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4909999999999999e-06 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:32.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862867+0000", 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00077663700000000005 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862888+0000", 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0305e-05 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:183069 8.14 e202/198 8:2e2655c3:::benchmark_data_smithi003_110242_object52152:head v 202'11102, mlcod=202'11099)", 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:32.862120+0000", 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00089220300000000005, 2024-03-21T14:11:32.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011722869999999999, 2024-03-21T14:11:32.882 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:32.882 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:32.882 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862120+0000", 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862120+0000", 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862120+0000", 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2e-07 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862125+0000", 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.386e-06 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862132+0000", 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9580000000000001e-06 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862137+0000", 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3029999999999999e-06 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862782+0000", 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064466499999999999 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862808+0000", 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6188999999999999e-05 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15132.0:183061 8.11 e202/198 8:88e10031:::benchmark_data_smithi003_110242_object52144:head v 202'11050, mlcod=202'11047)", 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:11:32.862784+0000", 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00022834, 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056686300000000002, 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:11:32.886 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862784+0000", 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862784+0000", 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:11:32.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862784+0000", 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2300000000000001e-07 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862790+0000", 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3929999999999999e-06 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862799+0000", 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6900000000000004e-06 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:11:32.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.862806+0000", 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.832e-06 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.863040+0000", 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000234107 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:11:32.863063+0000", 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.228e-05 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:32.890 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:32.890 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 6 2024-03-21T14:11:32.890 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:11:32.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:32.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:32 smithi118 ceph-mon[33862]: pgmap v376: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 212 MiB/s wr, 5.20k op/s; 47901/255324 objects misplaced (18.761%); 14 MiB/s, 223 objects/s recovering 2024-03-21T14:11:32.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:32.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:32.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:33.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:33.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:33.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:33.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:33.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:33.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:33.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:33.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:33.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:33.521 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:33.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:33.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.669 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:33.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:33.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:33.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.814 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:33.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:33.915 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:33.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:33.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:33.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:34.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:34.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:34.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:34.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:34.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:34.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:34.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:34.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T14:11:34.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:34.427 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:34.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:34.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:34.531 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:34.613 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:34.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:34.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:34.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:34.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:34.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:34.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:34.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:34.927 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:34.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:34 smithi118 ceph-mon[33862]: pgmap v377: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 969 op/s; 40730/240513 objects misplaced (16.935%); 22 MiB/s, 346 objects/s recovering 2024-03-21T14:11:34.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:34 smithi003 ceph-mon[29576]: pgmap v377: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 969 op/s; 40730/240513 objects misplaced (16.935%); 22 MiB/s, 346 objects/s recovering 2024-03-21T14:11:34.979 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:11:34.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:34 smithi003 ceph-mon[40510]: pgmap v377: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 969 op/s; 40730/240513 objects misplaced (16.935%); 22 MiB/s, 346 objects/s recovering 2024-03-21T14:11:34.981 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:11:34.981 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 202, 2024-03-21T14:11:34.981 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T14:11:34.981 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-03-21T13:56:54.530125+0000", 2024-03-21T14:11:34.981 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-03-21T14:11:30.639694+0000", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2024-03-21T14:02:19.321454+0000", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2024-03-21T14:02:01.236118+0000", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5806080, 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "nodeep-scrub", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "noscrub", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 18, 2024-03-21T14:11:34.982 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 8, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "luminous", 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "luminous", 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "squid", 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_crimson": false, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".mgr", 2024-03-21T14:11:34.983 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:00:12.925535+0000", 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2024-03-21T14:11:34.984 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "122", 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:34.985 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:34.986 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:34.987 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:11:34.988 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_max": 32, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": {} 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 7.8899998664855957, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 7.8899998664855957, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.37999999523162842, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 3, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 3, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:34.989 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "datapool", 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.373659+0000", 2024-03-21T14:11:34.990 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 3, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 3, 2024-03-21T14:11:34.991 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 3, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 3, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 3, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:11:34.992 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 2, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 68, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:34.993 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:34.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.6500000953674316, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.6500000953674316, 2024-03-21T14:11:34.995 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.87999999523162842, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-03-21T14:11:34.996 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".rgw.root", 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.794831+0000", 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:34.997 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:34.998 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "66", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:34.999 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:35.000 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:35.001 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.5, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.5, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:35.002 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 4, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.log", 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:33.750394+0000", 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:35.003 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:35.004 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:35.005 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:35.006 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:35.007 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.25, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.25, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:35.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 5, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.control", 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:35.711498+0000", 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:35.009 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:35.010 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "70", 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:35.011 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.012 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.013 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.5, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.5, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 6, 2024-03-21T14:11:35.014 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.meta", 2024-03-21T14:11:35.015 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:37.763124+0000", 2024-03-21T14:11:35.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:11:35.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:11:35.015 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:35.015 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:11:35.016 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "72", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:35.017 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:35.018 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:35.019 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.020 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.25, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.25, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 8, 2024-03-21T14:11:35.021 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "unique_pool_1", 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:08:59.821602+0000", 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:11:35.022 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 26, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 26, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 26, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 26, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 26, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "197", 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:11:35.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "195", 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:11:35.024 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:11:35.025 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "rados": {} 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:11:35.026 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.1500000953674316, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5399999618530273, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.1500000953674316, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.5399999618530273, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 1, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 1 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:11:35.027 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb", 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.73016357421875, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 8, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 198, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.028 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.029 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:11:35.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6801/930096433", 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6802/930096433", 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6804/930096433", 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6803/930096433", 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d", 2024-03-21T14:11:35.031 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 13, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 198, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.032 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.033 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.034 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6805/1526485482", 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6806/1526485482", 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6808/1526485482", 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/1526485482", 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c", 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 18, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 143, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.036 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2024-03-21T14:11:35.037 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.038 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6809/3751378277", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6810/3751378277", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6812/3751378277", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6811/3751378277", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.039 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c", 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.040 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.6562042236328125, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 25, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 198, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2024-03-21T14:11:35.041 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.042 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2024-03-21T14:11:35.043 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6813/1918209225", 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6814/1918209225", 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6816/1918209225", 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6815/1918209225", 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.044 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5", 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 30, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 198, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6800", 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6801", 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:11:35.046 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6803", 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6802", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6800/296472230", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6801/296472230", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6803/296472230", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6802/296472230", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "bff24605-38ce-47ce-9142-4ced3766476e", 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 40, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 198, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.049 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6804", 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6805", 2024-03-21T14:11:35.050 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6807", 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6806", 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6804/326385780", 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6805/326385780", 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6807/326385780", 2024-03-21T14:11:35.053 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6806/326385780", 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "73a781e7-8660-4885-b976-2aeb8515109a", 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.054 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 48, 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 197, 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6808", 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.055 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6809", 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6811", 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:11:35.056 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6810", 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6808/1322208109", 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6809/1322208109", 2024-03-21T14:11:35.057 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6811/1322208109", 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6810/1322208109", 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0", 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:11:35.058 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 58, 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 198, 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6812", 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:11:35.059 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6813", 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6815", 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:11:35.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6814", 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6812/2128678671", 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6813/2128678671", 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6815/2128678671", 2024-03-21T14:11:35.061 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6814/2128678671", 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.062 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:10:18.475650+0000", 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:29.858513+0000", 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.063 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:01.483011+0000", 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.064 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:07.436057+0000", 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:17.444073+0000", 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.065 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:02.427931+0000", 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.066 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:15.130075+0000", 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:11:27.016206+0000", 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.067 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [ 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.13", 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.0", 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.4", 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 5 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.b", 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.13", 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:11:35.070 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.1e", 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.071 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.0", 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.4", 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 5, 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.072 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.d", 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.13", 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:11:35.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_primaries": [], 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [ 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.10", 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-03-21T14:11:35.074 INFO:teuthology.orchestra.run.smithi003.stdout: 3 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.11", 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: 2 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.12", 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-03-21T14:11:35.075 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: 5 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.13", 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: 6 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.14", 2024-03-21T14:11:35.076 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: 4, 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: 2, 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: 5 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.15", 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: 7, 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: 4 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.077 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.16", 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: 7 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.17", 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: 1, 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: 3 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.18", 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: 3, 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: 7 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "8.19", 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: 5, 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: 6 2024-03-21T14:11:35.079 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklist": { 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/45234419": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/384540368": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3883474825": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/1598547836": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3687155565": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3884352348": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2319227492": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:11:35.080 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2154949954": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/4162970769": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2292819669": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3411869253": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/520312891": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2267426691": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3211426112": "2024-03-22T13:57:56.576834+0000" 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "range_blocklist": {}, 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2024-03-21T14:11:35.081 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {}, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": { 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_enabled": false, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_bucket_count": 0, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "degraded_stretch_mode": 0, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "recovering_stretch_mode": 0, 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_bucket": 0 2024-03-21T14:11:35.082 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:11:35.083 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:11:35.118 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:35.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:35.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:35.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.390 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:35.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:35.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:35.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:35.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:35.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:35.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:35.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:35.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3056614847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:11:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:35.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3056614847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:11:36.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3056614847' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:11:36.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:36.177 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:36.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:36.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:36.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:36.481 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:36.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:36.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:36.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:36.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:36.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:36 smithi003 ceph-mon[29576]: pgmap v378: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 747 op/s; 40730/240513 objects misplaced (16.935%); 17 MiB/s, 267 objects/s recovering 2024-03-21T14:11:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:11:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:11:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:11:36.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:36 smithi003 ceph-mon[40510]: pgmap v378: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 747 op/s; 40730/240513 objects misplaced (16.935%); 17 MiB/s, 267 objects/s recovering 2024-03-21T14:11:36.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:36.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:36.908 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:36.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:37.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:37.162 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:36 smithi118 ceph-mon[33862]: pgmap v378: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 657 B/s rd, 0 B/s wr, 747 op/s; 40730/240513 objects misplaced (16.935%); 17 MiB/s, 267 objects/s recovering 2024-03-21T14:11:37.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:11:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:11:37.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:37.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:37.320 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:37.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:37.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:37.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:37.595 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:37.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:37.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:37.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:37.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:37.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:37.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:37.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:38.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:38.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:38.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:38.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:38.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:38.288 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:38.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:38.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:38.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:38.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:38.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:38.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:38.700 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:38.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:38.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:38.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:38.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:38.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:38.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:38.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:38 smithi003 ceph-mon[29576]: pgmap v379: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 33733/224472 objects misplaced (15.028%); 30 MiB/s, 479 objects/s recovering 2024-03-21T14:11:39.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:38 smithi003 ceph-mon[40510]: pgmap v379: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 33733/224472 objects misplaced (15.028%); 30 MiB/s, 479 objects/s recovering 2024-03-21T14:11:39.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:39.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:38 smithi118 ceph-mon[33862]: pgmap v379: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.27k op/s; 33733/224472 objects misplaced (15.028%); 30 MiB/s, 479 objects/s recovering 2024-03-21T14:11:39.114 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:39.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:39.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:39.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:39.389 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:39.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:39.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:39.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:39.637 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:39.673 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:39.775 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.800 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:39.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:39.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:39.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:39.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:39.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:40.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:40.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:40.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:40.214 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:40.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:40.351 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:40.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:40.490 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:40.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:40.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:40.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:40.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:40.764 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:40.770 INFO:tasks.thrashosds.thrasher:j is {'epoch': 202, 'fsid': 'c60e082a-e78a-11ee-95cd-87774f69a715', 'created': '2024-03-21T13:56:54.530125+0000', 'modified': '2024-03-21T14:11:30.639694+0000', 'last_up_change': '2024-03-21T14:02:19.321454+0000', 'last_in_change': '2024-03-21T14:02:01.236118+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T14:00:12.925535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T14:02:31.373659+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T14:02:31.794831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T14:02:33.750394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T14:02:35.711498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T14:02:37.763124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-21T14:08:59.821602+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '197', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '195', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1500000953674316, 'score_stable': 1.5399999618530273, 'optimal_score': 1, 'raw_score_acting': 2.1500000953674316, 'raw_score_stable': 1.5399999618530273, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'daf9bc3c-4e73-4379-afb6-7ee181b686fb', 'up': 1, 'in': 1, 'weight': 0.73016357421875, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6801', 'nonce': 930096433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6802', 'nonce': 930096433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6804', 'nonce': 930096433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6803', 'nonce': 930096433}]}, 'public_addr': '172.21.15.3:6801/930096433', 'cluster_addr': '172.21.15.3:6802/930096433', 'heartbeat_back_addr': '172.21.15.3:6804/930096433', 'heartbeat_front_addr': '172.21.15.3:6803/930096433', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1652f24d-aa0e-471c-bf54-03814b1ee11d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6805', 'nonce': 1526485482}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6806', 'nonce': 1526485482}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6808', 'nonce': 1526485482}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6807', 'nonce': 1526485482}]}, 'public_addr': '172.21.15.3:6805/1526485482', 'cluster_addr': '172.21.15.3:6806/1526485482', 'heartbeat_back_addr': '172.21.15.3:6808/1526485482', 'heartbeat_front_addr': '172.21.15.3:6807/1526485482', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5e67d907-f8d1-4ecc-85d3-df88da5c364c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 143, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6809', 'nonce': 3751378277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6810', 'nonce': 3751378277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6812', 'nonce': 3751378277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6811', 'nonce': 3751378277}]}, 'public_addr': '172.21.15.3:6809/3751378277', 'cluster_addr': '172.21.15.3:6810/3751378277', 'heartbeat_back_addr': '172.21.15.3:6812/3751378277', 'heartbeat_front_addr': '172.21.15.3:6811/3751378277', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'ad8dabdf-fce5-4312-9e80-dbbb6ba2947c', 'up': 1, 'in': 1, 'weight': 0.6562042236328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6813', 'nonce': 1918209225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6814', 'nonce': 1918209225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6816', 'nonce': 1918209225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6815', 'nonce': 1918209225}]}, 'public_addr': '172.21.15.3:6813/1918209225', 'cluster_addr': '172.21.15.3:6814/1918209225', 'heartbeat_back_addr': '172.21.15.3:6816/1918209225', 'heartbeat_front_addr': '172.21.15.3:6815/1918209225', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b5d486b1-2cb6-46c7-8664-994af07ff7b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 296472230}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 296472230}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 296472230}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 296472230}]}, 'public_addr': '172.21.15.118:6800/296472230', 'cluster_addr': '172.21.15.118:6801/296472230', 'heartbeat_back_addr': '172.21.15.118:6803/296472230', 'heartbeat_front_addr': '172.21.15.118:6802/296472230', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bff24605-38ce-47ce-9142-4ced3766476e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 326385780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 326385780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 326385780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 326385780}]}, 'public_addr': '172.21.15.118:6804/326385780', 'cluster_addr': '172.21.15.118:6805/326385780', 'heartbeat_back_addr': '172.21.15.118:6807/326385780', 'heartbeat_front_addr': '172.21.15.118:6806/326385780', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '73a781e7-8660-4885-b976-2aeb8515109a', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 197, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 1322208109}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 1322208109}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 1322208109}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 1322208109}]}, 'public_addr': '172.21.15.118:6808/1322208109', 'cluster_addr': '172.21.15.118:6809/1322208109', 'heartbeat_back_addr': '172.21.15.118:6811/1322208109', 'heartbeat_front_addr': '172.21.15.118:6810/1322208109', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3e95559b-79f6-40ba-9d98-d350bcdd60e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 2128678671}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 2128678671}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 2128678671}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 2128678671}]}, 'public_addr': '172.21.15.118:6812/2128678671', 'cluster_addr': '172.21.15.118:6813/2128678671', 'heartbeat_back_addr': '172.21.15.118:6815/2128678671', 'heartbeat_front_addr': '172.21.15.118:6814/2128678671', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:10:18.475650+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:29.858513+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:01.483011+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:07.436057+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:17.444073+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:02.427931+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:15.130075+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:11:27.016206+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.13', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.0', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.0', 'mappings': [{'from': 3, 'to': 6}]}, {'pgid': '6.4', 'mappings': [{'from': 5, 'to': 0}]}, {'pgid': '6.d', 'mappings': [{'from': 2, 'to': 6}]}, {'pgid': '6.13', 'mappings': [{'from': 2, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.10', 'osds': [1, 5, 3]}, {'pgid': '8.11', 'osds': [0, 1, 2]}, {'pgid': '8.12', 'osds': [0, 4, 5]}, {'pgid': '8.13', 'osds': [1, 5, 6]}, {'pgid': '8.14', 'osds': [4, 2, 5]}, {'pgid': '8.15', 'osds': [7, 6, 4]}, {'pgid': '8.16', 'osds': [0, 6, 7]}, {'pgid': '8.17', 'osds': [1, 6, 3]}, {'pgid': '8.18', 'osds': [3, 5, 7]}, {'pgid': '8.19', 'osds': [5, 0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.3:0/45234419': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/384540368': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3883474825': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/1598547836': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:6800/3687155565': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3884352348': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2319227492': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2154949954': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/4162970769': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/2292819669': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/3411869253': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/520312891': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2267426691': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/3211426112': '2024-03-22T13:57:56.576834+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T14:11:40.770 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-21T14:11:40.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:40.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:40.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:40.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:40.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:40 smithi003 ceph-mon[29576]: pgmap v380: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 33733/224472 objects misplaced (15.028%); 26 MiB/s, 419 objects/s recovering 2024-03-21T14:11:40.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:40 smithi003 ceph-mon[40510]: pgmap v380: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 33733/224472 objects misplaced (15.028%); 26 MiB/s, 419 objects/s recovering 2024-03-21T14:11:41.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:41.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:41.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:41.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:41.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:40 smithi118 ceph-mon[33862]: pgmap v380: 158 pgs: 6 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 148 active+clean; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 33733/224472 objects misplaced (15.028%); 26 MiB/s, 419 objects/s recovering 2024-03-21T14:11:41.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:41.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:41.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:41.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:41.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:41.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:41.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:41.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:41.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:41.688 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:41.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:41.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:41.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:41.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:41.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:41 smithi003 ceph-mon[29576]: osdmap e203: 8 total, 8 up, 8 in 2024-03-21T14:11:42.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:41 smithi003 ceph-mon[40510]: osdmap e203: 8 total, 8 up, 8 in 2024-03-21T14:11:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:42.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:42.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:42.139 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:41 smithi118 ceph-mon[33862]: osdmap e203: 8 total, 8 up, 8 in 2024-03-21T14:11:42.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.275 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:42.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:42.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:42.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:42.536 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:42.572 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:42.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.705 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:42.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:42.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:42.839 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:42.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:42.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:42.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:42 smithi003 ceph-mon[29576]: pgmap v382: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 32888/221589 objects misplaced (14.842%); 29 MiB/s, 469 objects/s recovering 2024-03-21T14:11:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:42 smithi003 ceph-mon[29576]: osdmap e204: 8 total, 8 up, 8 in 2024-03-21T14:11:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:42 smithi003 ceph-mon[40510]: pgmap v382: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 32888/221589 objects misplaced (14.842%); 29 MiB/s, 469 objects/s recovering 2024-03-21T14:11:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:42 smithi003 ceph-mon[40510]: osdmap e204: 8 total, 8 up, 8 in 2024-03-21T14:11:43.050 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:11:43.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:43.102 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:11:43.123 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:43.143 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:42 smithi118 ceph-mon[33862]: pgmap v382: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 32888/221589 objects misplaced (14.842%); 29 MiB/s, 469 objects/s recovering 2024-03-21T14:11:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:42 smithi118 ceph-mon[33862]: osdmap e204: 8 total, 8 up, 8 in 2024-03-21T14:11:43.185 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:11:43.225 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:43.230 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:11:43.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:43.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:43.273 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:11:43.310 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:11:43.330 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:11:43.331 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:11:43.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:43.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:43.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:43.538 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:43.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:43.682 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:43.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:43.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:43.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:43.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:43.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:43 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2259300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:43 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2259300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:44.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:44.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:44.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:43 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2259300515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:11:44.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:44.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:44.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:44.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:44.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:44.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:44.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:44.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:44.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:44.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:44.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:44.925 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:44.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:44.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:45.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:45.059 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:45.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:45.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:44 smithi118 ceph-mon[33862]: pgmap v384: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 26253/209115 objects misplaced (12.554%); 34 MiB/s, 546 objects/s recovering 2024-03-21T14:11:45.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:45.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:45.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:44 smithi003 ceph-mon[29576]: pgmap v384: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 26253/209115 objects misplaced (12.554%); 34 MiB/s, 546 objects/s recovering 2024-03-21T14:11:45.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:44 smithi003 ceph-mon[40510]: pgmap v384: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s; 26253/209115 objects misplaced (12.554%); 34 MiB/s, 546 objects/s recovering 2024-03-21T14:11:45.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.327 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:45.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:45.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:45.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.465 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:45.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:45.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:45.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:45.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:45.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.731 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:45.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:45.772 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:11:45.772 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:11:45.772 INFO:tasks.thrashosds.thrasher:Reweighting osd 6 to 0.2286676370994563 2024-03-21T14:11:45.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 6 0.2286676370994563 2024-03-21T14:11:45.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:45.866 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:45.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:45.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:46.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:46.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:46.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:46.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:46.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:46.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:46.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:46.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:46.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:46.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:46.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:46.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:46.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:46.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:46.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:11:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:11:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:11:46.778 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:46.811 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:46.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:46.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:46.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:46.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:47.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:47.075 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:47.177 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:46 smithi118 ceph-mon[33862]: pgmap v385: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 640 op/s; 26253/209115 objects misplaced (12.554%); 18 MiB/s, 281 objects/s recovering 2024-03-21T14:11:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:47.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:11:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:11:47.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:47.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:46 smithi003 ceph-mon[29576]: pgmap v385: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 640 op/s; 26253/209115 objects misplaced (12.554%); 18 MiB/s, 281 objects/s recovering 2024-03-21T14:11:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:47.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:46 smithi003 ceph-mon[40510]: pgmap v385: 158 pgs: 1 peering, 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 148 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 640 op/s; 26253/209115 objects misplaced (12.554%); 18 MiB/s, 281 objects/s recovering 2024-03-21T14:11:47.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:11:47.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:47.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:47.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:47.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:47.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:47.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:47.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:47.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:47.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:47.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.718 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:47.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:47.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:47.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:47.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:47.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:47.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:47.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:48.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:48.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:47 smithi118 ceph-mon[33862]: osdmap e205: 8 total, 8 up, 8 in 2024-03-21T14:11:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:47 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:11:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:47 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:11:48.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[29576]: osdmap e205: 8 total, 8 up, 8 in 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[40510]: osdmap e205: 8 total, 8 up, 8 in 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:11:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:47 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:11:48.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:48.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:48.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:48.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:48.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:48.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:48.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:48.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:48.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:48.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.826 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:48.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:48.927 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:48.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:48.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:49.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:49.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:48 smithi003 ceph-mon[29576]: pgmap v387: 158 pgs: 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 149 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s; 22666/194799 objects misplaced (11.636%); 28 MiB/s, 450 objects/s recovering 2024-03-21T14:11:49.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:48 smithi003 ceph-mon[29576]: osdmap e206: 8 total, 8 up, 8 in 2024-03-21T14:11:49.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:48 smithi003 ceph-mon[40510]: pgmap v387: 158 pgs: 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 149 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s; 22666/194799 objects misplaced (11.636%); 28 MiB/s, 450 objects/s recovering 2024-03-21T14:11:49.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:48 smithi003 ceph-mon[40510]: osdmap e206: 8 total, 8 up, 8 in 2024-03-21T14:11:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:49.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:48 smithi118 ceph-mon[33862]: pgmap v387: 158 pgs: 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 149 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.48k op/s; 22666/194799 objects misplaced (11.636%); 28 MiB/s, 450 objects/s recovering 2024-03-21T14:11:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:48 smithi118 ceph-mon[33862]: osdmap e206: 8 total, 8 up, 8 in 2024-03-21T14:11:49.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:49.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:49.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:49.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:49.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:49.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:49.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:49.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:49.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.603 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:49.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:49.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:49.740 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:49.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:49.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:49.878 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:49.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:49.979 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:50.016 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:50.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:50.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:50.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:49 smithi118 ceph-mon[33862]: osdmap e207: 8 total, 8 up, 8 in 2024-03-21T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3792230777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:49 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:49 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:49 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:11:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:49 smithi118 ceph-mon[33862]: osdmap e208: 8 total, 8 up, 8 in 2024-03-21T14:11:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[29576]: osdmap e207: 8 total, 8 up, 8 in 2024-03-21T14:11:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3792230777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:11:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:11:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:11:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[29576]: osdmap e208: 8 total, 8 up, 8 in 2024-03-21T14:11:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[40510]: osdmap e207: 8 total, 8 up, 8 in 2024-03-21T14:11:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3792230777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:11:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:11:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:11:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:11:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:49 smithi003 ceph-mon[40510]: osdmap e208: 8 total, 8 up, 8 in 2024-03-21T14:11:50.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:50.257 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:11:50.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:50.395 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:50.427 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:50.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:50.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:50.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:50.702 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:50.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:50.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:50.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:50.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:50.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:51.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:51.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:50 smithi118 ceph-mon[33862]: pgmap v390: 158 pgs: 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 149 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 22666/194799 objects misplaced (11.636%); 7.9 MiB/s, 127 objects/s recovering 2024-03-21T14:11:51.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:50 smithi118 ceph-mon[33862]: 8.12 scrub starts 2024-03-21T14:11:51.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:50 smithi118 ceph-mon[33862]: 8.17 scrub starts 2024-03-21T14:11:51.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:50 smithi118 ceph-mon[33862]: 8.2 scrub starts 2024-03-21T14:11:51.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:50 smithi118 ceph-mon[33862]: 8.17 scrub ok 2024-03-21T14:11:51.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:51.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[29576]: pgmap v390: 158 pgs: 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 149 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 22666/194799 objects misplaced (11.636%); 7.9 MiB/s, 127 objects/s recovering 2024-03-21T14:11:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[29576]: 8.12 scrub starts 2024-03-21T14:11:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[29576]: 8.17 scrub starts 2024-03-21T14:11:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[29576]: 8.2 scrub starts 2024-03-21T14:11:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[29576]: 8.17 scrub ok 2024-03-21T14:11:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[40510]: pgmap v390: 158 pgs: 6 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 149 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 795 op/s; 22666/194799 objects misplaced (11.636%); 7.9 MiB/s, 127 objects/s recovering 2024-03-21T14:11:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[40510]: 8.12 scrub starts 2024-03-21T14:11:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[40510]: 8.17 scrub starts 2024-03-21T14:11:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[40510]: 8.2 scrub starts 2024-03-21T14:11:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:50 smithi003 ceph-mon[40510]: 8.17 scrub ok 2024-03-21T14:11:51.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:51.375 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:51.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:51.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:51.504 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:51.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:51.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:51.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:51.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:51.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:51.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 8.0 scrub starts 2024-03-21T14:11:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 8.12 scrub ok 2024-03-21T14:11:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 8.0 scrub ok 2024-03-21T14:11:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 8.9 scrub starts 2024-03-21T14:11:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 8.2 scrub ok 2024-03-21T14:11:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 8.0 scrub starts 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 8.12 scrub ok 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 8.0 scrub ok 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 8.9 scrub starts 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 8.2 scrub ok 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:11:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:51 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:11:52.033 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:52.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:52.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 8.0 scrub starts 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 8.12 scrub ok 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 8.0 scrub ok 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 8.9 scrub starts 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 8.2 scrub ok 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:11:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:51 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:11:52.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:52.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:52.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:52.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:52.446 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:52.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:52.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:52.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:52.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:52.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.762 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:52.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:52.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:52.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:52.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:53.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[29576]: 8.9 scrub ok 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[29576]: pgmap v392: 158 pgs: 6 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 150 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 21859/192210 objects misplaced (11.372%) 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[29576]: 8.5 scrub starts 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[29576]: 8.5 scrub ok 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[40510]: 8.9 scrub ok 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[40510]: pgmap v392: 158 pgs: 6 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 150 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 21859/192210 objects misplaced (11.372%) 2024-03-21T14:11:53.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[40510]: 8.5 scrub starts 2024-03-21T14:11:53.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:52 smithi003 ceph-mon[40510]: 8.5 scrub ok 2024-03-21T14:11:53.032 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:53.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:53.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:52 smithi118 ceph-mon[33862]: 8.9 scrub ok 2024-03-21T14:11:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:52 smithi118 ceph-mon[33862]: pgmap v392: 158 pgs: 6 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 150 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 21859/192210 objects misplaced (11.372%) 2024-03-21T14:11:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:52 smithi118 ceph-mon[33862]: 8.5 scrub starts 2024-03-21T14:11:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:52 smithi118 ceph-mon[33862]: 8.5 scrub ok 2024-03-21T14:11:53.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:53.310 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:53.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:53.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.453 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:53.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:53.554 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:53.586 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:53.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:53.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:53.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:53.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:53.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:53.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:53.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:54.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:54.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:11:54.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:11:54.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[29576]: 8.1 scrub starts 2024-03-21T14:11:54.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[29576]: 6.2 scrub starts 2024-03-21T14:11:54.107 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[29576]: 6.2 scrub ok 2024-03-21T14:11:54.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:54.113 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:11:54.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:11:54.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[40510]: 8.1 scrub starts 2024-03-21T14:11:54.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[40510]: 6.2 scrub starts 2024-03-21T14:11:54.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:53 smithi003 ceph-mon[40510]: 6.2 scrub ok 2024-03-21T14:11:54.145 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:54.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:53 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:53 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:53 smithi118 ceph-mon[33862]: 8.1 scrub starts 2024-03-21T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:53 smithi118 ceph-mon[33862]: 6.2 scrub starts 2024-03-21T14:11:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:53 smithi118 ceph-mon[33862]: 6.2 scrub ok 2024-03-21T14:11:54.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:54.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:54.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:54.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:54.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:54.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:54.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:54.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:54.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:54.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:54.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:54.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:54.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:54.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:54.920 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:54.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:55.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:55.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[29576]: pgmap v393: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18896/182178 objects misplaced (10.372%) 2024-03-21T14:11:55.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[29576]: 8.1 scrub ok 2024-03-21T14:11:55.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[29576]: 8.6 scrub starts 2024-03-21T14:11:55.071 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[29576]: osdmap e209: 8 total, 8 up, 8 in 2024-03-21T14:11:55.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[40510]: pgmap v393: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18896/182178 objects misplaced (10.372%) 2024-03-21T14:11:55.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[40510]: 8.1 scrub ok 2024-03-21T14:11:55.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[40510]: 8.6 scrub starts 2024-03-21T14:11:55.072 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:54 smithi003 ceph-mon[40510]: osdmap e209: 8 total, 8 up, 8 in 2024-03-21T14:11:55.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:55.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:54 smithi118 ceph-mon[33862]: pgmap v393: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18896/182178 objects misplaced (10.372%) 2024-03-21T14:11:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:54 smithi118 ceph-mon[33862]: 8.1 scrub ok 2024-03-21T14:11:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:54 smithi118 ceph-mon[33862]: 8.6 scrub starts 2024-03-21T14:11:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:54 smithi118 ceph-mon[33862]: osdmap e209: 8 total, 8 up, 8 in 2024-03-21T14:11:55.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:55.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:55.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:55.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:55.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:55.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:11:55.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:55.562 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:55.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:11:55.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:55.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:11:55.773 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:11:55.773 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:11:55.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:55.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:55.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:55.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:56.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:56.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:56.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:55 smithi003 ceph-mon[29576]: 8.6 scrub ok 2024-03-21T14:11:56.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:55 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:11:56.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:55 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:11:56.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:55 smithi003 ceph-mon[40510]: 8.6 scrub ok 2024-03-21T14:11:56.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:55 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:11:56.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:55 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:11:56.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:11:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.160 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:56.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:55 smithi118 ceph-mon[33862]: 8.6 scrub ok 2024-03-21T14:11:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:55 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:11:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:55 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:11:56.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:56.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:56.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:56.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:56.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:56.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:56.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:56.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:11:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:11:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:11:56.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:56.886 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:56.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:56.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:57.021 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:57.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:57.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:56 smithi118 ceph-mon[33862]: pgmap v395: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 772 B/s rd, 0 op/s; 18896/182178 objects misplaced (10.372%) 2024-03-21T14:11:57.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:11:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:11:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:56 smithi003 ceph-mon[29576]: pgmap v395: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 772 B/s rd, 0 op/s; 18896/182178 objects misplaced (10.372%) 2024-03-21T14:11:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:56 smithi003 ceph-mon[40510]: pgmap v395: 158 pgs: 6 active+remapped+backfill_toofull, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 772 B/s rd, 0 op/s; 18896/182178 objects misplaced (10.372%) 2024-03-21T14:11:57.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:57.313 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:57.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:57.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:57.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:57.584 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:57.686 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:57.721 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:57.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:57.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:11:57.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:57.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:11:57.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:58.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:58.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:57 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:11:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:57 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:11:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:57 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:11:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:57 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:11:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:58.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:58.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T14:11:58.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:58.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:57 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:57 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:11:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:11:58.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.276 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:58.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:58.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:58.428 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:58.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:58.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:58.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:11:58.709 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:58.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:11:58.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:58.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:58.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:11:58.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[29576]: pgmap v396: 158 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15214/163899 objects misplaced (9.283%) 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[40510]: pgmap v396: 158 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15214/163899 objects misplaced (9.283%) 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:11:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:58 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:11:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:11:59.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:11:59.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.149 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:58 smithi118 ceph-mon[33862]: pgmap v396: 158 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15214/163899 objects misplaced (9.283%) 2024-03-21T14:11:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:58 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:11:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:58 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:11:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:58 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:11:59.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:11:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.285 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:11:59.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:11:59.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:11:59.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:59.525 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 0.228668 (3a89) 2024-03-21T14:11:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.561 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:11:59.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:59.691 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:59.793 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:11:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.819 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:11:59.921 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:11:59.946 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:11:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:11:59.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:00.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:00.073 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:00.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:59 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:12:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:59 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:12:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1444744982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]: dispatch 2024-03-21T14:12:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:59 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]: dispatch 2024-03-21T14:12:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:59 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]': finished 2024-03-21T14:12:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:11:59 smithi118 ceph-mon[33862]: osdmap e210: 8 total, 8 up, 8 in 2024-03-21T14:12:00.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:12:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:12:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1444744982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]: dispatch 2024-03-21T14:12:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]: dispatch 2024-03-21T14:12:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]': finished 2024-03-21T14:12:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[29576]: osdmap e210: 8 total, 8 up, 8 in 2024-03-21T14:12:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:12:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:12:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1444744982' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]: dispatch 2024-03-21T14:12:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]: dispatch 2024-03-21T14:12:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 0.2286676370994563}]': finished 2024-03-21T14:12:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:11:59 smithi003 ceph-mon[40510]: osdmap e210: 8 total, 8 up, 8 in 2024-03-21T14:12:00.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:00.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:00.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.456 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:00.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:00.557 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:00.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.584 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:00.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:00.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.709 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:00.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:00.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:00.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:00.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:00.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:00.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:00.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:01.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:01.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:01.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:00 smithi118 ceph-mon[33862]: pgmap v397: 158 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 15214/163899 objects misplaced (9.283%) 2024-03-21T14:12:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:00 smithi118 ceph-mon[33862]: 8.19 scrub starts 2024-03-21T14:12:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:00 smithi118 ceph-mon[33862]: osdmap e211: 8 total, 8 up, 8 in 2024-03-21T14:12:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:00 smithi118 ceph-mon[33862]: 8.15 starting backfill to osd.0 from (0'0,0'0] MAX to 209'11774 2024-03-21T14:12:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[29576]: pgmap v397: 158 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 15214/163899 objects misplaced (9.283%) 2024-03-21T14:12:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[29576]: 8.19 scrub starts 2024-03-21T14:12:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[29576]: osdmap e211: 8 total, 8 up, 8 in 2024-03-21T14:12:01.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[29576]: 8.15 starting backfill to osd.0 from (0'0,0'0] MAX to 209'11774 2024-03-21T14:12:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[40510]: pgmap v397: 158 pgs: 3 active+remapped+backfilling, 3 active+remapped+backfill_toofull, 152 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 15214/163899 objects misplaced (9.283%) 2024-03-21T14:12:01.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[40510]: 8.19 scrub starts 2024-03-21T14:12:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[40510]: osdmap e211: 8 total, 8 up, 8 in 2024-03-21T14:12:01.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:00 smithi003 ceph-mon[40510]: 8.15 starting backfill to osd.0 from (0'0,0'0] MAX to 209'11774 2024-03-21T14:12:01.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:01.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:01.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:01.334 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:01.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:01.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:01.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:01.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:01.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:01.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:01.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:01.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:01.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:01.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:01.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:01.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:01.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:02.012 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:02.024 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:02.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:02.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:02.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 8.19 scrub ok 2024-03-21T14:12:02.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 8.15 continuing backfill to osd.1 from (174'4900,209'11774] 8:ac87b183:::benchmark_data_smithi003_110242_object82779:head to 209'11774 2024-03-21T14:12:02.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:12:02.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:12:02.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 8.a scrub starts 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 8.12 starting backfill to osd.4 from (0'0,0'0] MAX to 209'12121 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: osdmap e212: 8 total, 8 up, 8 in 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 209'11756 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[29576]: 8.17 starting backfill to osd.4 from (0'0,0'0] MAX to 209'11875 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 8.19 scrub ok 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 8.15 continuing backfill to osd.1 from (174'4900,209'11774] 8:ac87b183:::benchmark_data_smithi003_110242_object82779:head to 209'11774 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:12:02.164 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 8.a scrub starts 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 8.12 starting backfill to osd.4 from (0'0,0'0] MAX to 209'12121 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: osdmap e212: 8 total, 8 up, 8 in 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 209'11756 2024-03-21T14:12:02.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:01 smithi003 ceph-mon[40510]: 8.17 starting backfill to osd.4 from (0'0,0'0] MAX to 209'11875 2024-03-21T14:12:02.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 8.19 scrub ok 2024-03-21T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 8.15 continuing backfill to osd.1 from (174'4900,209'11774] 8:ac87b183:::benchmark_data_smithi003_110242_object82779:head to 209'11774 2024-03-21T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:12:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 8.a scrub starts 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 8.12 starting backfill to osd.4 from (0'0,0'0] MAX to 209'12121 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: osdmap e212: 8 total, 8 up, 8 in 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 8.5 starting backfill to osd.0 from (0'0,0'0] MAX to 209'11756 2024-03-21T14:12:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:01 smithi118 ceph-mon[33862]: 8.17 starting backfill to osd.4 from (0'0,0'0] MAX to 209'11875 2024-03-21T14:12:02.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:02.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:02.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:02.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:02.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:02.520 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:12:02.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:02.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:02.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:02.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:02.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:02.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:02.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:02.889 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:02.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:02.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:03.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:03.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:03.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: pgmap v400: 158 pgs: 4 remapped+peering, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 16090/160194 objects misplaced (10.044%); 18 MiB/s, 288 objects/s recovering 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 209'11926 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 209'12006 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: 8.11 starting backfill to osd.1 from (0'0,0'0] MAX to 209'11932 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 209'11855 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: 8.a scrub ok 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: 8.d scrub starts 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1360308450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:12:03.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1360308450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:03.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:02 smithi118 ceph-mon[33862]: osdmap e213: 8 total, 8 up, 8 in 2024-03-21T14:12:03.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: pgmap v400: 158 pgs: 4 remapped+peering, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 16090/160194 objects misplaced (10.044%); 18 MiB/s, 288 objects/s recovering 2024-03-21T14:12:03.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 209'11926 2024-03-21T14:12:03.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 209'12006 2024-03-21T14:12:03.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: 8.11 starting backfill to osd.1 from (0'0,0'0] MAX to 209'11932 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 209'11855 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: 8.a scrub ok 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: 8.d scrub starts 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1360308450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1360308450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:03.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[29576]: osdmap e213: 8 total, 8 up, 8 in 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: pgmap v400: 158 pgs: 4 remapped+peering, 2 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 148 active+clean; 3.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 16090/160194 objects misplaced (10.044%); 18 MiB/s, 288 objects/s recovering 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: 8.7 starting backfill to osd.2 from (0'0,0'0] MAX to 209'11926 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: 8.3 starting backfill to osd.2 from (0'0,0'0] MAX to 209'12006 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: 8.11 starting backfill to osd.1 from (0'0,0'0] MAX to 209'11932 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: 8.6 starting backfill to osd.2 from (0'0,0'0] MAX to 209'11855 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: 8.a scrub ok 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: 8.d scrub starts 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1360308450' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1360308450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:03.259 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:02 smithi003 ceph-mon[40510]: osdmap e213: 8 total, 8 up, 8 in 2024-03-21T14:12:03.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.331 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:03.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:03.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:03.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:03.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:03.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:03.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:03.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:03.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:03.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:03.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:03.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:04.029 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:04.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:04.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[40510]: 8.d scrub ok 2024-03-21T14:12:04.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:12:04.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:12:04.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[40510]: osdmap e214: 8 total, 8 up, 8 in 2024-03-21T14:12:04.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[29576]: 8.d scrub ok 2024-03-21T14:12:04.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:12:04.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:12:04.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:03 smithi003 ceph-mon[29576]: osdmap e214: 8 total, 8 up, 8 in 2024-03-21T14:12:04.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:04.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:03 smithi118 ceph-mon[33862]: 8.d scrub ok 2024-03-21T14:12:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:03 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:12:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:03 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:12:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:03 smithi118 ceph-mon[33862]: osdmap e214: 8 total, 8 up, 8 in 2024-03-21T14:12:04.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:04.308 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:04.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:04.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:04.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:04.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:04.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:04.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:04.729 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:04.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:04.867 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:04.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:04.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:04.876 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:12:04.928 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:12:04.969 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:04.972 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:12:05.004 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:05.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:05.013 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:12:05.062 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:12:05.106 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:12:05.106 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.146 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:05.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:05.149 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:12:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:04 smithi118 ceph-mon[33862]: pgmap v403: 158 pgs: 1 active+remapped, 1 peering, 5 remapped+peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 17531/152862 objects misplaced (11.469%); 9.7 MiB/s, 171 objects/s recovering 2024-03-21T14:12:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:04 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:04 smithi118 ceph-mon[33862]: 8.8 scrub starts 2024-03-21T14:12:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/315374139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:05.194 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:12:05.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[29576]: pgmap v403: 158 pgs: 1 active+remapped, 1 peering, 5 remapped+peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 17531/152862 objects misplaced (11.469%); 9.7 MiB/s, 171 objects/s recovering 2024-03-21T14:12:05.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:05.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[29576]: 8.8 scrub starts 2024-03-21T14:12:05.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/315374139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:05.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[40510]: pgmap v403: 158 pgs: 1 active+remapped, 1 peering, 5 remapped+peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 17531/152862 objects misplaced (11.469%); 9.7 MiB/s, 171 objects/s recovering 2024-03-21T14:12:05.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:05.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[40510]: 8.8 scrub starts 2024-03-21T14:12:05.195 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/315374139' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:05.200 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:12:05.201 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:12:05.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:05.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:05.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:05.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:12:05.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:05.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:05.557 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:05.598 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:05.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:05.736 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:05.839 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:05.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:05.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:05.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:05.993 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:12:05.993 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:12:05.994 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T14:12:05.994 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 0.000000 2024-03-21T14:12:05.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 0 2024-03-21T14:12:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:05 smithi003 ceph-mon[29576]: 8.8 scrub ok 2024-03-21T14:12:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:05 smithi003 ceph-mon[29576]: 8.4 scrub starts 2024-03-21T14:12:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:05 smithi003 ceph-mon[40510]: 8.8 scrub ok 2024-03-21T14:12:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:05 smithi003 ceph-mon[40510]: 8.4 scrub starts 2024-03-21T14:12:06.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:06.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:06.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:05 smithi118 ceph-mon[33862]: 8.8 scrub ok 2024-03-21T14:12:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:05 smithi118 ceph-mon[33862]: 8.4 scrub starts 2024-03-21T14:12:06.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:06.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:06.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:06.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:06.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:06.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:06.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:06.748 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:12:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:12:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:12:06.850 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:06.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:06.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:06.885 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:06.892 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:140727 7.e 7:7c6116bf:::benchmark_data_smithi003_80362_object19710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:12:06.898 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:30.549679+0000", 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 336.34472898000001, 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.140970928, 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140727 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549679+0000", 2024-03-21T14:12:06.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549679+0000", 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549681+0000", 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.576e-06 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549688+0000", 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1790000000000002e-06 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549710+0000", 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2541e-05 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549722+0000", 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1779e-05 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549778+0000", 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5767999999999999e-05 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.549894+0000", 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011581 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.550015+0000", 2024-03-21T14:12:06.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121532 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.553472+0000", 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003456359 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.555122+0000", 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016506820000000001 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.690582+0000", 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13545950400000001 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.690618+0000", 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5834000000000003e-05 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:30.690650+0000", 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2363999999999997e-05 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.904 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:141678 7.e 7:7e298d8b:::benchmark_data_smithi003_80362_object20661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:31.587643+0000", 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 335.30676533799999, 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.132807325, 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.905 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141678 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587643+0000", 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587643+0000", 2024-03-21T14:12:06.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587644+0000", 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.587e-06 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587651+0000", 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4439999999999996e-06 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587667+0000", 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6446000000000001e-05 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.587676+0000", 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2539999999999998e-06 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.597199+0000", 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0095225520000000001 2024-03-21T14:12:06.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.597318+0000", 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118934 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.597436+0000", 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118468 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.909 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.602852+0000", 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0054152159999999996 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.603071+0000", 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000219435 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.720379+0000", 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.117308329 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.720418+0000", 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9060999999999998e-05 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.720450+0000", 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1599e-05 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151001 7.e 7:71a6f342:::benchmark_data_smithi003_80362_object29984:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.792444+0000", 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 321.10196368800001, 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.150560518, 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151001 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792444+0000", 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792444+0000", 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792446+0000", 2024-03-21T14:12:06.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6220000000000001e-06 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792452+0000", 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.212e-06 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792464+0000", 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2109e-05 2024-03-21T14:12:06.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792473+0000", 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.619e-06 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792566+0000", 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3263999999999993e-05 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792699+0000", 2024-03-21T14:12:06.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013310000000000001 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792812+0000", 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112281 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.800854+0000", 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080425960000000008 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.801137+0000", 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000282422 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.942687+0000", 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14155068200000001 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.942852+0000", 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016438800000000001 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943005+0000", 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015322299999999999 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151003 7.e 7:783de3cd:::benchmark_data_smithi003_80362_object29986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.792496+0000", 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 321.10191195099998, 2024-03-21T14:12:06.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15094943299999999, 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151003 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792496+0000", 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792496+0000", 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792497+0000", 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9699999999999996e-07 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792500+0000", 2024-03-21T14:12:06.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1159999999999999e-06 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792507+0000", 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1709999999999996e-06 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.792513+0000", 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6810000000000003e-06 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.796583+0000", 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040704900000000004 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.796702+0000", 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011917999999999999 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.796826+0000", 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123504 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.801070+0000", 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042447459999999998 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.801160+0000", 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9766999999999996e-05 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943163+0000", 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.142002613 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943313+0000", 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000150588 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943446+0000", 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013208 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.924 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151012 7.e 7:79355c2f:::benchmark_data_smithi003_80362_object29995:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.807247+0000", 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 321.08716077899999, 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.136874252, 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151012 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807247+0000", 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807247+0000", 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807249+0000", 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1229999999999998e-06 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807257+0000", 2024-03-21T14:12:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5020000000000003e-06 2024-03-21T14:12:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807277+0000", 2024-03-21T14:12:06.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9769999999999999e-05 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807286+0000", 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5189999999999998e-06 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.807489+0000", 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000203301 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.808025+0000", 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053579400000000003 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.808323+0000", 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029820500000000001 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825027+0000", 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016703273000000001 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.834701+0000", 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096738910000000004 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.943838+0000", 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109137516 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.944000+0000", 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016142000000000001 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.944122+0000", 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121938 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.931 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151016 7.e 7:7ed0d353:::benchmark_data_smithi003_80362_object29999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.813183+0000", 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 321.081225265, 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13364878899999999, 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151016 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813183+0000", 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813183+0000", 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813185+0000", 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.221e-06 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813191+0000", 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3080000000000004e-06 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813208+0000", 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6954999999999999e-05 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.813217+0000", 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1470000000000007e-06 2024-03-21T14:12:06.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.820048+0000", 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0068310599999999999 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.820197+0000", 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014832800000000001 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.820330+0000", 2024-03-21T14:12:06.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133487 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825236+0000", 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004905385 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.834728+0000", 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0094926720000000006 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.946306+0000", 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11157792 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.946617+0000", 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031110000000000003 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.946832+0000", 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021420599999999999 2024-03-21T14:12:06.937 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151020 7.e 7:7885fc55:::benchmark_data_smithi003_80362_object30003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:45.824394+0000", 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 321.07001396499999, 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.123109119, 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.938 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151020 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824394+0000", 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824394+0000", 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824396+0000", 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.728e-06 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824401+0000", 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5899999999999998e-06 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824418+0000", 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6558999999999998e-05 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.824427+0000", 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8149999999999995e-06 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825402+0000", 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097474200000000001 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.825755+0000", 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000353767 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.826716+0000", 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00096025099999999996 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.834886+0000", 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081701729999999993 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.838069+0000", 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031830299999999999 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.947044+0000", 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10897557400000001 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.947337+0000", 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029230899999999998 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:45.947503+0000", 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016658100000000001 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.943 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151160 7.e 7:7eca1350:::benchmark_data_smithi003_80362_object30143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:46.141625+0000", 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 320.75278256299998, 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.152556267, 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151160 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141625+0000", 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141625+0000", 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141627+0000", 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6169999999999999e-06 2024-03-21T14:12:06.945 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141633+0000", 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6799999999999998e-06 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141655+0000", 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2582000000000002e-05 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.141664+0000", 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4780000000000008e-06 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.142204+0000", 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054035599999999997 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.142502+0000", 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029758299999999997 2024-03-21T14:12:06.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.142874+0000", 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037233600000000002 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.147284+0000", 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044097900000000002 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.147380+0000", 2024-03-21T14:12:06.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6098e-05 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294083+0000", 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14670290699999999 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294137+0000", 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4388999999999997e-05 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294182+0000", 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4450999999999998e-05 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151179 7.e 7:7b27c9b6:::benchmark_data_smithi003_80362_object30162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:46.173411+0000", 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 320.72099714299998, 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120895954, 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.950 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151179 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173411+0000", 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173411+0000", 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173413+0000", 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0229999999999999e-06 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173420+0000", 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.34e-06 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173436+0000", 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6217000000000001e-05 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173448+0000", 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1602e-05 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.173637+0000", 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018898400000000001 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.174000+0000", 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00036340999999999999 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.174394+0000", 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000393419 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.178269+0000", 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038754890000000002 2024-03-21T14:12:06.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.178308+0000", 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8575999999999998e-05 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294221+0000", 2024-03-21T14:12:06.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115913008 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294271+0000", 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0358000000000001e-05 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.294307+0000", 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5528000000000003e-05 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152526 7.e 7:72655171:::benchmark_data_smithi003_80362_object31509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.324698+0000", 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 318.56970993099998, 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.134498811, 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.957 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152526 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324698+0000", 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324698+0000", 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324699+0000", 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.302e-06 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324705+0000", 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3739999999999997e-06 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324717+0000", 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.253e-05 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.324725+0000", 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1750000000000005e-06 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.325984+0000", 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012589019999999999 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.326314+0000", 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000329535 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.960 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.326428+0000", 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113921 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335437+0000", 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0090096570000000008 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335452+0000", 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4775e-05 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459046+0000", 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12359379199999999 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459144+0000", 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7990000000000002e-05 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459197+0000", 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2858000000000001e-05 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.962 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152530 7.e 7:79192372:::benchmark_data_smithi003_80362_object31513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.329026+0000", 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 318.56538164300002, 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13036171099999999, 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152530 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.963 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329026+0000", 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329026+0000", 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329028+0000", 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5629999999999999e-06 2024-03-21T14:12:06.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329034+0000", 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.835e-06 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329051+0000", 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.685e-05 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.329064+0000", 2024-03-21T14:12:06.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3219e-05 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.330419+0000", 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013551469999999999 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.330719+0000", 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030028800000000001 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.331228+0000", 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00050899500000000004 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335489+0000", 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042610110000000003 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335500+0000", 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0553e-05 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459255+0000", 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.123755485 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459345+0000", 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9462000000000005e-05 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459388+0000", 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3303000000000002e-05 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152536 7.e 7:74894159:::benchmark_data_smithi003_80362_object31519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.334871+0000", 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 318.55953717099999, 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.124694136, 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152536 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334871+0000", 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334871+0000", 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334873+0000", 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.784e-06 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334877+0000", 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1439999999999996e-06 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334888+0000", 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1365e-05 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.334893+0000", 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2800000000000003e-06 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335627+0000", 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00073397600000000001 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.335925+0000", 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029732000000000001 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.336298+0000", 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037340700000000002 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.343414+0000", 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071161480000000001 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.343433+0000", 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9108999999999999e-05 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459435+0000", 2024-03-21T14:12:06.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.116001458 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459523+0000", 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8332999999999996e-05 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459565+0000", 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1811999999999999e-05 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.974 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:152538 7.e 7:77c3ed5e:::benchmark_data_smithi003_80362_object31521:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:48.338885+0000", 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 318.555522795, 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120848841, 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 152538 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.975 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338885+0000", 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338885+0000", 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338886+0000", 2024-03-21T14:12:06.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7100000000000001e-07 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338890+0000", 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.794e-06 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338902+0000", 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2140000000000001e-05 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.338908+0000", 2024-03-21T14:12:06.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1929999999999998e-06 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.339836+0000", 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092747499999999998 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.340014+0000", 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017830100000000001 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.340117+0000", 2024-03-21T14:12:06.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102717 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.344340+0000", 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042231459999999997 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.344355+0000", 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5058000000000001e-05 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459610+0000", 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115255682 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459694+0000", 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3944999999999997e-05 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:48.459734+0000", 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9718999999999997e-05 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153387 7.e 7:7b4816c7:::benchmark_data_smithi003_80362_object32370:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.627694+0000", 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 317.26671356499998, 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13285661700000001, 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153387 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627694+0000", 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627694+0000", 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627696+0000", 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.683e-06 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627704+0000", 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1329999999999999e-06 2024-03-21T14:12:06.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627719+0000", 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4761e-05 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627728+0000", 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1719999999999996e-06 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627810+0000", 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2275999999999996e-05 2024-03-21T14:12:06.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.627946+0000", 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135752 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.628070+0000", 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123917 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.632122+0000", 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040513629999999997 2024-03-21T14:12:06.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.633010+0000", 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00088858399999999999 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.760389+0000", 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12737904899999999 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.760499+0000", 2024-03-21T14:12:06.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010972999999999999 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.760551+0000", 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2197000000000002e-05 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153393 7.e 7:7f50553e:::benchmark_data_smithi003_80362_object32376:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.640759+0000", 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 317.25364942200002, 2024-03-21T14:12:06.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.126200215, 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153393 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640759+0000", 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640759+0000", 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640760+0000", 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5099999999999999e-06 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640766+0000", 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.631e-06 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640776+0000", 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.029e-05 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.640783+0000", 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6590000000000001e-06 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.641111+0000", 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00032799499999999998 2024-03-21T14:12:06.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.641231+0000", 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012035 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.641346+0000", 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011500100000000001 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.646310+0000", 2024-03-21T14:12:06.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049644310000000001 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.648887+0000", 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025767889999999999 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.766776+0000", 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.117888702 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:06.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.766908+0000", 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000131662 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.766959+0000", 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1195e-05 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153700 7.e 7:767bfc88:::benchmark_data_smithi003_80362_object32683:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.113168+0000", 2024-03-21T14:12:06.993 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 316.78124047, 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15167361400000001, 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153700 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113168+0000", 2024-03-21T14:12:06.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113168+0000", 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113169+0000", 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.562e-06 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113174+0000", 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8590000000000004e-06 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113192+0000", 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7941999999999999e-05 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:06.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.113200+0000", 2024-03-21T14:12:06.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6260000000000003e-06 2024-03-21T14:12:06.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:06.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.114412+0000", 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001212791 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.114526+0000", 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113539 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.114645+0000", 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119078 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.118486+0000", 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038406989999999999 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.118575+0000", 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.954e-05 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.264731+0000", 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.146156233 2024-03-21T14:12:06.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.264794+0000", 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2465999999999995e-05 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.264841+0000", 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7278999999999999e-05 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154042 7.e 7:7e3a347b:::benchmark_data_smithi003_80362_object33025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.643263+0000", 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 316.25114534300002, 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16783620799999999, 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154042 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643263+0000", 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643263+0000", 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643264+0000", 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.463e-06 2024-03-21T14:12:07.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643269+0000", 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2900000000000002e-06 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643289+0000", 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9221000000000001e-05 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643297+0000", 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.909e-06 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643350+0000", 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3458999999999998e-05 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643459+0000", 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108498 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.643569+0000", 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110876 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.648285+0000", 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047160789999999998 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.648366+0000", 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0090000000000001e-05 2024-03-21T14:12:07.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.810900+0000", 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.162534278 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811043+0000", 2024-03-21T14:12:07.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014269 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811099+0000", 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6354999999999998e-05 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154058 7.e 7:733d882b:::benchmark_data_smithi003_80362_object33041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 316.22937837900002, 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.146220038, 2024-03-21T14:12:07.007 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154058 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665030+0000", 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.23e-07 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665034+0000", 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.109e-06 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665043+0000", 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4710000000000007e-06 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665049+0000", 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5480000000000001e-06 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665097+0000", 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8627e-05 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665183+0000", 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6013000000000005e-05 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.665254+0000", 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0959999999999998e-05 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.667703+0000", 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024487559999999998 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.668694+0000", 2024-03-21T14:12:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00099065200000000007 2024-03-21T14:12:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811155+0000", 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14246165799999999 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811207+0000", 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2009999999999998e-05 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:07.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.811250+0000", 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2511e-05 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154760 7.e 7:7bbbc488:::benchmark_data_smithi003_80362_object33743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:51.850395+0000", 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 315.04401305300001, 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11783648200000001, 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:07.015 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154760 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850395+0000", 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850395+0000", 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850397+0000", 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7129999999999999e-06 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850403+0000", 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4300000000000003e-06 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850419+0000", 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5654999999999998e-05 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850428+0000", 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1800000000000002e-06 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850493+0000", 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4937999999999994e-05 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850622+0000", 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128801 2024-03-21T14:12:07.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.850703+0000", 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1545999999999995e-05 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.854287+0000", 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035834679999999998 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.858521+0000", 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042342430000000004 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.967804+0000", 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.109283279 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.968057+0000", 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025259799999999998 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:51.968231+0000", 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017463099999999999 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.020 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:176341 7.e 7:72b7b1c7:::benchmark_data_smithi003_80362_object55324:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:07:13.608919+0000", 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 293.28548912799999, 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.118495692, 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 176341 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.021 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608919+0000", 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608919+0000", 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608921+0000", 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.61e-06 2024-03-21T14:12:07.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608933+0000", 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.225e-05 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608947+0000", 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3934000000000001e-05 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.608956+0000", 2024-03-21T14:12:07.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8210000000000008e-06 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.626731+0000", 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017775076000000001 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.626844+0000", 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000112972 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:12:07.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.626962+0000", 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0001187 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.677437+0000", 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050474473999999998 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.681328+0000", 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038916240000000002 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.727293+0000", 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045964694 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.727368+0000", 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4567999999999995e-05 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:07:13.727415+0000", 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6968999999999997e-05 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.026 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:07.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:07.027 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:07.027 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:12:07.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:06 smithi003 ceph-mon[29576]: pgmap v405: 158 pgs: 1 active+remapped, 1 peering, 5 remapped+peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 834 B/s rd, 0 op/s; 17531/152862 objects misplaced (11.469%); 7.9 MiB/s, 139 objects/s recovering 2024-03-21T14:12:07.040 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:06 smithi003 ceph-mon[29576]: 8.4 scrub ok 2024-03-21T14:12:07.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:06 smithi003 ceph-mon[40510]: pgmap v405: 158 pgs: 1 active+remapped, 1 peering, 5 remapped+peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 834 B/s rd, 0 op/s; 17531/152862 objects misplaced (11.469%); 7.9 MiB/s, 139 objects/s recovering 2024-03-21T14:12:07.040 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:06 smithi003 ceph-mon[40510]: 8.4 scrub ok 2024-03-21T14:12:07.046 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:07.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:07.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:07.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:06 smithi118 ceph-mon[33862]: pgmap v405: 158 pgs: 1 active+remapped, 1 peering, 5 remapped+peering, 2 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 144 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 834 B/s rd, 0 op/s; 17531/152862 objects misplaced (11.469%); 7.9 MiB/s, 139 objects/s recovering 2024-03-21T14:12:07.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:06 smithi118 ceph-mon[33862]: 8.4 scrub ok 2024-03-21T14:12:07.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:12:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:12:07.187 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:07.289 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:07.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:07.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:07.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:07.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:07.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.627 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:07.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:07.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:07.760 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:07.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:07.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:07.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:07.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:08.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:08.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:08.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:08.172 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:08.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:08.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.309 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:08.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:08.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:08.438 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:08.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:12:08.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:08.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:08.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:08.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:08.717 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:08.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:08.843 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:08.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:08.971 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:08.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:08.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:09.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:09.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:09.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:09.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[29576]: pgmap v406: 158 pgs: 1 active+remapped, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 17658/136338 objects misplaced (12.952%); 42 MiB/s, 679 objects/s recovering 2024-03-21T14:12:09.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:12:09.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:12:09.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[29576]: osdmap e215: 8 total, 8 up, 8 in 2024-03-21T14:12:09.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:12:09.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:12:09.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[40510]: pgmap v406: 158 pgs: 1 active+remapped, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 17658/136338 objects misplaced (12.952%); 42 MiB/s, 679 objects/s recovering 2024-03-21T14:12:09.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:12:09.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:12:09.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[40510]: osdmap e215: 8 total, 8 up, 8 in 2024-03-21T14:12:09.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:12:09.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:08 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:12:09.238 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:09.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:09.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:09.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:08 smithi118 ceph-mon[33862]: pgmap v406: 158 pgs: 1 active+remapped, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 17658/136338 objects misplaced (12.952%); 42 MiB/s, 679 objects/s recovering 2024-03-21T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:08 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:08 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:08 smithi118 ceph-mon[33862]: osdmap e215: 8 total, 8 up, 8 in 2024-03-21T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:08 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:12:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:08 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:12:09.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:09.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:09.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:09.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:09.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:09.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:09.926 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:09.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:10.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.053 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:10.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:10.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:10.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:12:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:12:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:12:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: osdmap e216: 8 total, 8 up, 8 in 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[29576]: osdmap e217: 8 total, 8 up, 8 in 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:12:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: osdmap e216: 8 total, 8 up, 8 in 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:10.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:09 smithi003 ceph-mon[40510]: osdmap e217: 8 total, 8 up, 8 in 2024-03-21T14:12:10.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:10.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:10.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:10.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: osdmap e216: 8 total, 8 up, 8 in 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:12:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:12:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:12:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:12:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:09 smithi118 ceph-mon[33862]: osdmap e217: 8 total, 8 up, 8 in 2024-03-21T14:12:10.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:10.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:10.534 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:10.568 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:10.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:10.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:10.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.834 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:10.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:10.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:10.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:10.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:11.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T14:12:11.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:11.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:11.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: pgmap v409: 158 pgs: 1 active+remapped, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 17658/136338 objects misplaced (12.952%); 41 MiB/s, 653 objects/s recovering 2024-03-21T14:12:11.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:12:11.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:12:11.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:12:11.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 8.7 scrub starts 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 8.18 scrub starts 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: osdmap e218: 8 total, 8 up, 8 in 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[29576]: 8.18 scrub ok 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: pgmap v409: 158 pgs: 1 active+remapped, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 17658/136338 objects misplaced (12.952%); 41 MiB/s, 653 objects/s recovering 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 8.7 scrub starts 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 8.18 scrub starts 2024-03-21T14:12:11.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: osdmap e218: 8 total, 8 up, 8 in 2024-03-21T14:12:11.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:10 smithi003 ceph-mon[40510]: 8.18 scrub ok 2024-03-21T14:12:11.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:11.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: pgmap v409: 158 pgs: 1 active+remapped, 5 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 146 active+clean; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 17658/136338 objects misplaced (12.952%); 41 MiB/s, 653 objects/s recovering 2024-03-21T14:12:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:12:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:12:11.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:12:11.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:12:11.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 8.7 scrub starts 2024-03-21T14:12:11.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 8.18 scrub starts 2024-03-21T14:12:11.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: osdmap e218: 8 total, 8 up, 8 in 2024-03-21T14:12:11.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:10 smithi118 ceph-mon[33862]: 8.18 scrub ok 2024-03-21T14:12:11.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:11.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:11.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:11.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:11.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:11.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:11.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:12.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:12.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:12.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:12.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 8.7 scrub ok 2024-03-21T14:12:12.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:12:12.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:12:12.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:12:12.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[29576]: osdmap e219: 8 total, 8 up, 8 in 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 8.7 scrub ok 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:12:12.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:11 smithi003 ceph-mon[40510]: osdmap e219: 8 total, 8 up, 8 in 2024-03-21T14:12:12.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:12.308 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:12.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:12.344 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:12.409 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:12.426 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 8.7 scrub ok 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:12:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:11 smithi118 ceph-mon[33862]: osdmap e219: 8 total, 8 up, 8 in 2024-03-21T14:12:12.443 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:12.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:12.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:12.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:12.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:12.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:12.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:12.864 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:12.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:12.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:13.002 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:12:13.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:13.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:13.144 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:13.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[29576]: pgmap v412: 158 pgs: 1 active+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 149 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15686/133284 objects misplaced (11.769%) 2024-03-21T14:12:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:12:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:12:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/661984157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:12:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:12:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[40510]: pgmap v412: 158 pgs: 1 active+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 149 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15686/133284 objects misplaced (11.769%) 2024-03-21T14:12:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:12:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:12:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/661984157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:12:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:12 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:12:13.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:13.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:13.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:12 smithi118 ceph-mon[33862]: pgmap v412: 158 pgs: 1 active+remapped, 2 active+remapped+backfilling, 6 active+remapped+backfill_toofull, 149 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 15686/133284 objects misplaced (11.769%) 2024-03-21T14:12:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:12 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:12:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:12 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:12:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/661984157' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:12:13.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:12 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:12:13.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:13.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:13.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:13.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:13.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:13.828 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:13.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:13.930 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:13.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:13.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:14.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:14.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:14.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:14.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: 8.3 deep-scrub starts 2024-03-21T14:12:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:12:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:12:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:12:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: osdmap e220: 8 total, 8 up, 8 in 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: 8.3 deep-scrub starts 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: osdmap e220: 8 total, 8 up, 8 in 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:13 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:14.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:14.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: 8.3 deep-scrub starts 2024-03-21T14:12:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:12:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:12:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:12:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: osdmap e220: 8 total, 8 up, 8 in 2024-03-21T14:12:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:14.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:13 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:14.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:14.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:14.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:14.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:14.754 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:14.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:14.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:14.891 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:14.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:14.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:14.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:15.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:15.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:15.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:15.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:15 smithi003 ceph-mon[29576]: pgmap v415: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13263/126186 objects misplaced (10.511%) 2024-03-21T14:12:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:15 smithi003 ceph-mon[29576]: osdmap e221: 8 total, 8 up, 8 in 2024-03-21T14:12:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:15 smithi003 ceph-mon[29576]: osdmap e222: 8 total, 8 up, 8 in 2024-03-21T14:12:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:15 smithi003 ceph-mon[40510]: pgmap v415: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13263/126186 objects misplaced (10.511%) 2024-03-21T14:12:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:15 smithi003 ceph-mon[40510]: osdmap e221: 8 total, 8 up, 8 in 2024-03-21T14:12:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:15 smithi003 ceph-mon[40510]: osdmap e222: 8 total, 8 up, 8 in 2024-03-21T14:12:15.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.319 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:15.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:15.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:15 smithi118 ceph-mon[33862]: pgmap v415: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13263/126186 objects misplaced (10.511%) 2024-03-21T14:12:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:15 smithi118 ceph-mon[33862]: osdmap e221: 8 total, 8 up, 8 in 2024-03-21T14:12:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:15 smithi118 ceph-mon[33862]: osdmap e222: 8 total, 8 up, 8 in 2024-03-21T14:12:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:15.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:15.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.593 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:15.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:15.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:15.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:15.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:15.864 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:15.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:15.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:15.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:16.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:16.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:16.133 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:16.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:16.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:16.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:16.398 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:16 smithi118 ceph-mon[33862]: pgmap v418: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 13263/126186 objects misplaced (10.511%); 3.2 MiB/s, 50 objects/s recovering 2024-03-21T14:12:16.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:16 smithi003 ceph-mon[29576]: pgmap v418: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 13263/126186 objects misplaced (10.511%); 3.2 MiB/s, 50 objects/s recovering 2024-03-21T14:12:16.492 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:16 smithi003 ceph-mon[40510]: pgmap v418: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing, 6 active+remapped+backfill_toofull, 150 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 13263/126186 objects misplaced (10.511%); 3.2 MiB/s, 50 objects/s recovering 2024-03-21T14:12:16.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:16.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.533 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:16.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:16.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:16.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:12:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:12:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:12:16.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:16.812 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:16.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:16.939 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:16.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:16.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:17.041 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:17.070 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:17.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:17.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:12:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:12:17.197 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:17.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:17.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:17.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:17.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:17.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:17.334 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:17.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:17.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:17.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:17.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:17.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:17.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.767 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:17.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:17.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:17.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:17.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:17.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:18.010 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:18.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:18.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:18.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:18.283 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:18.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.319 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:18.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:18.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:12:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:18 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:18 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:18 smithi118 ceph-mon[33862]: pgmap v419: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 866 B/s rd, 0 op/s; 11632/110907 objects misplaced (10.488%); 2.1 MiB/s, 34 objects/s recovering 2024-03-21T14:12:18.453 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:18.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:18.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:12:18.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:18.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:18.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[29576]: pgmap v419: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 866 B/s rd, 0 op/s; 11632/110907 objects misplaced (10.488%); 2.1 MiB/s, 34 objects/s recovering 2024-03-21T14:12:18.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:12:18.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:18.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:18.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:18 smithi003 ceph-mon[40510]: pgmap v419: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 866 B/s rd, 0 op/s; 11632/110907 objects misplaced (10.488%); 2.1 MiB/s, 34 objects/s recovering 2024-03-21T14:12:18.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.590 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:18.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:18.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:18.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:18.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:18.870 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:18.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:18.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:19.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:19.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:19.150 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:19.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:19 smithi003 ceph-mon[29576]: osdmap e223: 8 total, 8 up, 8 in 2024-03-21T14:12:19.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:19 smithi003 ceph-mon[40510]: osdmap e223: 8 total, 8 up, 8 in 2024-03-21T14:12:19.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:19.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:19.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:19 smithi118 ceph-mon[33862]: osdmap e223: 8 total, 8 up, 8 in 2024-03-21T14:12:19.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.550 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:19.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:19.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:19.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:19.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:19.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:19.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:12:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:19.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:19.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:19.930 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:19.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.0 primary-affinity to 0 (802) 2024-03-21T14:12:20.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:20.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:20.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3091272882' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-21T14:12:20.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-21T14:12:20.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[40510]: pgmap v421: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 849 op/s; 11632/110907 objects misplaced (10.488%) 2024-03-21T14:12:20.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3091272882' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-21T14:12:20.098 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-21T14:12:20.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:20.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:20.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:20.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:20.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3091272882' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-21T14:12:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:20 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]: dispatch 2024-03-21T14:12:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:20 smithi118 ceph-mon[33862]: pgmap v421: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 849 op/s; 11632/110907 objects misplaced (10.488%) 2024-03-21T14:12:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:20 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-03-21T14:12:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:20 smithi118 ceph-mon[33862]: osdmap e224: 8 total, 8 up, 8 in 2024-03-21T14:12:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:20.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[29576]: pgmap v421: 158 pgs: 1 active+remapped, 1 active+clean+scrubbing+deep, 6 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 849 op/s; 11632/110907 objects misplaced (10.488%) 2024-03-21T14:12:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-03-21T14:12:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[29576]: osdmap e224: 8 total, 8 up, 8 in 2024-03-21T14:12:20.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 0.0}]': finished 2024-03-21T14:12:20.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:20 smithi003 ceph-mon[40510]: osdmap e224: 8 total, 8 up, 8 in 2024-03-21T14:12:20.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:20.562 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:20.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:20.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.698 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:20.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:20.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:20.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:20.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:20.961 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:20.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:20.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:21.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:21.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.088 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:21.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:21.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:21.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:21.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:21.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:21.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:21 smithi003 ceph-mon[29576]: 8.3 deep-scrub ok 2024-03-21T14:12:21.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:21 smithi003 ceph-mon[29576]: osdmap e225: 8 total, 8 up, 8 in 2024-03-21T14:12:21.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:21 smithi003 ceph-mon[40510]: 8.3 deep-scrub ok 2024-03-21T14:12:21.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:21 smithi003 ceph-mon[40510]: osdmap e225: 8 total, 8 up, 8 in 2024-03-21T14:12:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:21 smithi118 ceph-mon[33862]: 8.3 deep-scrub ok 2024-03-21T14:12:21.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:21 smithi118 ceph-mon[33862]: osdmap e225: 8 total, 8 up, 8 in 2024-03-21T14:12:21.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:21.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:21.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:21.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:21.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:21.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:21.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:21.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:21.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:21.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:22.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:22.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:22.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:22.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:22.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:22.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:22.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.437 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:22.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:22.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:22.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:22.679 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:22.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.710 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:22.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:22.811 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:22.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: pgmap v424: 158 pgs: 1 remapped+peering, 1 active+remapped, 1 active+clean+scrubbing+deep, 5 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.06k op/s; 10202/107127 objects misplaced (9.523%) 2024-03-21T14:12:22.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: osdmap e226: 8 total, 8 up, 8 in 2024-03-21T14:12:22.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: 8.6 continuing backfill to osd.2 from (168'2679,223'12421] MIN to 223'12421 2024-03-21T14:12:22.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:22.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:12:22.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:12:22.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:12:22.823 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: pgmap v424: 158 pgs: 1 remapped+peering, 1 active+remapped, 1 active+clean+scrubbing+deep, 5 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.06k op/s; 10202/107127 objects misplaced (9.523%) 2024-03-21T14:12:22.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: osdmap e226: 8 total, 8 up, 8 in 2024-03-21T14:12:22.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: 8.6 continuing backfill to osd.2 from (168'2679,223'12421] MIN to 223'12421 2024-03-21T14:12:22.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:22.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:12:22.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:12:22.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:22 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:12:22.842 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: pgmap v424: 158 pgs: 1 remapped+peering, 1 active+remapped, 1 active+clean+scrubbing+deep, 5 active+remapped+backfill_toofull, 150 active+clean; 2.2 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.06k op/s; 10202/107127 objects misplaced (9.523%) 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: osdmap e226: 8 total, 8 up, 8 in 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: 8.6 continuing backfill to osd.2 from (168'2679,223'12421] MIN to 223'12421 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:12:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:22 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:12:22.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:22.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:22.976 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:22.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:23.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:23.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:23.248 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:23.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:23.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:23.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:23.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:23.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:23.520 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:23.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:23.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:23.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:23.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:23.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:23.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:24.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:24.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:24.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:24.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:24.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:24.362 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:12:24.362 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:12:24.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.362 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:232086 8.0 8:07953347:::benchmark_data_smithi003_110242_object101169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:24.362206+0000", 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00016087799999999999, 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019587700000000001, 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 232086 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362206+0000", 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362206+0000", 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362208+0000", 2024-03-21T14:12:24.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.968e-06 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362215+0000", 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2250000000000003e-06 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362232+0000", 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6560999999999999e-05 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:24.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362241+0000", 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6500000000000008e-06 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.362298+0000", 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6613999999999997e-05 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:232077 8.f 8:ff3d1373:::benchmark_data_smithi003_110242_object101160:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:24.348950+0000", 2024-03-21T14:12:24.366 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.013417178, 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013543395999999999, 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 232077 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:24.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.348950+0000", 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.348950+0000", 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:24.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.348951+0000", 2024-03-21T14:12:24.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5200000000000001e-06 2024-03-21T14:12:24.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.348957+0000", 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5829999999999997e-06 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.348973+0000", 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6116999999999998e-05 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.348981+0000", 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1589999999999993e-06 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.349066+0000", 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4765999999999994e-05 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.349181+0000", 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115612 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.349455+0000", 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000273615 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:24.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:24.358025+0000", 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0085701270000000003 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 2 2024-03-21T14:12:24.372 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:12:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.373 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:24.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:24.384 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:12:24.384 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:12:24.401 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:12:24.445 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:12:24.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:24.497 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:12:24.504 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:24.541 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:12:24.586 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:12:24.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:24.626 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:12:24.639 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:12:24.639 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:12:24.639 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:12:24.639 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:12:24.639 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:12:24.640 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:12:24.640 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:12:24.640 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:12:24.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:24.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:24 smithi003 ceph-mon[29576]: pgmap v426: 158 pgs: 1 active+remapped, 1 remapped+peering, 5 active+remapped+backfill_toofull, 151 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 379 B/s rd, 0 op/s; 9442/99603 objects misplaced (9.480%) 2024-03-21T14:12:24.657 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:24 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1949722474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:24.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:24 smithi003 ceph-mon[40510]: pgmap v426: 158 pgs: 1 active+remapped, 1 remapped+peering, 5 active+remapped+backfill_toofull, 151 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 379 B/s rd, 0 op/s; 9442/99603 objects misplaced (9.480%) 2024-03-21T14:12:24.657 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:24 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1949722474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:24.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:24.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:24.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:24 smithi118 ceph-mon[33862]: pgmap v426: 158 pgs: 1 active+remapped, 1 remapped+peering, 5 active+remapped+backfill_toofull, 151 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 379 B/s rd, 0 op/s; 9442/99603 objects misplaced (9.480%) 2024-03-21T14:12:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:24 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1949722474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:24.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:24.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:25.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:25.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:25.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:25.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:25.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:25.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:25.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.510 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:25.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:25.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:25.643 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:25.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:25.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:25.776 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:25.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:25.917 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:25.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:26.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.042 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:26.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:26.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:26.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:26.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:26.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:26.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:26.388 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:26.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:26.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T14:12:26.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:26.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:26.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:26.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:26.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:26 smithi003 ceph-mon[29576]: pgmap v427: 158 pgs: 1 active+remapped, 1 remapped+peering, 5 active+remapped+backfill_toofull, 151 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 9442/99603 objects misplaced (9.480%) 2024-03-21T14:12:26.717 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:12:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:12:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:12:26.717 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:26 smithi003 ceph-mon[40510]: pgmap v427: 158 pgs: 1 active+remapped, 1 remapped+peering, 5 active+remapped+backfill_toofull, 151 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 9442/99603 objects misplaced (9.480%) 2024-03-21T14:12:26.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.768 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:26.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:26.812 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:26 smithi118 ceph-mon[33862]: pgmap v427: 158 pgs: 1 active+remapped, 1 remapped+peering, 5 active+remapped+backfill_toofull, 151 active+clean; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 9442/99603 objects misplaced (9.480%) 2024-03-21T14:12:26.869 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:26.901 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:26.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:26.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:27.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:27.037 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:27.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:27.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:27.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:27.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:12:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:12:27.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:27.323 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:27.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:27.461 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:27.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:12:27.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:27.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:27.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:27.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:27.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:27.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:27.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:27.915 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:12:27.915 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:12:27.916 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-03-21T14:12:27.916 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.0 'trim stale osdmaps' 2024-03-21T14:12:27.917 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:27.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:27.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:28.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:28.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:28.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:28.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:28.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:28.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:28.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:28.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:28.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:28.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:28.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:28.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:28.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:28.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:28.796 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:28.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:28.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:28.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:28 smithi118 ceph-mon[33862]: pgmap v428: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 4 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 906 B/s rd, 0 B/s wr, 326 op/s; 7728/82149 objects misplaced (9.407%) 2024-03-21T14:12:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:28 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:28.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:28 smithi003 ceph-mon[29576]: pgmap v428: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 4 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 906 B/s rd, 0 B/s wr, 326 op/s; 7728/82149 objects misplaced (9.407%) 2024-03-21T14:12:28.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:28 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:28.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:28 smithi003 ceph-mon[40510]: pgmap v428: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 4 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 906 B/s rd, 0 B/s wr, 326 op/s; 7728/82149 objects misplaced (9.407%) 2024-03-21T14:12:28.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:28 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:28.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:29.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:29.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:29.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:29.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:29.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:29.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:29.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:29.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:29.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:29.540 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:12:29.546 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:29.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:29.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:29.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:29.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3961822530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[29576]: osdmap e227: 8 total, 8 up, 8 in 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3961822530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:29.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:29.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:29 smithi003 ceph-mon[40510]: osdmap e227: 8 total, 8 up, 8 in 2024-03-21T14:12:29.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.840 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:29.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3961822530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:29.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:29 smithi118 ceph-mon[33862]: osdmap e227: 8 total, 8 up, 8 in 2024-03-21T14:12:29.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:29.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:29.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:29.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:30.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:30.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:30.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:30.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:30.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.360 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:30.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:30.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:30.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:30.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:30.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:30.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:30 smithi003 ceph-mon[29576]: pgmap v429: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 4 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 289 op/s; 7728/82149 objects misplaced (9.407%) 2024-03-21T14:12:30.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:30 smithi003 ceph-mon[29576]: osdmap e228: 8 total, 8 up, 8 in 2024-03-21T14:12:30.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:30 smithi003 ceph-mon[40510]: pgmap v429: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 4 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 289 op/s; 7728/82149 objects misplaced (9.407%) 2024-03-21T14:12:30.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:30 smithi003 ceph-mon[40510]: osdmap e228: 8 total, 8 up, 8 in 2024-03-21T14:12:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.762 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:30.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:30.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:30.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:30.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:30 smithi118 ceph-mon[33862]: pgmap v429: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 4 active+remapped+backfill_toofull, 151 active+clean; 1.7 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 803 B/s rd, 0 B/s wr, 289 op/s; 7728/82149 objects misplaced (9.407%) 2024-03-21T14:12:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:30 smithi118 ceph-mon[33862]: osdmap e228: 8 total, 8 up, 8 in 2024-03-21T14:12:30.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:31.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:31.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:31.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:31.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:31.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:31.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.407 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:31.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:31.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:31.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:31.635 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:31.670 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:31.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:31.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:31 smithi003 ceph-mon[29576]: osdmap e229: 8 total, 8 up, 8 in 2024-03-21T14:12:31.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:31.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:31 smithi003 ceph-mon[40510]: osdmap e229: 8 total, 8 up, 8 in 2024-03-21T14:12:31.772 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.805 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:31.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:31.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:31.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:31.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:31 smithi118 ceph-mon[33862]: osdmap e229: 8 total, 8 up, 8 in 2024-03-21T14:12:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:31.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:31.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:32.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.088 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:32.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:32.132 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:12:32.133 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:12:32.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.228 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:32.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:32.329 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:32.365 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:32.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:32.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:32.500 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:32.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:32.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:32.638 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:32.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:32.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:32.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:32.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:32.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:32.919 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:32.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:32 smithi118 ceph-mon[33862]: pgmap v432: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 3 active+remapped+backfill_toofull, 152 active+clean; 1.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 911 op/s; 6508/77748 objects misplaced (8.371%); 12 MiB/s, 191 objects/s recovering 2024-03-21T14:12:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:32 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:32 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:32.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:32 smithi003 ceph-mon[40510]: pgmap v432: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 3 active+remapped+backfill_toofull, 152 active+clean; 1.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 911 op/s; 6508/77748 objects misplaced (8.371%); 12 MiB/s, 191 objects/s recovering 2024-03-21T14:12:32.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:32 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:32.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:32 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:32.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:32 smithi003 ceph-mon[29576]: pgmap v432: 158 pgs: 2 active+remapped+backfilling, 1 active+remapped, 3 active+remapped+backfill_toofull, 152 active+clean; 1.6 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 911 op/s; 6508/77748 objects misplaced (8.371%); 12 MiB/s, 191 objects/s recovering 2024-03-21T14:12:32.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:32 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:32.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:32 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:33.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:33.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:33.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:33.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:33.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.337 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:33.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:33.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:33.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:33.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T14:12:33.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:12:33.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:33.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.607 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:33.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:33.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:33.739 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:33.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:33 smithi003 ceph-mon[40510]: osdmap e230: 8 total, 8 up, 8 in 2024-03-21T14:12:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:33 smithi003 ceph-mon[29576]: osdmap e230: 8 total, 8 up, 8 in 2024-03-21T14:12:33.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:33.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:33.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:33.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:33 smithi118 ceph-mon[33862]: osdmap e230: 8 total, 8 up, 8 in 2024-03-21T14:12:33.990 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:34.022 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:34.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:34.156 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:34.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:34.289 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:34.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:34.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:34.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:34.517 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.545 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:34.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:34.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:34.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:34.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:34.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:34.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:34.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[29576]: pgmap v435: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 4812/68361 objects misplaced (7.039%); 20 MiB/s, 315 objects/s recovering 2024-03-21T14:12:34.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:34.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[29576]: osdmap e231: 8 total, 8 up, 8 in 2024-03-21T14:12:34.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[29576]: osdmap e232: 8 total, 8 up, 8 in 2024-03-21T14:12:34.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[40510]: pgmap v435: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 4812/68361 objects misplaced (7.039%); 20 MiB/s, 315 objects/s recovering 2024-03-21T14:12:34.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:34.838 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[40510]: osdmap e231: 8 total, 8 up, 8 in 2024-03-21T14:12:34.838 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:34 smithi003 ceph-mon[40510]: osdmap e232: 8 total, 8 up, 8 in 2024-03-21T14:12:34.838 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:12:34.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:12:34.927 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:12:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:34 smithi118 ceph-mon[33862]: pgmap v435: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s; 4812/68361 objects misplaced (7.039%); 20 MiB/s, 315 objects/s recovering 2024-03-21T14:12:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:34 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:34 smithi118 ceph-mon[33862]: osdmap e231: 8 total, 8 up, 8 in 2024-03-21T14:12:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:34 smithi118 ceph-mon[33862]: osdmap e232: 8 total, 8 up, 8 in 2024-03-21T14:12:34.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:34.973 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:12:34.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:34.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:35.014 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:12:35.060 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:12:35.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:35.099 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:12:35.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:35.139 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:12:35.147 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:12:35.147 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:12:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:12:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:12:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:12:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:12:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:12:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:12:35.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:35.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:35.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:35.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:35.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:35.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:35.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:35.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:35.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:35.713 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:35.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/449104490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:35.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:35 smithi003 ceph-mon[29576]: osdmap e233: 8 total, 8 up, 8 in 2024-03-21T14:12:35.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/449104490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:35.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:35 smithi003 ceph-mon[40510]: osdmap e233: 8 total, 8 up, 8 in 2024-03-21T14:12:35.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:35.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:35.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/449104490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:35 smithi118 ceph-mon[33862]: osdmap e233: 8 total, 8 up, 8 in 2024-03-21T14:12:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:35.946 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:35.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:36.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:36.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:36.116 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:36.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.200 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:36.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:36.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:36.327 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:36.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:36.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:36.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:36.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:36.583 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:36.636 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:12:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:12:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:12:36.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:36.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.712 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:36.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:36.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:36.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:36.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:36 smithi118 ceph-mon[33862]: pgmap v438: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 4812/68361 objects misplaced (7.039%); 7.0 MiB/s, 112 objects/s recovering 2024-03-21T14:12:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:36 smithi118 ceph-mon[33862]: osdmap e234: 8 total, 8 up, 8 in 2024-03-21T14:12:36.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:12:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:12:36.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:36.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:36.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:36 smithi003 ceph-mon[29576]: pgmap v438: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 4812/68361 objects misplaced (7.039%); 7.0 MiB/s, 112 objects/s recovering 2024-03-21T14:12:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:36 smithi003 ceph-mon[29576]: osdmap e234: 8 total, 8 up, 8 in 2024-03-21T14:12:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:36 smithi003 ceph-mon[40510]: pgmap v438: 158 pgs: 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 153 active+clean; 1.4 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 0 B/s wr, 781 op/s; 4812/68361 objects misplaced (7.039%); 7.0 MiB/s, 112 objects/s recovering 2024-03-21T14:12:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:36 smithi003 ceph-mon[40510]: osdmap e234: 8 total, 8 up, 8 in 2024-03-21T14:12:37.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:37.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:37.204 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:37.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:37.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:37.381 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:37.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:37.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:37.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:37.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:37.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.663 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:37.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:37 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:37.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:37.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:37.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:37 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:37 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:37 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:37 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:37.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:37.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:38.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:38.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:38.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:12:38.179 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:38.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:38.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:38.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:38.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.336 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:38.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:38.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:38.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:38.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:38.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:38.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:38.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:38.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.741 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:38.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:38.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:38 smithi003 ceph-mon[29576]: pgmap v441: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 2786/53361 objects misplaced (5.221%); 14 MiB/s, 230 objects/s recovering 2024-03-21T14:12:38.831 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:38 smithi003 ceph-mon[29576]: osdmap e235: 8 total, 8 up, 8 in 2024-03-21T14:12:38.831 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:38 smithi003 ceph-mon[40510]: pgmap v441: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 2786/53361 objects misplaced (5.221%); 14 MiB/s, 230 objects/s recovering 2024-03-21T14:12:38.831 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:38 smithi003 ceph-mon[40510]: osdmap e235: 8 total, 8 up, 8 in 2024-03-21T14:12:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:12:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:12:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:12:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:12:38.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:38.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:38.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:38 smithi118 ceph-mon[33862]: pgmap v441: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 2786/53361 objects misplaced (5.221%); 14 MiB/s, 230 objects/s recovering 2024-03-21T14:12:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:38 smithi118 ceph-mon[33862]: osdmap e235: 8 total, 8 up, 8 in 2024-03-21T14:12:38.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:39.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:39.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:39.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:39.261 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:39.293 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:39.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:39.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:39.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:39.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:39.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:39.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:39.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:39.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:39.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:39.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:39 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:39 smithi118 ceph-mon[33862]: osdmap e236: 8 total, 8 up, 8 in 2024-03-21T14:12:39.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:39 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:39 smithi003 ceph-mon[29576]: osdmap e236: 8 total, 8 up, 8 in 2024-03-21T14:12:40.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:39 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:12:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:39 smithi003 ceph-mon[40510]: osdmap e236: 8 total, 8 up, 8 in 2024-03-21T14:12:40.009 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:40.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.148 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:40.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:40.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:40.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:40.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:40.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:40.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:40.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:12:40.583 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:12:40.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.1 'trim stale osdmaps' 2024-03-21T14:12:40.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:40.707 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:40.809 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:40.842 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:40.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:40.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:40 smithi118 ceph-mon[33862]: pgmap v443: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s; 2786/53361 objects misplaced (5.221%); 12 MiB/s, 187 objects/s recovering 2024-03-21T14:12:40.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:40.944 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:40 smithi003 ceph-mon[29576]: pgmap v443: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s; 2786/53361 objects misplaced (5.221%); 12 MiB/s, 187 objects/s recovering 2024-03-21T14:12:40.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:40 smithi003 ceph-mon[40510]: pgmap v443: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 1.1 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.02k op/s; 2786/53361 objects misplaced (5.221%); 12 MiB/s, 187 objects/s recovering 2024-03-21T14:12:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:41.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:41.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:41.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:41.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:41.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:41.400 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:41.436 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:41.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:41.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:41.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:41.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:41.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:41.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:41.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:41.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:41.859 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:41.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:41.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:41.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:41.974 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:42.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:42.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:42.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:42.292 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:42.323 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:42.361 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:12:42.361 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:216280 7.f 7:f89a58b1:::benchmark_data_smithi003_80362_object95263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:02.224319+0000", 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 280.13266448299999, 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113031408, 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216280 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224319+0000", 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224319+0000", 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.363 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224320+0000", 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7599999999999997e-07 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224323+0000", 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.19e-06 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224330+0000", 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.922e-06 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.364 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224336+0000", 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2910000000000004e-06 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.249153+0000", 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.024816110999999998 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.249276+0000", 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000123471 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.365 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.249406+0000", 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013040999999999999 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253123+0000", 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003716254 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253754+0000", 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00063169099999999998 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.366 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327175+0000", 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073420507999999995 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337107+0000", 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0099322019999999994 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337209+0000", 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101482 2024-03-21T14:12:42.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337351+0000", 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014229999999999999 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:216283 7.f 7:f841127a:::benchmark_data_smithi003_80362_object95266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:02.224403+0000", 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 280.13258047800002, 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113449484, 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.368 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 216283 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224403+0000", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224403+0000", 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224404+0000", 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9999999999999998e-07 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224406+0000", 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3029999999999998e-06 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224412+0000", 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3879999999999999e-06 2024-03-21T14:12:42.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.224416+0000", 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8969999999999999e-06 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253807+0000", 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.029390899000000002 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.253936+0000", 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000128765 2024-03-21T14:12:42.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,4,7", 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.254105+0000", 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016864199999999999 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327220+0000", 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073115515000000006 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327259+0000", 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8886e-05 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.372 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.327298+0000", 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8504000000000003e-05 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337363+0000", 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010065780999999999 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337522+0000", 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015878800000000001 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.373 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:02.337853+0000", 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000330716 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221713 7.f 7:fc6b6dba:::benchmark_data_smithi003_80362_object100696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.15900581199998, 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93373331800000003, 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.374 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221713 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197980+0000", 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.705e-06 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197988+0000", 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2139999999999996e-06 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198007+0000", 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9247999999999999e-05 2024-03-21T14:12:42.376 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198018+0000", 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0268000000000001e-05 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198077+0000", 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9280999999999997e-05 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198085+0000", 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7959999999999996e-06 2024-03-21T14:12:42.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119714+0000", 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92162893099999998 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119872+0000", 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158045 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.120011+0000", 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013935900000000001 2024-03-21T14:12:42.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131462+0000", 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011450777000000001 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131476+0000", 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4005999999999999e-05 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131627+0000", 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000150791 2024-03-21T14:12:42.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131673+0000", 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6919000000000003e-05 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131711+0000", 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7978000000000001e-05 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.380 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221725 7.f 7:fe480980:::benchmark_data_smithi003_80362_object100708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.13119081000002, 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90603860999999997, 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221725 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225795+0000", 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0959999999999999e-06 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225804+0000", 2024-03-21T14:12:42.382 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8510000000000005e-06 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225818+0000", 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3557000000000001e-05 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225828+0000", 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0644e-05 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225939+0000", 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110461 2024-03-21T14:12:42.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225950+0000", 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1712999999999999e-05 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123657+0000", 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.897706589 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123780+0000", 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012321000000000001 2024-03-21T14:12:42.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123890+0000", 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110142 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131508+0000", 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076180889999999998 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131594+0000", 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5677e-05 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.385 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131747+0000", 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015268700000000001 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131788+0000", 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1032999999999998e-05 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131832+0000", 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3860999999999999e-05 2024-03-21T14:12:42.386 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.389 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.389 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.389 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.389 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221726 7.f 7:fbed2f09:::benchmark_data_smithi003_80362_object100709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.13113455500002, 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91368115900000002, 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221726 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.391 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225850+0000", 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7000000000000005e-07 2024-03-21T14:12:42.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225854+0000", 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5920000000000001e-06 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225861+0000", 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4329999999999998e-06 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225867+0000", 2024-03-21T14:12:42.393 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2999999999999998e-06 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225979+0000", 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111437 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225986+0000", 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9e-06 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127686+0000", 2024-03-21T14:12:42.394 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90170028599999996 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127815+0000", 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012929099999999999 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127949+0000", 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013351500000000001 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139334+0000", 2024-03-21T14:12:42.395 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011385072 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139378+0000", 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3952999999999997e-05 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139425+0000", 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6764999999999997e-05 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139479+0000", 2024-03-21T14:12:42.396 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4245e-05 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139531+0000", 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999999e-05 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221737 7.f 7:f1651cbf:::benchmark_data_smithi003_80362_object100720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.11946910400002, 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90934260099999997, 2024-03-21T14:12:42.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221737 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.398 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237516+0000", 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0100000000000001e-06 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237521+0000", 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5430000000000001e-06 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.399 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237538+0000", 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7096e-05 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237547+0000", 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9870000000000005e-06 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237607+0000", 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9367000000000003e-05 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.400 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237616+0000", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5820000000000008e-06 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131857+0000", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89424055899999999 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131984+0000", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012695299999999999 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.401 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.132098+0000", 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113804 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139390+0000", 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072926889999999998 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139610+0000", 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000219626 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146690+0000", 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070798809999999997 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146801+0000", 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110599 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146857+0000", 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6904999999999998e-05 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221741 7.f 7:fa1cb433:::benchmark_data_smithi003_80362_object100724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.11555024799998, 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90562954600000001, 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221741 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241435+0000", 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4470000000000001e-06 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241442+0000", 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2640000000000002e-06 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241460+0000", 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7186000000000001e-05 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241469+0000", 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9719999999999998e-06 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241541+0000", 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2210000000000002e-05 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241551+0000", 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9399999999999997e-06 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135494+0000", 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89394319300000002 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135628+0000", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013400099999999999 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135788+0000", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016006200000000001 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139568+0000", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037803009999999998 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.408 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139659+0000", 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0662999999999996e-05 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146934+0000", 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072749190000000004 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.147017+0000", 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2694000000000005e-05 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.147063+0000", 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6693999999999997e-05 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221746 7.f 7:fec8ebdb:::benchmark_data_smithi003_80362_object100729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.110992264, 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90374667900000005, 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.410 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221746 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245993+0000", 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.432e-06 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245999+0000", 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.088e-06 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246013+0000", 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4158999999999999e-05 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246022+0000", 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7050000000000005e-06 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246089+0000", 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6668999999999995e-05 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246098+0000", 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6609999999999995e-06 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139691+0000", 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89359230599999995 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139805+0000", 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114026 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139913+0000", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108721 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146871+0000", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069575490000000004 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149464+0000", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025927440000000001 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149513+0000", 2024-03-21T14:12:42.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8949000000000002e-05 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149686+0000", 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000173308 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149738+0000", 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2361999999999998e-05 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.417 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221753 7.f 7:f88a5a14:::benchmark_data_smithi003_80362_object100736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.417 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:12:42.417 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.104919494, 2024-03-21T14:12:42.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89806965699999997, 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221753 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252065+0000", 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0200000000000002e-07 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252069+0000", 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7479999999999999e-06 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252076+0000", 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252083+0000", 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5230000000000001e-06 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252137+0000", 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4317999999999997e-05 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252147+0000", 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0257e-05 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143316+0000", 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89116910800000004 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143431+0000", 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011446300000000001 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.421 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143540+0000", 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108909 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149748+0000", 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062084599999999998 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149786+0000", 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7997000000000002e-05 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149826+0000", 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9373999999999999e-05 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150000+0000", 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017394 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150134+0000", 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013446700000000001 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.423 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221755 7.f 7:f52429fb:::benchmark_data_smithi003_80362_object100738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.10026174000001, 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90123331200000001, 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221755 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256723+0000", 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.223e-06 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256729+0000", 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7960000000000001e-06 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256742+0000", 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2394e-05 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256750+0000", 2024-03-21T14:12:42.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9270000000000005e-06 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256802+0000", 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2578999999999999e-05 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256811+0000", 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1349999999999998e-06 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150166+0000", 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89335426299999998 2024-03-21T14:12:42.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150275+0000", 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109422 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150383+0000", 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108297 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157790+0000", 2024-03-21T14:12:42.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074066669999999996 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157802+0000", 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2245e-05 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157842+0000", 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0120000000000002e-05 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157908+0000", 2024-03-21T14:12:42.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6168999999999996e-05 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157956+0000", 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7074999999999999e-05 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221760 7.f 7:f2fb3067:::benchmark_data_smithi003_80362_object100743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 273.07280272100002, 2024-03-21T14:12:42.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89272659300000001, 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221760 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284183+0000", 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.711e-06 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284190+0000", 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5209999999999997e-06 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284207+0000", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6325e-05 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284215+0000", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4319999999999999e-06 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284280+0000", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5042000000000005e-05 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:12:42.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284292+0000", 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2099000000000001e-05 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154224+0000", 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86993135600000004 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154329+0000", 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010535300000000001 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154439+0000", 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011024 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.166586+0000", 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012147120000000001 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.166631+0000", 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4297e-05 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176731+0000", 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010100074000000001 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176826+0000", 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5345000000000003e-05 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176908+0000", 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1678e-05 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.436 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30436 8.f 8:f8a7c51e:::benchmark_data_smithi003_110242_object30435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.502101469, 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12503477399999999, 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30436 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854883+0000", 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854905+0000", 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1386000000000002e-05 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854915+0000", 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0138e-05 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854922+0000", 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2810000000000003e-06 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860219+0000", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052964709999999996 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860345+0000", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012642600000000001 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860468+0000", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122581 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866120+0000", 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056519700000000001 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866164+0000", 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4041000000000001e-05 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866321+0000", 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015716800000000001 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866370+0000", 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8786999999999999e-05 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979917+0000", 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11354760999999999 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30438 8.f 8:fe80adc5:::benchmark_data_smithi003_110242_object30437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.50186745900001, 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12806651599999999, 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30438 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855117+0000", 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8999999999999995e-07 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855163+0000", 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6081000000000001e-05 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855173+0000", 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6180000000000001e-06 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855180+0000", 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.81e-06 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863014+0000", 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078345389999999997 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863134+0000", 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119196 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863267+0000", 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133738 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866211+0000", 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002943373 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979935+0000", 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113724297 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.982232+0000", 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002297112 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.982341+0000", 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010846800000000001 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983183+0000", 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084249399999999995 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30445 8.f 8:f69cc62f:::benchmark_data_smithi003_110242_object30444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.49871650899999, 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12520763700000001, 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30445 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858269+0000", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.344e-06 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858324+0000", 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4917000000000003e-05 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858335+0000", 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1732999999999999e-05 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858344+0000", 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2150000000000001e-06 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864522+0000", 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061783519999999998 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864635+0000", 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113094 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864765+0000", 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129839 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979940+0000", 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115175427 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980642+0000", 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070184399999999997 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983236+0000", 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025941480000000001 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983433+0000", 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019648300000000001 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983475+0000", 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2240999999999998e-05 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30459 8.0 8:00d9c66a:::benchmark_data_smithi003_110242_object30458:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.862891+0000", 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.494092886, 2024-03-21T14:12:42.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.121083247, 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30459 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862891+0000", 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862891+0000", 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862892+0000", 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2099999999999996e-07 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862911+0000", 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8842e-05 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862921+0000", 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0312999999999999e-05 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.862938+0000", 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7220000000000001e-05 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863494+0000", 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00055609799999999999 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863653+0000", 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158682 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863788+0000", 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000135127 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865255+0000", 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001466619 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865972+0000", 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00071715400000000001 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983870+0000", 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.117898002 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983934+0000", 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3995e-05 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983974+0000", 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0473999999999999e-05 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30464 8.0 8:02e6bb23:::benchmark_data_smithi003_110242_object30463:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.865950+0000", 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.49103430700001, 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120651603, 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.461 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30464 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865950+0000", 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865950+0000", 2024-03-21T14:12:42.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865951+0000", 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.248e-06 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865970+0000", 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8947999999999999e-05 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865979+0000", 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.431e-06 2024-03-21T14:12:42.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.865986+0000", 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3810000000000004e-06 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866019+0000", 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3278000000000002e-05 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866113+0000", 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4518000000000005e-05 2024-03-21T14:12:42.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866253+0000", 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013929099999999999 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981357+0000", 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11510411 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.984021+0000", 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026637779999999999 2024-03-21T14:12:42.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.986471+0000", 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024504570000000001 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.986560+0000", 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9351999999999994e-05 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.986601+0000", 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0810999999999999e-05 2024-03-21T14:12:42.466 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15132.0:30445 8.f e167/162 ondisk, result = 0)", 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.49050313800001, 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114169123, 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:12:42.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866481+0000", 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8700000000000005e-07 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866486+0000", 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7090000000000001e-06 2024-03-21T14:12:42.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979873+0000", 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113386367 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979886+0000", 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3909e-05 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980628+0000", 2024-03-21T14:12:42.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074143099999999997 2024-03-21T14:12:42.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980636+0000", 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3909999999999998e-06 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980650+0000", 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3628999999999999e-05 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15132.0:30464 8.0 e167/162 ondisk, result = 0)", 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:12:42.471 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 197.489630336, 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114010399, 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867354+0000", 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7700000000000004e-07 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.867358+0000", 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5760000000000002e-06 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979959+0000", 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11260105500000001 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979969+0000", 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0672999999999999e-05 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981344+0000", 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013744600000000001 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981351+0000", 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0090000000000001e-06 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.981364+0000", 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3049e-05 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:94635 8.0 8:01bfea60:::benchmark_data_smithi003_110242_object94634:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:10:09.868225+0000", 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.48875870099999, 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114449703, 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.475 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94635 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868225+0000", 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868225+0000", 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868227+0000", 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3909999999999999e-06 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868259+0000", 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2153999999999997e-05 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868269+0000", 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0375e-05 2024-03-21T14:12:42.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868277+0000", 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4989999999999997e-06 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868330+0000", 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3783999999999999e-05 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868457+0000", 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127021 2024-03-21T14:12:42.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868590+0000", 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132523 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.872254+0000", 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003663777 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.872411+0000", 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000156809 2024-03-21T14:12:42.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.982498+0000", 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11008781300000001 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.982560+0000", 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1329999999999997e-05 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.982675+0000", 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115227 2024-03-21T14:12:42.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:94636 8.f 8:f10a5fc1:::benchmark_data_smithi003_110242_object94635:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:10:09.868298+0000", 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 152.48868584799999, 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114544959, 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 94636 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.481 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868298+0000", 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868298+0000", 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868299+0000", 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2200000000000004e-07 2024-03-21T14:12:42.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868318+0000", 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9355999999999999e-05 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868325+0000", 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7390000000000002e-06 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868330+0000", 2024-03-21T14:12:42.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5249999999999996e-06 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868397+0000", 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6854999999999996e-05 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868540+0000", 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000143289 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.868670+0000", 2024-03-21T14:12:42.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012951999999999999 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.872199+0000", 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035295259999999998 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.873116+0000", 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00091664100000000005 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.982539+0000", 2024-03-21T14:12:42.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10942302800000001 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.982781+0000", 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024190600000000001 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:10:09.982843+0000", 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2051999999999995e-05 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:12:42.486 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:12:42.487 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:12:42.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:42.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:42.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:42.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:42.685 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:12:42.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:42.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:42.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:42 smithi118 ceph-mon[33862]: pgmap v445: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 971 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 1.22k op/s; 2605/47178 objects misplaced (5.522%); 9.8 MiB/s, 156 objects/s recovering 2024-03-21T14:12:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:42 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3222583918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:12:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:12:42.933 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:42.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:42.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:42 smithi003 ceph-mon[29576]: pgmap v445: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 971 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 1.22k op/s; 2605/47178 objects misplaced (5.522%); 9.8 MiB/s, 156 objects/s recovering 2024-03-21T14:12:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:42 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3222583918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:12:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:12:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:42 smithi003 ceph-mon[40510]: pgmap v445: 158 pgs: 1 peering, 3 active+remapped+backfill_toofull, 154 active+clean; 971 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 1.22k op/s; 2605/47178 objects misplaced (5.522%); 9.8 MiB/s, 156 objects/s recovering 2024-03-21T14:12:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:42 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3222583918' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:12:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:12:43.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:43.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:43.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:43.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:43.342 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:43.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:43.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:43.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:43.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:43.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:43.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:43.754 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:43.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:43.887 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:43.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:43.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:43 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:12:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:43 smithi003 ceph-mon[29576]: osdmap e237: 8 total, 8 up, 8 in 2024-03-21T14:12:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:43 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:12:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:43 smithi003 ceph-mon[40510]: osdmap e237: 8 total, 8 up, 8 in 2024-03-21T14:12:44.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:44.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:44.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:44.166 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:44.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:43 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:12:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:12:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:43 smithi118 ceph-mon[33862]: osdmap e237: 8 total, 8 up, 8 in 2024-03-21T14:12:44.268 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.302 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:44.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:44.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:44.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:44.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:44.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:44.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:44.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:44.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:44.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:44.853 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:44.955 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:44.984 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:44.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[29576]: 8.13 scrub starts 2024-03-21T14:12:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[29576]: 8.13 scrub ok 2024-03-21T14:12:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[29576]: pgmap v447: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 712 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 1955/34749 objects misplaced (5.626%) 2024-03-21T14:12:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[29576]: osdmap e238: 8 total, 8 up, 8 in 2024-03-21T14:12:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[40510]: 8.13 scrub starts 2024-03-21T14:12:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[40510]: 8.13 scrub ok 2024-03-21T14:12:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[40510]: pgmap v447: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 712 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 1955/34749 objects misplaced (5.626%) 2024-03-21T14:12:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:44 smithi003 ceph-mon[40510]: osdmap e238: 8 total, 8 up, 8 in 2024-03-21T14:12:45.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:45.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:44 smithi118 ceph-mon[33862]: 8.13 scrub starts 2024-03-21T14:12:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:44 smithi118 ceph-mon[33862]: 8.13 scrub ok 2024-03-21T14:12:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:44 smithi118 ceph-mon[33862]: pgmap v447: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 712 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 1955/34749 objects misplaced (5.626%) 2024-03-21T14:12:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:44 smithi118 ceph-mon[33862]: osdmap e238: 8 total, 8 up, 8 in 2024-03-21T14:12:45.225 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:45.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:45.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:45.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:45.509 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.534 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:45.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:45.635 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.670 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:45.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:45.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:45.804 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:45.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:45.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:45.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: 8.15 scrub starts 2024-03-21T14:12:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: 8.15 scrub ok 2024-03-21T14:12:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:12:46.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[29576]: osdmap e239: 8 total, 8 up, 8 in 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: 8.15 scrub starts 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: 8.15 scrub ok 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:12:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:45 smithi003 ceph-mon[40510]: osdmap e239: 8 total, 8 up, 8 in 2024-03-21T14:12:46.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.080 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:46.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:46.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:46.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:46.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: 8.15 scrub starts 2024-03-21T14:12:46.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: 8.15 scrub ok 2024-03-21T14:12:46.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:12:46.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:12:46.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:45 smithi118 ceph-mon[33862]: osdmap e239: 8 total, 8 up, 8 in 2024-03-21T14:12:46.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:46.323 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:46.354 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:46.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:46.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:46.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:46.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:46.741 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[29576]: pgmap v449: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 712 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 1955/34749 objects misplaced (5.626%) 2024-03-21T14:12:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:12:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:12:46.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:46.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:12:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:12:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[40510]: pgmap v449: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 712 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 1955/34749 objects misplaced (5.626%) 2024-03-21T14:12:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:12:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:12:46.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:46.775 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:46.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:46.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:46.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:46.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:47.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:47.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:47.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:47.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:12:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:12:47.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:46 smithi118 ceph-mon[33862]: pgmap v449: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 712 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 1955/34749 objects misplaced (5.626%) 2024-03-21T14:12:47.184 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:46 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:12:47.184 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:46 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:12:47.184 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:12:47.188 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:47.290 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:47.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:47.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:47.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:47.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:47.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.593 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:47.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:47.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:47.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:47.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:47.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:47.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:47.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:47.993 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:47.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:48.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:48.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:47 smithi003 ceph-mon[29576]: 8.11 scrub starts 2024-03-21T14:12:48.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:47 smithi003 ceph-mon[29576]: 8.11 scrub ok 2024-03-21T14:12:48.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:48.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:47 smithi003 ceph-mon[40510]: 8.11 scrub starts 2024-03-21T14:12:48.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:47 smithi003 ceph-mon[40510]: 8.11 scrub ok 2024-03-21T14:12:48.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:48.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:47 smithi118 ceph-mon[33862]: 8.11 scrub starts 2024-03-21T14:12:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:47 smithi118 ceph-mon[33862]: 8.11 scrub ok 2024-03-21T14:12:48.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:48.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:48.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:48.375 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:48.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:48.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:48.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:48.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:48.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:48.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:48.819 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:48.921 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:48.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:48.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:48 smithi003 ceph-mon[29576]: pgmap v451: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 60 KiB/s rd, 59 op/s; 1028/16845 objects misplaced (6.103%) 2024-03-21T14:12:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:48 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:12:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:48 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:12:49.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:48 smithi003 ceph-mon[40510]: pgmap v451: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 60 KiB/s rd, 59 op/s; 1028/16845 objects misplaced (6.103%) 2024-03-21T14:12:49.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:48 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:12:49.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:48 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:12:49.064 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.099 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:49.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:48 smithi118 ceph-mon[33862]: pgmap v451: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 60 KiB/s rd, 59 op/s; 1028/16845 objects misplaced (6.103%) 2024-03-21T14:12:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:48 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:12:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:48 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:12:49.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:49.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:49.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:49.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:49.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:49.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:49.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:49.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:49.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:49.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 8.14 scrub starts 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 8.14 scrub ok 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:12:49.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 8.e scrub starts 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: 8.e scrub ok 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[29576]: osdmap e240: 8 total, 8 up, 8 in 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 8.14 scrub starts 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 8.14 scrub ok 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 8.e scrub starts 2024-03-21T14:12:49.813 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:12:49.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:12:49.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: 8.e scrub ok 2024-03-21T14:12:49.814 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:49 smithi003 ceph-mon[40510]: osdmap e240: 8 total, 8 up, 8 in 2024-03-21T14:12:49.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:49.929 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:49.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:49.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:50.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:50.062 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.063 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:50.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:12:50.146 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 8.14 scrub starts 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 8.14 scrub ok 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 8.e scrub starts 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: 8.e scrub ok 2024-03-21T14:12:50.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:49 smithi118 ceph-mon[33862]: osdmap e240: 8 total, 8 up, 8 in 2024-03-21T14:12:50.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:50.166 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:50.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:50.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:50.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:50.469 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:50.493 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:12:50.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:50.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:50.620 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.655 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:50.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:50.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:50.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:50.894 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 130914 objects 2024-03-21T14:12:50.894 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :127.986 2024-03-21T14:12:50.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:50.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:50.925 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:50.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:50 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:50 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:50 smithi118 ceph-mon[33862]: pgmap v452: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 33 KiB/s rd, 33 op/s; 1028/16845 objects misplaced (6.103%) 2024-03-21T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:50 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:12:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:50 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[29576]: pgmap v452: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 33 KiB/s rd, 33 op/s; 1028/16845 objects misplaced (6.103%) 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:12:51.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:12:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[40510]: pgmap v452: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 339 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 33 KiB/s rd, 33 op/s; 1028/16845 objects misplaced (6.103%) 2024-03-21T14:12:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:12:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:50 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:12:51.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:51.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:51.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:12:51.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:51.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:51.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:51.405 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:12:51.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:51.427 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:12:51.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:51.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:51.450 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:12:51.470 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:12:51.491 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:12:51.511 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:12:51.531 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:12:51.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:51.553 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:12:51.554 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:12:51.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:51.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:51.691 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:51.793 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:51.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:51.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:51 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:12:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:51 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:12:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:51 smithi118 ceph-mon[33862]: 4.4 scrub starts 2024-03-21T14:12:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:51 smithi118 ceph-mon[33862]: 4.4 scrub ok 2024-03-21T14:12:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:51 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/873065535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:51.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:12:51.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:12:51.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[29576]: 4.4 scrub starts 2024-03-21T14:12:51.941 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[29576]: 4.4 scrub ok 2024-03-21T14:12:51.942 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/873065535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:51.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:12:51.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:12:51.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[40510]: 4.4 scrub starts 2024-03-21T14:12:51.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[40510]: 4.4 scrub ok 2024-03-21T14:12:51.942 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:51 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/873065535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:12:51.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:51.946 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:51.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:52.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:52.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:52.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:52.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:52.186 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:52.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:52.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.326 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:52.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:52.377 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:12:52.377 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:12:52.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:52.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:52.480 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-21T14:12:52.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-21T14:12:52.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:52.580 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:52.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:52.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:52.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:52.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.832 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:52.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:52 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:12:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:52 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:12:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:52 smithi118 ceph-mon[33862]: pgmap v454: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 214 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 33 op/s; 864/10845 objects misplaced (7.967%) 2024-03-21T14:12:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:52 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:12:52.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:52 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:12:52.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:52.957 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:52.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:52.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[40510]: pgmap v454: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 214 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 33 op/s; 864/10845 objects misplaced (7.967%) 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:12:52.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[29576]: pgmap v454: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 214 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 33 KiB/s rd, 33 op/s; 864/10845 objects misplaced (7.967%) 2024-03-21T14:12:52.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:12:52.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:52 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:12:53.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:12:53.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:53.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:53.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:53.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:53.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:53.272 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:12:53.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.338 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:53.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:53.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:53.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:53.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:53.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:53.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:53.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:53.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:53.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:53.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:53.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:12:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:12:53.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:53 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:12:53.966 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:53.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:53.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:12:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:12:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:12:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:12:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:12:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:12:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:12:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:12:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:12:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:12:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:12:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:12:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:12:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:12:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:12:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:53 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:12:54.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:54.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:54.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:54.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.243 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:54.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:54.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.368 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:54.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:54.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:54.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:54.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:54.521 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:12:54.522 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:12:54.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:54.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:54.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:54.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:54.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:54.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:54.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:54.871 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:12:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:12:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: pgmap v455: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 28 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:12:54.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:54 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:12:54.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:54.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:12:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:54.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:54.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: pgmap v455: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 28 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:12:55.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: pgmap v455: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 28 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:12:55.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:54 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:12:55.100 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.124 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:55.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:55.157 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:12:55.157 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:12:55.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.2 'trim stale osdmaps' 2024-03-21T14:12:55.225 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:55.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:55.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:55.329 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:12:55.329 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:12:55.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.376 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:55.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:55.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.503 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:55.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:55.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:55.617 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:55.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:55.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:55.643 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:55.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:55.767 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:55.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:55.868 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:55.892 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:55.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:55.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:55.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: 8.b scrub starts 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: 8.b scrub ok 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: osdmap e241: 8 total, 8 up, 8 in 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: 8.b scrub starts 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: 8.b scrub ok 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: osdmap e241: 8 total, 8 up, 8 in 2024-03-21T14:12:56.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:12:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:12:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:12:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:55 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:12:56.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.018 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:56.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:56.050 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: 8.b scrub starts 2024-03-21T14:12:56.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: 8.b scrub ok 2024-03-21T14:12:56.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: osdmap e241: 8 total, 8 up, 8 in 2024-03-21T14:12:56.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:12:56.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:12:56.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:12:56.051 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:55 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:12:56.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:12:56.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:56.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:56.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:56.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:56.354 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:12:56.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:56.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:56.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:56.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:56.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:56.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:56.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:56.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:56.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:12:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:12:56.767 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:12:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:56.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:56.882 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:56.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:56.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:57.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:57.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:57.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:12:57.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: pgmap v457: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3771370790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: pgmap v457: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3771370790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:57.108 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:12:57.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:56 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:12:57.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:57.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:57.174 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: pgmap v457: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3771370790' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:12:57.175 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:56 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:12:57.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:57.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:57.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:57.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.414 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:57.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:57.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:57.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:57.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:57.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:57.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:57.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:57.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:57.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:57.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:12:57.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:12:57.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: osdmap e242: 8 total, 8 up, 8 in 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:57.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: osdmap e242: 8 total, 8 up, 8 in 2024-03-21T14:12:57.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:57.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:12:57.855 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:57 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:12:57.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:57.934 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:57.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:58.035 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:58.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:12:58.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:12:58.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:12:58.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:12:58.054 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: osdmap e242: 8 total, 8 up, 8 in 2024-03-21T14:12:58.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:12:58.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:12:58.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:57 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:12:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.060 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:58.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:58.112 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:12:58.112 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:12:58.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:12:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.187 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:58.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:12:58.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:12:58.312 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:12:58.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:58.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:58.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:58.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:58.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:58.654 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:12:58.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:58.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:58.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:12:58.793 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:58.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:58.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:58.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:58.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:58.944 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:58.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:58.976 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:12:59.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:59.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:59.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:12:59.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:58 smithi118 ceph-mon[33862]: pgmap v459: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:59.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:58 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:12:59.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:58 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:12:59.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:58 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:12:59.098 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:58 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:12:59.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[29576]: pgmap v459: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:59.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:12:59.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:12:59.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:12:59.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:12:59.160 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[40510]: pgmap v459: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:12:59.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:12:59.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:12:59.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:12:59.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:58 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:12:59.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:12:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:59.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:12:59.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:12:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:59.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:12:59.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:59.445 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:59.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:59.541 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_1' removed 2024-03-21T14:12:59.547 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:12:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.574 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:59.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:12:59.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:12:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:59.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:12:59.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:12:59.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.832 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:12:59.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:12:59.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 6.17 deep-scrub starts 2024-03-21T14:12:59.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 6.17 deep-scrub ok 2024-03-21T14:12:59.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 3.0 scrub starts 2024-03-21T14:12:59.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 3.0 scrub ok 2024-03-21T14:12:59.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1712550478' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:12:59.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1712550478' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[29576]: osdmap e243: 8 total, 8 up, 8 in 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 6.17 deep-scrub starts 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 6.17 deep-scrub ok 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 3.0 scrub starts 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 3.0 scrub ok 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1712550478' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:12:59.927 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:12:59.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:12:59.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:12:59.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1712550478' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:12:59.928 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:12:59 smithi003 ceph-mon[40510]: osdmap e243: 8 total, 8 up, 8 in 2024-03-21T14:12:59.933 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:12:59.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:12:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:12:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 6.17 deep-scrub starts 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 6.17 deep-scrub ok 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 3.0 scrub starts 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 3.0 scrub ok 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1712550478' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1712550478' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:13:00.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:12:59 smithi118 ceph-mon[33862]: osdmap e243: 8 total, 8 up, 8 in 2024-03-21T14:13:00.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:00.082 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:00.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:00.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:00.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:00.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:00.397 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T14:13:00.398 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T14:13:00.399 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-21T14:13:00.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-21T14:13:00.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:00.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:00.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:00.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:00.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:00.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.720 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:00.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:00.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:00.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:00.922 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 8.c scrub starts 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 8.c scrub ok 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: pgmap v460: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 6.1 deep-scrub starts 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 6.1 deep-scrub ok 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:13:00.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:00 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:13:00.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:00.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:00.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 8.c scrub starts 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 8.c scrub ok 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: pgmap v460: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 6.1 deep-scrub starts 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 6.1 deep-scrub ok 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:13:01.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 8.c scrub starts 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 8.c scrub ok 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: pgmap v460: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 7.8 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 276/942 objects misplaced (29.299%) 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 6.1 deep-scrub starts 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 6.1 deep-scrub ok 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:13:01.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:13:01.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:13:01.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:13:01.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:00 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:13:01.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:01.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:01.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:01.103 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:01.146 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:13:01.146 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:13:01.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:01.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:01.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:01.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:01.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.349 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:01.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:01.450 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.476 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:01.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:01.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:01.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:01.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:01.616 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:13:01.617 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:13:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:13:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:13:01.618 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:13:01.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:13:01.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:01.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.730 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:01.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:01.811 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:13:01.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:01.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:01.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:01.980 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:01.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:01.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:02.048 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:02.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:02.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:13:02.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:13:02.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:02.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:02.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:13:02.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:13:02.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:02.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.108 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:02.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:02.170 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:01 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:13:02.170 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:01 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:13:02.170 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:01 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:02.170 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:02.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:02.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:02.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:02.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:02.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:02.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:02.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.487 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:02.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:02.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:02.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:02.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:02.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:02.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:02.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.865 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:02.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[29576]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[29576]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[29576]: 5.6 deep-scrub starts 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[29576]: 5.6 deep-scrub ok 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[40510]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:13:02.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[40510]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-03-21T14:13:02.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:13:02.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[40510]: 5.6 deep-scrub starts 2024-03-21T14:13:02.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:02 smithi003 ceph-mon[40510]: 5.6 deep-scrub ok 2024-03-21T14:13:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:13:02.907 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:13:02.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.3 'trim stale osdmaps' 2024-03-21T14:13:02.966 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:02.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:02.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:02.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:03.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:03.094 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:02 smithi118 ceph-mon[33862]: pgmap v462: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:13:03.094 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:02 smithi118 ceph-mon[33862]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull) 2024-03-21T14:13:03.094 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:02 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:13:03.094 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:02 smithi118 ceph-mon[33862]: 5.6 deep-scrub starts 2024-03-21T14:13:03.094 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:02 smithi118 ceph-mon[33862]: 5.6 deep-scrub ok 2024-03-21T14:13:03.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:03.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:03.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:03.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:03.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:03.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:03.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:03.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:03.470 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:03.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:03.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:03.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:03.688 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:13:03.688 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:13:03.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:03.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:03.848 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:13:03.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:03.868 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:13:03.871 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:03.887 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:13:03.907 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:13:03.928 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:13:03.948 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:13:03.969 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:13:03.969 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:13:03.969 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:13:03.969 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[40510]: 3.6 deep-scrub starts 2024-03-21T14:13:03.969 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[40510]: 3.6 deep-scrub ok 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[29576]: 3.6 deep-scrub starts 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[29576]: 3.6 deep-scrub ok 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:13:03.970 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:03 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:13:03.973 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:03.990 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:13:03.991 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:13:03.997 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:03.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:04.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:03 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:13:04.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:03 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:13:04.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:03 smithi118 ceph-mon[33862]: 3.6 deep-scrub starts 2024-03-21T14:13:04.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:03 smithi118 ceph-mon[33862]: 3.6 deep-scrub ok 2024-03-21T14:13:04.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:03 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:13:04.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:03 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:13:04.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:04.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:04.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:04.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:04.292 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:13:04.352 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:04.398 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:04.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:04.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:04.584 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:04.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:04.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:04.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:04.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:04.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.778 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:04.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:04.879 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:04.904 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:04.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[29576]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:13:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:13:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:13:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:13:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:13:05.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2554004540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[40510]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:13:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:13:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:13:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:13:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:13:05.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2554004540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:05.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:05.029 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:05.131 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:05.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.156 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:04 smithi118 ceph-mon[33862]: pgmap v463: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:04 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:04 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:04 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:04 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:13:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2554004540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:05.257 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:05.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:05.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.408 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:05.509 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:05.534 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:05.635 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:05.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.662 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:05.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:05.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:05.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:05.915 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:05.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:13:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:13:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[29576]: 6.4 deep-scrub starts 2024-03-21T14:13:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[29576]: 6.4 deep-scrub ok 2024-03-21T14:13:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:13:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:13:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:13:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:13:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[40510]: 6.4 deep-scrub starts 2024-03-21T14:13:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[40510]: 6.4 deep-scrub ok 2024-03-21T14:13:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:13:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:05 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:13:06.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:06.039 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:05 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:13:06.039 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:05 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:13:06.039 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:05 smithi118 ceph-mon[33862]: 6.4 deep-scrub starts 2024-03-21T14:13:06.039 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:05 smithi118 ceph-mon[33862]: 6.4 deep-scrub ok 2024-03-21T14:13:06.039 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:05 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:13:06.039 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:05 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:13:06.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:06.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:06.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:06.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:06.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.304 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:06.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:06.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:06.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:06.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:06.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:06.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:06.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:13:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:13:06.787 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.814 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:06.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:06.915 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:06.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:06.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:06.970 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:06 smithi118 ceph-mon[33862]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 471 B/s rd, 0 op/s 2024-03-21T14:13:06.970 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:06 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:13:06.973 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:06 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:13:06.973 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:13:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:13:07.041 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:07.065 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:07.160 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-mon[29576]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 471 B/s rd, 0 op/s 2024-03-21T14:13:07.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:13:07.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:13:07.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-mon[40510]: pgmap v464: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 471 B/s rd, 0 op/s 2024-03-21T14:13:07.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:13:07.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:06 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:13:07.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.192 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:07.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:07.278 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:13:07.278 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:13:07.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:13:07.278 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:13:07.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.278 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:30468 8.8 8:10cf6a6e:::benchmark_data_smithi003_110242_object30467:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:09:24.869329+0000", 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.40651004700001, 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.118890227, 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 30468 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869329+0000", 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869329+0000", 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869330+0000", 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.23e-07 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869384+0000", 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4286000000000002e-05 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869392+0000", 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2339999999999994e-06 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869397+0000", 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6099999999999999e-06 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869436+0000", 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9310999999999997e-05 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869516+0000", 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9581999999999998e-05 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.869607+0000", 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1323999999999995e-05 2024-03-21T14:13:07.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.870569+0000", 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00096230700000000003 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.870832+0000", 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026244099999999998 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.988111+0000", 2024-03-21T14:13:07.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.11727965899999999 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.988171+0000", 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9354000000000001e-05 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:09:24.988219+0000", 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8296e-05 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243719 8.11 8:8ef64855:::benchmark_data_smithi003_110242_object112802:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.546761+0000", 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.729078131000001, 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0279333799999999, 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243719 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546761+0000", 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546761+0000", 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546762+0000", 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.263e-06 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546770+0000", 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1790000000000008e-06 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546780+0000", 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0131e-05 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546788+0000", 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0669999999999992e-06 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546842+0000", 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3109000000000003e-05 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.546851+0000", 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4469999999999995e-06 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.567295+0000", 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.020444444 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.567417+0000", 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121119 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.567831+0000", 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041481200000000003 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.572323+0000", 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044914739999999996 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574427+0000", 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00210436 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574583+0000", 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000156193 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574640+0000", 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6749999999999997e-05 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574694+0000", 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4032e-05 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243799 8.11 8:89295bb2:::benchmark_data_smithi003_110242_object112882:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.598770+0000", 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.677068404, 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.976086495, 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.291 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243799 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598770+0000", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598770+0000", 2024-03-21T14:13:07.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598771+0000", 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8500000000000002e-07 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598773+0000", 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.424e-06 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598795+0000", 2024-03-21T14:13:07.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1407999999999999e-05 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598800+0000", 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.135e-06 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598853+0000", 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2879e-05 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.598861+0000", 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7170000000000008e-06 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.570045+0000", 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97118488800000002 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.570183+0000", 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137255 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.570479+0000", 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029630500000000002 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574757+0000", 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042776139999999999 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574771+0000", 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4698999999999999e-05 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574800+0000", 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8836999999999999e-05 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574829+0000", 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8979000000000002e-05 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574857+0000", 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7869999999999999e-05 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243802 8.11 8:8bd3ba69:::benchmark_data_smithi003_110242_object112885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.599542+0000", 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.676296806, 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97546381699999996, 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243802 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599542+0000", 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599542+0000", 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599543+0000", 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9699999999999996e-07 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599550+0000", 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6480000000000003e-06 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599556+0000", 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7289999999999997e-06 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599560+0000", 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1729999999999997e-06 2024-03-21T14:13:07.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599598+0000", 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7879000000000001e-05 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.599606+0000", 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6399999999999997e-06 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.572374+0000", 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97276807799999998 2024-03-21T14:13:07.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.572476+0000", 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000102747 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.572704+0000", 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000227135 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574778+0000", 2024-03-21T14:13:07.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020743049999999998 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574903+0000", 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012519699999999999 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574953+0000", 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9944999999999997e-05 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.574980+0000", 2024-03-21T14:13:07.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7351999999999998e-05 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.575006+0000", 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5491999999999999e-05 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:07.306 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243856 8.11 8:89b57920:::benchmark_data_smithi003_110242_object112939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.637062+0000", 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.638776592999999, 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94453151300000004, 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243856 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637062+0000", 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637062+0000", 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637064+0000", 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5060000000000001e-06 2024-03-21T14:13:07.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637072+0000", 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1079999999999993e-06 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637086+0000", 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3869000000000001e-05 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637094+0000", 2024-03-21T14:13:07.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0099999999999995e-06 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637141+0000", 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7426000000000003e-05 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.637147+0000", 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6350000000000003e-06 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.310 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.575042+0000", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.93789502499999999 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.575174+0000", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131944 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.575455+0000", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028151000000000001 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581404+0000", 2024-03-21T14:13:07.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0059489779999999997 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581469+0000", 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5135000000000006e-05 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581485+0000", 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5503e-05 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581540+0000", 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4759000000000002e-05 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581594+0000", 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4104999999999998e-05 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243867 8.11 8:8e5c3dee:::benchmark_data_smithi003_110242_object112950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.647579+0000", 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.628259446000001, 2024-03-21T14:13:07.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.93419456300000003, 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243867 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647579+0000", 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647579+0000", 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647580+0000", 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2699999999999998e-07 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647585+0000", 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9919999999999998e-06 2024-03-21T14:13:07.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647607+0000", 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1270000000000001e-05 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647614+0000", 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7440000000000007e-06 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647673+0000", 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9083000000000003e-05 2024-03-21T14:13:07.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.647681+0000", 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1179999999999992e-06 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.577445+0000", 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.92976326099999995 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.577568+0000", 2024-03-21T14:13:07.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012307 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.577940+0000", 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000371911 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581604+0000", 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036639559999999999 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581629+0000", 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5075e-05 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581679+0000", 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0108000000000002e-05 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581732+0000", 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2663e-05 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581774+0000", 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2484999999999997e-05 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243917 8.11 8:89163da5:::benchmark_data_smithi003_110242_object113000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.683435+0000", 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.592404294000001, 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90286713699999999, 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.320 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243917 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683435+0000", 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683435+0000", 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683435+0000", 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.32e-07 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683438+0000", 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.841e-06 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683444+0000", 2024-03-21T14:13:07.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9769999999999997e-06 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683449+0000", 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1619999999999999e-06 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683857+0000", 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040839499999999998 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.683866+0000", 2024-03-21T14:13:07.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8219999999999996e-06 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.579801+0000", 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.895934958 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.579931+0000", 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012970200000000001 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.580169+0000", 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023787100000000001 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581786+0000", 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001617081 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586156+0000", 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043697270000000003 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586199+0000", 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3302e-05 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586248+0000", 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9475999999999999e-05 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586302+0000", 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3390999999999998e-05 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.326 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243922 8.11 8:8b7e97df:::benchmark_data_smithi003_110242_object113005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.686174+0000", 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.589665081, 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90029779399999998, 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243922 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686174+0000", 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686174+0000", 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686175+0000", 2024-03-21T14:13:07.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1310000000000001e-06 2024-03-21T14:13:07.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686185+0000", 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5589999999999994e-06 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686196+0000", 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1935e-05 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686205+0000", 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0600000000000008e-06 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686256+0000", 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1616000000000002e-05 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.686265+0000", 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.755e-06 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581830+0000", 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89556495000000003 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.581951+0000", 2024-03-21T14:13:07.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120951 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.582196+0000", 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024533299999999998 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586168+0000", 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039716789999999997 2024-03-21T14:13:07.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586340+0000", 2024-03-21T14:13:07.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017205399999999999 2024-03-21T14:13:07.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586382+0000", 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1668000000000002e-05 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586433+0000", 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1530000000000003e-05 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586472+0000", 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8572999999999998e-05 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:243989 8.11 8:8e2dd805:::benchmark_data_smithi003_110242_object113072:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.336 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.743069+0000", 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.532770132, 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85136204800000004, 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.337 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 243989 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743069+0000", 2024-03-21T14:13:07.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743069+0000", 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743070+0000", 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743075+0000", 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1290000000000004e-06 2024-03-21T14:13:07.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743085+0000", 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.679e-06 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743099+0000", 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4419e-05 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743167+0000", 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8308999999999999e-05 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.743176+0000", 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6670000000000002e-06 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.583793+0000", 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84061706000000003 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.583918+0000", 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000124487 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.584218+0000", 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000299981 2024-03-21T14:13:07.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586350+0000", 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021323179999999998 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594228+0000", 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078784630000000005 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594336+0000", 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108014 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594384+0000", 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7973000000000003e-05 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594431+0000", 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6372000000000003e-05 2024-03-21T14:13:07.345 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244011 8.11 8:8e900183:::benchmark_data_smithi003_110242_object113094:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.759406+0000", 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.516433031999998, 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.83524907999999998, 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.346 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244011 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759406+0000", 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759406+0000", 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759406+0000", 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0900000000000001e-07 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759410+0000", 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8650000000000003e-06 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759423+0000", 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2181000000000001e-05 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759430+0000", 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9319999999999996e-06 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.349 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759484+0000", 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3236000000000003e-05 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.759492+0000", 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5299999999999996e-06 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586499+0000", 2024-03-21T14:13:07.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82700686800000001 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586621+0000", 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121662 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.586864+0000", 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000243203 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594247+0000", 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0073826940000000004 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594284+0000", 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7753999999999998e-05 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594492+0000", 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020757300000000001 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594580+0000", 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7713999999999994e-05 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594655+0000", 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5259000000000006e-05 2024-03-21T14:13:07.353 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244037 8.11 8:898348dc:::benchmark_data_smithi003_110242_object113120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.778359+0000", 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.497479859999999, 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.816586967, 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.354 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244037 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778359+0000", 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778359+0000", 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778360+0000", 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0999999999999997e-07 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778367+0000", 2024-03-21T14:13:07.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2119999999999997e-06 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778379+0000", 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.188e-05 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778387+0000", 2024-03-21T14:13:07.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7449999999999995e-06 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778451+0000", 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4412000000000006e-05 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.778461+0000", 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9049999999999992e-06 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.588621+0000", 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81016001299999996 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.588725+0000", 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104287 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.589036+0000", 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031077200000000001 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594300+0000", 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052640459999999997 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594727+0000", 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042729900000000002 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594769+0000", 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1337000000000001e-05 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594817+0000", 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7840000000000003e-05 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594946+0000", 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012940900000000001 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244045 8.11 8:8ef9cc1b:::benchmark_data_smithi003_110242_object113128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.782852+0000", 2024-03-21T14:13:07.362 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.492986746, 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81227314100000003, 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244045 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782852+0000", 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782852+0000", 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782853+0000", 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9100000000000004e-07 2024-03-21T14:13:07.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782856+0000", 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7330000000000001e-06 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782866+0000", 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1500000000000005e-06 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782871+0000", 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3349999999999997e-06 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782933+0000", 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1731999999999998e-05 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.782941+0000", 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7930000000000003e-06 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.591029+0000", 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.80808713200000004 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.591150+0000", 2024-03-21T14:13:07.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121426 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.591459+0000", 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000308509 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594667+0000", 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032080289999999998 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.594993+0000", 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000326888 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.595036+0000", 2024-03-21T14:13:07.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2277999999999997e-05 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.595086+0000", 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9935000000000002e-05 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.595125+0000", 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9610000000000002e-05 2024-03-21T14:13:07.370 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244073 8.11 8:8b50be3a:::benchmark_data_smithi003_110242_object113156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.809406+0000", 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.466433067000001, 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.79318344299999999, 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.371 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244073 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809406+0000", 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809406+0000", 2024-03-21T14:13:07.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809407+0000", 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.33e-07 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809410+0000", 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.698e-06 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809419+0000", 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1710000000000002e-06 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809424+0000", 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0250000000000002e-06 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809476+0000", 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2502e-05 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.809484+0000", 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1820000000000006e-06 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.595159+0000", 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.78567468500000004 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.595256+0000", 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6867000000000006e-05 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.595639+0000", 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038311200000000001 2024-03-21T14:13:07.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602413+0000", 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067741800000000003 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602455+0000", 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1903000000000002e-05 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602501+0000", 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5657e-05 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602548+0000", 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7515999999999997e-05 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602589+0000", 2024-03-21T14:13:07.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1012e-05 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244169 8.11 8:8b6ccc4e:::benchmark_data_smithi003_110242_object113252:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.902394+0000", 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.373445161999999, 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.70799501499999995, 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.379 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244169 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902394+0000", 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902394+0000", 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902395+0000", 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2410000000000001e-06 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902400+0000", 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6920000000000001e-06 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902410+0000", 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0739e-05 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902418+0000", 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6059999999999997e-06 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.382 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902472+0000", 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4431000000000001e-05 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.902479+0000", 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6919999999999996e-06 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.597747+0000", 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.69526806500000005 2024-03-21T14:13:07.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.597856+0000", 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010857399999999999 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.598204+0000", 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034825200000000001 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.384 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602467+0000", 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042625500000000004 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.607468+0000", 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050012049999999999 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610297+0000", 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028292679999999998 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.385 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610346+0000", 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9150000000000002e-05 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610389+0000", 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2549999999999997e-05 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.386 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244175 8.11 8:8970d6d0:::benchmark_data_smithi003_110242_object113258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.907994+0000", 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.367844458, 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.70997841500000003, 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244175 2024-03-21T14:13:07.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.388 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.388 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.388 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.388 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.907994+0000", 2024-03-21T14:13:07.388 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.389 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.389 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.907994+0000", 2024-03-21T14:13:07.389 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.389 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.907995+0000", 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4799999999999998e-07 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.907999+0000", 2024-03-21T14:13:07.390 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.211e-06 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.908011+0000", 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1228e-05 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.908018+0000", 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6930000000000007e-06 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.908073+0000", 2024-03-21T14:13:07.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4549000000000002e-05 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.908081+0000", 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5909999999999996e-06 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.600063+0000", 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.69198209600000005 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.600169+0000", 2024-03-21T14:13:07.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105938 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.600432+0000", 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026291599999999997 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.607612+0000", 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071794679999999996 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.607625+0000", 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2884000000000001e-05 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.617839+0000", 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010213874 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.617930+0000", 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1905999999999997e-05 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.617973+0000", 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2413000000000002e-05 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244201 8.11 8:8d4d448e:::benchmark_data_smithi003_110242_object113284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:35.948877+0000", 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.326962290000001, 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.66921844500000005, 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.395 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244201 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948877+0000", 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948877+0000", 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948878+0000", 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2720000000000001e-06 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948884+0000", 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7010000000000001e-06 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948895+0000", 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1836e-05 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948901+0000", 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8109999999999999e-06 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948984+0000", 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2649000000000001e-05 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:35.948990+0000", 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2480000000000001e-06 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602634+0000", 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.65364361699999995 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.602740+0000", 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106069 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.603050+0000", 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031045399999999999 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610224+0000", 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071738419999999997 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610258+0000", 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4230000000000003e-05 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.618011+0000", 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077528570000000002 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.618058+0000", 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.706e-05 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.618095+0000", 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6798999999999998e-05 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.402 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:244221 8.11 8:8fe1dda9:::benchmark_data_smithi003_110242_object113304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:36.004817+0000", 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 31.271021495999999, 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.61339977599999995, 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244221 2024-03-21T14:13:07.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.004817+0000", 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.004817+0000", 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.004820+0000", 2024-03-21T14:13:07.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3599999999999999e-06 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.004829+0000", 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6889999999999999e-06 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.004855+0000", 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5933000000000001e-05 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.004877+0000", 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1455999999999999e-05 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.005047+0000", 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017008299999999999 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.005065+0000", 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7629999999999999e-05 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.605041+0000", 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.59997660200000003 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.605173+0000", 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013206200000000001 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.605467+0000", 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029354000000000001 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.407 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610269+0000", 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048022500000000001 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.610686+0000", 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041692500000000001 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.618133+0000", 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074475490000000004 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.408 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.618181+0000", 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7036999999999997e-05 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:36.618217+0000", 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6659999999999998e-05 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:254060 8.11 8:8f905e4a:::benchmark_data_smithi003_110242_object123143:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:44.556468+0000", 2024-03-21T14:13:07.409 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.719370673, 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.119204558, 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 254060 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556468+0000", 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556468+0000", 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556470+0000", 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3549999999999999e-06 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556475+0000", 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4009999999999996e-06 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556488+0000", 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2984e-05 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556497+0000", 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9930000000000001e-06 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556558+0000", 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0704e-05 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.556674+0000", 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011600899999999999 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.557082+0000", 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00040883 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.563627+0000", 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0065448629999999997 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.563641+0000", 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4031e-05 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.675597+0000", 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.111955234 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.675632+0000", 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5704999999999998e-05 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.675673+0000", 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0448999999999998e-05 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:254064 8.11 8:8bf8e7d1:::benchmark_data_smithi003_110242_object123147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:44.559927+0000", 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 22.715912165999999, 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.11590102300000001, 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:07.415 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 254064 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.559927+0000", 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.559927+0000", 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.559929+0000", 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9099999999999999e-06 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.559937+0000", 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1040000000000007e-06 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.559952+0000", 2024-03-21T14:13:07.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5355000000000001e-05 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.560012+0000", 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9933999999999999e-05 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.560067+0000", 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4500999999999999e-05 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.560252+0000", 2024-03-21T14:13:07.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018540700000000001 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.560673+0000", 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00042140199999999997 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.563777+0000", 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003103532 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.564324+0000", 2024-03-21T14:13:07.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000547202 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.675765+0000", 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.111440658 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.675793+0000", 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8663999999999999e-05 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:44.675828+0000", 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4353999999999997e-05 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "rep_scrubmap(6.10 e224 from shard 0)", 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:13:06.269009+0000", 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 1.006830283, 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027333700000000002, 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269009+0000", 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269009+0000", 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269010+0000", 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5060000000000001e-06 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269019+0000", 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.867e-06 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269041+0000", 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2365999999999999e-05 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269054+0000", 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2417e-05 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269171+0000", 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000117347 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269200+0000", 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8866000000000001e-05 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:13:06.269282+0000", 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1967999999999999e-05 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.424 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:07.425 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:07.425 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:13:07.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.426 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:07.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:07.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.551 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:07.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:07.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:07.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.677 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:07.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:07.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:07.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:07.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:07.811 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_2' created 2024-03-21T14:13:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:13:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:13:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2999540175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T14:13:07.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T14:13:07.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:07.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:13:07.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:13:07.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2999540175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T14:13:07.822 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:07 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T14:13:07.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:07.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:07.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:07.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:08.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:08.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:07 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:13:08.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:07 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:13:08.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2999540175' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T14:13:08.033 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:07 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-21T14:13:08.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:08.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:08.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:08.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:13:08.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:08.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:08.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:08.307 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:08.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:08.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:08.413 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:08.438 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:08.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:08.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:08.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:13:08.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:08.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:08.614 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-21T14:13:08.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:08.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.691 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:08.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:08.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:08.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:08.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[29576]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[29576]: osdmap e244: 8 total, 8 up, 8 in 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[40510]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-21T14:13:08.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:08 smithi003 ceph-mon[40510]: osdmap e244: 8 total, 8 up, 8 in 2024-03-21T14:13:08.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:08.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:09.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:09.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:09.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:09.165 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:08 smithi118 ceph-mon[33862]: pgmap v465: 132 pgs: 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-21T14:13:09.165 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:08 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:13:09.165 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:08 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:13:09.165 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:08 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-21T14:13:09.165 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:08 smithi118 ceph-mon[33862]: osdmap e244: 8 total, 8 up, 8 in 2024-03-21T14:13:09.177 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.201 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:09.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:13:09.204 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:13:09.205 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:13:09.205 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:13:09.205 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:13:09.205 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:13:09.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:09.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:09.428 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:09.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.453 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:09.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:09.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:09.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:09.679 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:09.703 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:09.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:09.831 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:09.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:09.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:09.950 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:13:09.950 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:13:09.950 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-03-21T14:13:09.951 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:13:09.956 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:09.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:13:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:13:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[29576]: osdmap e245: 8 total, 8 up, 8 in 2024-03-21T14:13:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:13:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:13:10.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:13:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:13:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[40510]: osdmap e245: 8 total, 8 up, 8 in 2024-03-21T14:13:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:13:10.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:09 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:13:10.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:10.078 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:09 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:13:10.079 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:09 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:13:10.079 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:09 smithi118 ceph-mon[33862]: osdmap e245: 8 total, 8 up, 8 in 2024-03-21T14:13:10.079 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:09 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:13:10.079 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:09 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:13:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:10.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:10.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:10.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:10.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:13:10.374 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:13:10.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.4 'trim stale osdmaps' 2024-03-21T14:13:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:10.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:10.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.462 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:10.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:10.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:10.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T14:13:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:10.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.715 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:10.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:10.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:10.822 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:13:10.836 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:10.842 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: pgmap v468: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: osdmap e246: 8 total, 8 up, 8 in 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1236540732' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: 3.1d deep-scrub starts 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: 3.1d deep-scrub ok 2024-03-21T14:13:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:10 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:13:10.944 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:10.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:10.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:11.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:11.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:11.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:11.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: pgmap v468: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:13:11.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:13:11.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: osdmap e246: 8 total, 8 up, 8 in 2024-03-21T14:13:11.180 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1236540732' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: 3.1d deep-scrub starts 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: 3.1d deep-scrub ok 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: pgmap v468: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: osdmap e246: 8 total, 8 up, 8 in 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1236540732' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: 3.1d deep-scrub starts 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: 3.1d deep-scrub ok 2024-03-21T14:13:11.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:10 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:13:11.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:11.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:11.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:11.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:11.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:11.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:11.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:11.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:11.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:11.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.724 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:11.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:11.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:11.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:11 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:13:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:11 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:13:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:11 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:13:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:11 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:11 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1236540732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:13:11.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:11 smithi118 ceph-mon[33862]: osdmap e247: 8 total, 8 up, 8 in 2024-03-21T14:13:11.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:11.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:11.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:11.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:12.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:12.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:12.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:12.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1236540732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[29576]: osdmap e247: 8 total, 8 up, 8 in 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:12.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1236540732' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:13:12.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:11 smithi003 ceph-mon[40510]: osdmap e247: 8 total, 8 up, 8 in 2024-03-21T14:13:12.202 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:12.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.229 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:12.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:12.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:12.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:12.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:12.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:12.465 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:13:12.465 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:13:12.465 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:13:12.465 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:13:12.465 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:13:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:12.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:12.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:12.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:12.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:12.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:12.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:12.835 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.859 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:12.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:12.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:12.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:12.985 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:12.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:13.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T14:13:13.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:13.090 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:12 smithi003 ceph-mon[40510]: pgmap v471: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:13:13.090 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:12 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:13:13.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:12 smithi003 ceph-mon[29576]: pgmap v471: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:13:13.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:12 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:13:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:13.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:13.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:12 smithi118 ceph-mon[33862]: pgmap v471: 148 pgs: 12 unknown, 136 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:13:13.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:12 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:13:13.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:13.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:13.278 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:13.296 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:13.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:13.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:13.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:13.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:13.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:13:13.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:13.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:13.720 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.745 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:13.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:13.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:13.871 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:13.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:13.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:14.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.042 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:14.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:14.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:14.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:14.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:14.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:14.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:14.394 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:14.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:14.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:14.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:14.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:14.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:14.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.672 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:14.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:14.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:14.785 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:13:14.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.797 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:14.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:14.805 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:13:14.824 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:13:14.842 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-21T14:13:14.848 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:13:14.871 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:13:14.892 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:13:14.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:14.909 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:13:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:13:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:13:14.909 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154023 7.7 7:ec8384af:::benchmark_data_smithi003_80362_object33006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 384.28624925499997, 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.14729518, 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.910 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154023 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621105+0000", 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.155e-06 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621110+0000", 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5790000000000003e-06 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621119+0000", 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1670000000000005e-06 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621125+0000", 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9370000000000001e-06 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665294+0000", 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044168488999999998 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665370+0000", 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6055000000000003e-05 2024-03-21T14:13:14.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665441+0000", 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1031000000000005e-05 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.668869+0000", 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034284250000000001 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.767406+0000", 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.098536532999999996 2024-03-21T14:13:14.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768291+0000", 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00088539800000000002 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768347+0000", 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5769000000000001e-05 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768399+0000", 2024-03-21T14:13:14.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1641999999999998e-05 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154716 7.7 7:eb3d6bd8:::benchmark_data_smithi003_80362_object33699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 383.13351921899999, 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.135279182, 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.916 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154716 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773834+0000", 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0399999999999995e-07 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773838+0000", 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9910000000000004e-06 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773846+0000", 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2209999999999997e-06 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773853+0000", 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3890000000000002e-06 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814039+0000", 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040186573000000003 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814159+0000", 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012002600000000001 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814282+0000", 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000123091 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817930+0000", 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036476630000000002 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831508+0000", 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013578365 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.908985+0000", 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077476415000000007 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909037+0000", 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1929999999999999e-05 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909113+0000", 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5814000000000003e-05 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.921 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154717 7.7 7:e048d7ab:::benchmark_data_smithi003_80362_object33700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 383.13325852399998, 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13536177999999999, 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154717 2024-03-21T14:13:14.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.923 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.923 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:13:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:14 smithi003 ceph-mon[29576]: pgmap v472: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-21T14:13:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2147009289' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:13:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4170054602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:14 smithi003 ceph-mon[40510]: pgmap v472: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-21T14:13:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:14 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2147009289' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:13:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:14 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4170054602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774095+0000", 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774098+0000", 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1640000000000001e-06 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774106+0000", 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3059999999999998e-06 2024-03-21T14:13:14.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774112+0000", 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5740000000000003e-06 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816429+0000", 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042316843 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816497+0000", 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8421000000000001e-05 2024-03-21T14:13:14.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816574+0000", 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6460999999999999e-05 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820900+0000", 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043267039999999998 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831554+0000", 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010653673000000001 2024-03-21T14:13:14.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909389+0000", 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077835369000000001 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909424+0000", 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4275e-05 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909456+0000", 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2254e-05 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154718 7.7 7:eaddd76b:::benchmark_data_smithi003_80362_object33701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 383.13251358500003, 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13475282299999999, 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.930 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154718 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.931 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774840+0000", 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774844+0000", 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.783e-06 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774853+0000", 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5459999999999997e-06 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774859+0000", 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2639999999999997e-06 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817968+0000", 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043108377000000003 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818046+0000", 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8118e-05 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818119+0000", 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2712000000000004e-05 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820930+0000", 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028118449999999999 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909502+0000", 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.088571516000000003 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909536+0000", 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3813999999999998e-05 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909563+0000", 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7297e-05 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909592+0000", 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8881e-05 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221486 7.7 7:ee81363c:::benchmark_data_smithi003_80362_object100469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.812938766, 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1032354099999999, 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.936 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221486 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094415+0000", 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.164e-06 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094423+0000", 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3129999999999999e-06 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094436+0000", 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2646999999999999e-05 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094445+0000", 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7069999999999998e-06 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094500+0000", 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4882000000000001e-05 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094509+0000", 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9369999999999993e-06 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192346+0000", 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0978373850000001 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192463+0000", 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011746799999999999 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192545+0000", 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2050000000000002e-05 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197520+0000", 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049742279999999998 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197578+0000", 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8791e-05 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197589+0000", 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0293e-05 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197620+0000", 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1065000000000002e-05 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197649+0000", 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479999999999999e-05 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221488 7.7 7:e6035441:::benchmark_data_smithi003_80362_object100471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.81166040300002, 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.107609375, 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221488 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095693+0000", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5999999999999991e-07 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095698+0000", 2024-03-21T14:13:14.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0239999999999997e-06 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095711+0000", 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2445e-05 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095719+0000", 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0730000000000005e-06 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095770+0000", 2024-03-21T14:13:14.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1650000000000002e-05 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095777+0000", 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2169999999999999e-06 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194772+0000", 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.09899547 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194854+0000", 2024-03-21T14:13:14.946 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2081000000000002e-05 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194929+0000", 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5067999999999993e-05 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199113+0000", 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041835680000000004 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.947 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203213+0000", 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040997869999999997 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203248+0000", 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5432999999999998e-05 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203277+0000", 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8952e-05 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203302+0000", 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4647e-05 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221491 7.7 7:eaaedede:::benchmark_data_smithi003_80362_object100474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.80885055099998, 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1086137819999999, 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.949 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221491 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098503+0000", 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3090000000000001e-06 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098508+0000", 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1899999999999997e-06 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098518+0000", 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0258000000000001e-05 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098525+0000", 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7229999999999998e-06 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098580+0000", 2024-03-21T14:13:14.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5015000000000001e-05 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098590+0000", 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.061e-05 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196184+0000", 2024-03-21T14:13:14.953 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0975940799999999 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196262+0000", 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7281999999999996e-05 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196328+0000", 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6224999999999997e-05 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199116+0000", 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027880639999999998 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203355+0000", 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042392979999999999 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207064+0000", 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037090000000000001 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207093+0000", 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8351999999999999e-05 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207116+0000", 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3376000000000001e-05 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221516 7.7 7:ee74cf53:::benchmark_data_smithi003_80362_object100499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:13:14.956 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.78938259699999, 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.095088184, 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221516 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117972+0000", 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6029999999999999e-06 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117979+0000", 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6650000000000003e-06 2024-03-21T14:13:14.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117994+0000", 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5021999999999999e-05 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118002+0000", 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6129999999999998e-06 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118062+0000", 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9768000000000001e-05 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118072+0000", 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0028e-05 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197714+0000", 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0796421430000001 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197781+0000", 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6649000000000005e-05 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197849+0000", 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8150000000000003e-05 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203310+0000", 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054610889999999997 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203389+0000", 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8960000000000003e-05 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.212981+0000", 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095917339999999993 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213031+0000", 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0633999999999997e-05 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213058+0000", 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7126000000000001e-05 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.962 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221558 7.7 7:e23945db:::benchmark_data_smithi003_80362_object100541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.75208279399999, 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0579410659999999, 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221558 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.963 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155271+0000", 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-03-21T14:13:14.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155279+0000", 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9899999999999997e-06 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155295+0000", 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5366000000000002e-05 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155303+0000", 2024-03-21T14:13:14.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9030000000000004e-06 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155469+0000", 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016613000000000001 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155488+0000", 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9151e-05 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199159+0000", 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0436712960000001 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199258+0000", 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9086999999999995e-05 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199325+0000", 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6824999999999998e-05 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203317+0000", 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003992125 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213108+0000", 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009791213 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213160+0000", 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1712000000000002e-05 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213189+0000", 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8915999999999999e-05 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213211+0000", 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1837000000000001e-05 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221566 7.7 7:e351aeff:::benchmark_data_smithi003_80362_object100549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.969 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.74224062399998, 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0550122820000001, 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221566 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:13:14.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165113+0000", 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1999999999999999e-06 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165118+0000", 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3710000000000002e-06 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165126+0000", 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1980000000000001e-06 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165131+0000", 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1229999999999999e-06 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165172+0000", 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1094999999999999e-05 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165178+0000", 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6659999999999996e-06 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200612+0000", 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.035434781 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200723+0000", 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011056 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200803+0000", 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9759000000000007e-05 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203324+0000", 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002521129 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213257+0000", 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099332840000000006 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220034+0000", 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067764390000000004 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220082+0000", 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8179000000000001e-05 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220124+0000", 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2497999999999999e-05 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221598 7.7 7:e857fd29:::benchmark_data_smithi003_80362_object100581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.67290941300001, 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99124171000000005, 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.976 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221598 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:14.979 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:14.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234445+0000", 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5460000000000001e-06 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234452+0000", 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6590000000000007e-06 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234470+0000", 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7876999999999998e-05 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234480+0000", 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8770000000000005e-06 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234548+0000", 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7343999999999998e-05 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234560+0000", 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2707999999999999e-05 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.201983+0000", 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96742289599999998 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.202047+0000", 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3736000000000001e-05 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.202116+0000", 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.902e-05 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203398+0000", 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001282327 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220150+0000", 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016752229 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225577+0000", 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054261459999999997 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225637+0000", 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0489999999999999e-05 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225685+0000", 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7855000000000002e-05 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.984 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221605 7.7 7:e86450e5:::benchmark_data_smithi003_80362_object100588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.63738465500001, 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96634132500000003, 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221605 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.985 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269969+0000", 2024-03-21T14:13:14.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2470000000000001e-06 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269975+0000", 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7320000000000003e-06 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269990+0000", 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4745e-05 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269999+0000", 2024-03-21T14:13:14.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0990000000000005e-06 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.270076+0000", 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7044000000000003e-05 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.270085+0000", 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.55e-06 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203422+0000", 2024-03-21T14:13:14.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.93333645799999998 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203489+0000", 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7290000000000001e-05 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203555+0000", 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6123000000000004e-05 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207125+0000", 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035693439999999999 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225732+0000", 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018607716999999999 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236215+0000", 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010482131 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236265+0000", 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0158000000000003e-05 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236309+0000", 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4687000000000001e-05 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221618 7.7 7:e934ecf7:::benchmark_data_smithi003_80362_object100601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.62417677500002, 2024-03-21T14:13:14.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95328595999999999, 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221618 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283177+0000", 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283183+0000", 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5829999999999997e-06 2024-03-21T14:13:14.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283199+0000", 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5786000000000001e-05 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283207+0000", 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6370000000000005e-06 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283266+0000", 2024-03-21T14:13:14.994 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8603000000000001e-05 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283274+0000", 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4249999999999998e-06 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204661+0000", 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.92138625600000001 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204729+0000", 2024-03-21T14:13:14.995 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7949999999999998e-05 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204796+0000", 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7284000000000002e-05 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209602+0000", 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00480606 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227784+0000", 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018181953000000001 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236361+0000", 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085767780000000002 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236418+0000", 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7538999999999999e-05 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:14.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236462+0000", 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3683000000000002e-05 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221625 7.7 7:e490a489:::benchmark_data_smithi003_80362_object100608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.61931657600002, 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94858037900000003, 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:14.998 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221625 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:14.999 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288038+0000", 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288044+0000", 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0360000000000003e-06 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288060+0000", 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6498e-05 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288068+0000", 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079999999999998e-06 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288124+0000", 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6141999999999999e-05 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288133+0000", 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4350000000000003e-06 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.205877+0000", 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91774332000000003 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.205944+0000", 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7387999999999999e-05 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.206008+0000", 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4078000000000006e-05 2024-03-21T14:13:15.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209608+0000", 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035997080000000001 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227901+0000", 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018292940000000001 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236522+0000", 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086215389999999992 2024-03-21T14:13:15.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236574+0000", 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1910000000000003e-05 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236616+0000", 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2265999999999998e-05 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.004 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221680 7.7 7:ec92397d:::benchmark_data_smithi003_80362_object100663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.55522515600001, 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.884639709, 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221680 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352130+0000", 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0250000000000001e-06 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352138+0000", 2024-03-21T14:13:15.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5930000000000006e-06 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352157+0000", 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9318e-05 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352168+0000", 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0098000000000001e-05 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352233+0000", 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5362000000000002e-05 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352244+0000", 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0985000000000001e-05 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207147+0000", 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85490348999999999 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207222+0000", 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4796999999999996e-05 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207287+0000", 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4978e-05 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.212923+0000", 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056357789999999996 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227945+0000", 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015022064 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236673+0000", 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087278890000000008 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236725+0000", 2024-03-21T14:13:15.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2469999999999997e-05 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236767+0000", 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1860999999999998e-05 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221689 7.7 7:e59419c3:::benchmark_data_smithi003_80362_object100672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.54174304600002, 2024-03-21T14:13:15.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87131467699999998, 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221689 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365611+0000", 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2020000000000001e-06 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365616+0000", 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3789999999999997e-06 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365628+0000", 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1807000000000001e-05 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365635+0000", 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0899999999999999e-06 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365698+0000", 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2691e-05 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365709+0000", 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0706999999999999e-05 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208374+0000", 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84266594399999994 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208442+0000", 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7799999999999995e-05 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208504+0000", 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1441999999999999e-05 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213123+0000", 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046188710000000001 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227986+0000", 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01486326 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236829+0000", 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088433829999999998 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236880+0000", 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0977999999999999e-05 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236924+0000", 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4122999999999998e-05 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.017 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221705 7.7 7:e134962a:::benchmark_data_smithi003_80362_object100688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.52104720599999, 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85076709500000003, 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221705 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.018 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386307+0000", 2024-03-21T14:13:15.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.336e-06 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386313+0000", 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8209999999999998e-06 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386324+0000", 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0895e-05 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386332+0000", 2024-03-21T14:13:15.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2740000000000007e-06 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386407+0000", 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5479e-05 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386417+0000", 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3039999999999994e-06 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209643+0000", 2024-03-21T14:13:15.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82322670899999995 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209715+0000", 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1707999999999999e-05 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209777+0000", 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2005e-05 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213129+0000", 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033520189999999999 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.228015+0000", 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014885631 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236988+0000", 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089736309999999993 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237035+0000", 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7126999999999998e-05 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237073+0000", 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7156000000000001e-05 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221707 7.7 7:ee1b885c:::benchmark_data_smithi003_80362_object100690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.51872463799998, 2024-03-21T14:13:15.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84853804399999999, 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221707 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.025 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:15.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:13:15.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388630+0000", 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4929999999999999e-06 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388636+0000", 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.207e-06 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388650+0000", 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4283e-05 2024-03-21T14:13:15.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388659+0000", 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5290000000000008e-06 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388717+0000", 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8374e-05 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388729+0000", 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1666e-05 2024-03-21T14:13:15.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.210963+0000", 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82223472099999995 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.211082+0000", 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011889500000000001 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.211202+0000", 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119864 2024-03-21T14:13:15.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213270+0000", 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020684119999999999 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.228042+0000", 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014771993000000001 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237112+0000", 2024-03-21T14:13:15.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090695329999999994 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237143+0000", 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0642999999999998e-05 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237166+0000", 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3431e-05 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221710 7.7 7:e7a06586:::benchmark_data_smithi003_80362_object100693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.51337494799998, 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84331796999999997, 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221710 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:15.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393979+0000", 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.17e-06 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393985+0000", 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8660000000000002e-06 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393997+0000", 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2218999999999999e-05 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394006+0000", 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0899999999999994e-06 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394056+0000", 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0389000000000002e-05 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394063+0000", 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7619999999999998e-06 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213298+0000", 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81923456500000003 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213370+0000", 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1940999999999994e-05 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213432+0000", 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2206999999999996e-05 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.218798+0000", 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005365791 2024-03-21T14:13:15.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.231526+0000", 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012728385 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237210+0000", 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056836960000000002 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237258+0000", 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8300000000000002e-05 2024-03-21T14:13:15.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237296+0000", 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7589000000000002e-05 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221712 7.7 7:efee874f:::benchmark_data_smithi003_80362_object100695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 306.508552882, 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.83858521399999997, 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:13:15.038 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221712 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:13:15.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398801+0000", 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398810+0000", 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7399999999999993e-06 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398823+0000", 2024-03-21T14:13:15.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3444e-05 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398832+0000", 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0459999999999994e-06 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398880+0000", 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7713000000000002e-05 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398886+0000", 2024-03-21T14:13:15.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2949999999999999e-06 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215610+0000", 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81672409700000004 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215726+0000", 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115151 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215844+0000", 2024-03-21T14:13:15.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011856100000000001 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.219013+0000", 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003168424 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.231671+0000", 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012658584000000001 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237340+0000", 2024-03-21T14:13:15.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056683879999999999 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237366+0000", 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6338000000000001e-05 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237385+0000", 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9133000000000001e-05 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:13:15.044 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:13:15.045 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:13:15.045 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:14 smithi118 ceph-mon[33862]: pgmap v472: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-21T14:13:15.045 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:14 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2147009289' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:13:15.045 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:14 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4170054602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:15.045 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:13:15.046 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:13:15.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:15.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:15.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:15.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:15.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:15.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:15.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:15.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:15.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:15.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:15.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:15.859 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:15.915 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:13:15.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:15.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:15.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:16.086 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.115 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:16.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:16.155 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:16.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:15 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2147009289' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:13:16.161 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:15 smithi003 ceph-mon[29576]: osdmap e248: 8 total, 8 up, 8 in 2024-03-21T14:13:16.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:15 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2147009289' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:13:16.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:15 smithi003 ceph-mon[40510]: osdmap e248: 8 total, 8 up, 8 in 2024-03-21T14:13:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:15 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2147009289' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:13:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:15 smithi118 ceph-mon[33862]: osdmap e248: 8 total, 8 up, 8 in 2024-03-21T14:13:16.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:16.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:16.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:16.342 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:16.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.367 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:16.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:16.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:16.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:16.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:16.618 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:16.714 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:13:16.714 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:13:16.714 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:13:16.714 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:13:16.714 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:13:16.715 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:13:16.715 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:13:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:13:16.720 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:16.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:16.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: pgmap v474: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]: dispatch 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]: dispatch 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]: dispatch 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]: dispatch 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]: dispatch 2024-03-21T14:13:16.868 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]: dispatch 2024-03-21T14:13:16.869 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]: dispatch 2024-03-21T14:13:16.869 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]: dispatch 2024-03-21T14:13:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:16.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:16.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:16.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:16.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: pgmap v474: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]: dispatch 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]: dispatch 2024-03-21T14:13:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: pgmap v474: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]: dispatch 2024-03-21T14:13:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]: dispatch 2024-03-21T14:13:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]: dispatch 2024-03-21T14:13:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]: dispatch 2024-03-21T14:13:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]: dispatch 2024-03-21T14:13:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]: dispatch 2024-03-21T14:13:17.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]: dispatch 2024-03-21T14:13:17.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:17.120 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:17.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:17.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:13:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:13:17.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:17.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:17.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:17.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:17.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:17.479 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:17.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.503 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:17.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:17.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:17.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:17.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:17.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:17.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:17.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:17.780 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:13:17.780 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:13:17.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:17.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:17.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:13:17.901 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:13:17.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.5 'trim stale osdmaps' 2024-03-21T14:13:17.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:18.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:18.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:18.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.136 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:18.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:18.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:18.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]': finished 2024-03-21T14:13:18.156 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[29576]: osdmap e249: 8 total, 8 up, 8 in 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]': finished 2024-03-21T14:13:18.157 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:17 smithi003 ceph-mon[40510]: osdmap e249: 8 total, 8 up, 8 in 2024-03-21T14:13:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [2, 0]}]': finished 2024-03-21T14:13:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.4", "id": [5, 0, 3, 6]}]': finished 2024-03-21T14:13:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.a", "id": [2, 1]}]': finished 2024-03-21T14:13:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.e", "id": [2, 6]}]': finished 2024-03-21T14:13:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:17 smithi118 ceph-mon[33862]: osdmap e249: 8 total, 8 up, 8 in 2024-03-21T14:13:18.183 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-21T14:13:18.185 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T14:13:18.185 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T14:13:18.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:13:18.234 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-03-21T14:13:18.234 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T14:13:18.234 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_135814 2024-03-21T14:13:18.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:13:18.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-03-21T14:13:18.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:18.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:18.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:18.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:18.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:13:18.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.553 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:18.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:18.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:18.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:18.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.840 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:18.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:18.855 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:18.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:18.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:18.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:18.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:18 smithi003 ceph-mon[29576]: pgmap v475: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:13:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:18 smithi003 ceph-mon[29576]: osdmap e250: 8 total, 8 up, 8 in 2024-03-21T14:13:19.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:18 smithi003 ceph-mon[40510]: pgmap v475: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:13:19.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:18 smithi003 ceph-mon[40510]: osdmap e250: 8 total, 8 up, 8 in 2024-03-21T14:13:19.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:19.129 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:19.130 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:18 smithi118 ceph-mon[33862]: pgmap v475: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:13:19.131 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:18 smithi118 ceph-mon[33862]: osdmap e250: 8 total, 8 up, 8 in 2024-03-21T14:13:19.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:19.234 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3254 3238 202.363 202.375 0.00418202 0.00492684 2024-03-21T14:13:19.258 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:19.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.402 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:19.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:19.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:19.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:19.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:19.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:19.784 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:19.810 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:19.817 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:19.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:19.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:19.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:20.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:20.092 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:20.194 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:20.230 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:20.234 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 15 6411 6396 199.86 197.375 0.00240856 0.00499242 2024-03-21T14:13:20.236 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:20.332 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:20.369 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:20.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:20.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:20.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:20.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:20.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:20.725 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:13:20.725 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:13:20.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:20.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:20.894 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:20.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:20.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:20.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:20.969 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:20 smithi003 ceph-mon[29576]: pgmap v478: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T14:13:20.969 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:20 smithi003 ceph-mon[40510]: pgmap v478: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T14:13:20.983 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:20 smithi118 ceph-mon[33862]: pgmap v478: 148 pgs: 148 active+clean; 582 KiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T14:13:21.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:21.061 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:21.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:21.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:21.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 9306 9290 193.526 180.875 0.00676201 0.00515865 2024-03-21T14:13:21.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:21.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:21.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:13:21.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:21.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:21.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:21.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:21.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:21.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:21.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:21.847 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:13:21.875 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:13:21.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:21.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:21.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:22.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:22.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:22.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:21 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2227937156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:21 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:22.204 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:22.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:22.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 12193 12177 190.248 180.438 0.00522269 0.00524804 2024-03-21T14:13:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2227937156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:21 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:21 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2227937156' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:21 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:22.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.341 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:22.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:22.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.479 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:22.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:22.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:22.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:22.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:22.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:22.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:22.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:22.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[29576]: pgmap v479: 148 pgs: 148 active+clean; 158 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 24 MiB/s wr, 383 op/s 2024-03-21T14:13:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[29576]: osdmap e251: 8 total, 8 up, 8 in 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[40510]: pgmap v479: 148 pgs: 148 active+clean; 158 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 24 MiB/s wr, 383 op/s 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[40510]: osdmap e251: 8 total, 8 up, 8 in 2024-03-21T14:13:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:22 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:13:23.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:23.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:23.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:22 smithi118 ceph-mon[33862]: pgmap v479: 148 pgs: 148 active+clean; 158 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 24 MiB/s wr, 383 op/s 2024-03-21T14:13:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:22 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:22 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:13:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:22 smithi118 ceph-mon[33862]: osdmap e251: 8 total, 8 up, 8 in 2024-03-21T14:13:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:22 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:13:23.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:23.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 14457 14441 180.494 141.5 0.00311392 0.00553272 2024-03-21T14:13:23.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:23.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:23.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:23.495 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:23.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:23.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:23.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:23.781 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:23.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:23.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:23.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:24.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:24.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:24.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:24.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:13:24.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:13:24.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:23 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:24.155 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:13:24.156 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:13:24.156 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.156 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.156 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.156 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:13:24.157 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:23 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:13:24.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:24.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:24.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 15 16768 16753 174.493 144.5 0.00729595 0.00572384 2024-03-21T14:13:24.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:24.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:24.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:24.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:24.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:24.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:24.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:24.607 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:24.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:24.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:24.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.745 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:24.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:24.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:13:24.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:24.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:24.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:24.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:24.954 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:13:25.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:25.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:25.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:25.084 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:13:25.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:25.131 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:13:25.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:24 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi118 to 3248M 2024-03-21T14:13:25.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:24 smithi003 ceph-mon[29576]: pgmap v481: 148 pgs: 148 active+clean; 651 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T14:13:25.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:24 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi118 to 3248M 2024-03-21T14:13:25.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:24 smithi003 ceph-mon[40510]: pgmap v481: 148 pgs: 148 active+clean; 651 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T14:13:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:25.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:25.177 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:13:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:24 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi118 to 3248M 2024-03-21T14:13:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:24 smithi118 ceph-mon[33862]: pgmap v481: 148 pgs: 148 active+clean; 651 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-21T14:13:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:13:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:13:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:13:25.202 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:13:25.203 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:13:25.218 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:13:25.235 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 18785 18769 167.564 126 0.00528483 0.00596145 2024-03-21T14:13:25.260 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:13:25.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:25.298 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:13:25.308 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:25.341 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:13:25.346 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:13:25.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:25.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.448 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:25.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:25.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:13:25.549 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:25.586 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:25.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:25.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:25.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:25.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:25.874 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:25.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:25.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2370264275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:26.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:25 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2370264275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:26.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:26.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:26.118 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:26.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:25 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2370264275' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:13:26.220 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:13:26.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.6 'trim stale osdmaps' 2024-03-21T14:13:26.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 21275 21259 166.059 155.625 0.00518545 0.00601663 2024-03-21T14:13:26.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:26.292 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:26.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:26.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:26.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:26.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:26.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:26.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:26.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:26.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:26.734 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:26.753 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:26.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:13:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:13:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:13:26.784 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:26.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:26.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:26.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:26.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:27.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:27.119 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:27.157 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:27.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:26 smithi003 ceph-mon[29576]: pgmap v482: 148 pgs: 148 active+clean; 651 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 646 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T14:13:27.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:26 smithi003 ceph-mon[40510]: pgmap v482: 148 pgs: 148 active+clean; 651 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 646 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T14:13:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:26 smithi118 ceph-mon[33862]: pgmap v482: 148 pgs: 148 active+clean; 651 MiB data, 4.4 GiB used, 711 GiB / 715 GiB avail; 646 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T14:13:27.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:13:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:13:27.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 15 23652 23637 164.121 148.625 0.00776732 0.00608725 2024-03-21T14:13:27.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:27.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:27.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:27.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:27.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:27.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:27.572 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:27.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:27.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:27.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:27.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:27.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.852 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:27.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:27.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:27.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:27.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:27.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:28.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:28.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:28.181 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:28.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:28.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 26141 26125 163.257 155.5 0.00445649 0.00611908 2024-03-21T14:13:28.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:28.278 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:28.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:28.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:28.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:28.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:28.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:28.682 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:13:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:28.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:28.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:28.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:28.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:28.979 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:28.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:28 smithi003 ceph-mon[29576]: pgmap v483: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-03-21T14:13:29.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:28 smithi003 ceph-mon[40510]: pgmap v483: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-03-21T14:13:29.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:29.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:29.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:28 smithi118 ceph-mon[33862]: pgmap v483: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 156 MiB/s wr, 2.50k op/s 2024-03-21T14:13:29.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:29.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 28410 28394 161.307 141.812 0.00528829 0.00619337 2024-03-21T14:13:29.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:29.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:29.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:29.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:29.498 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:29.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:29.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:29.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:29.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:29.778 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:29.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:29.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:29.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:29.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:29.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:30.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:30.091 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:30.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:30.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:30.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 30444 30428 158.458 127.125 0.0067587 0.00630517 2024-03-21T14:13:30.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.371 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:30.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:30.473 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:30.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:30.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:30.650 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:30.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:30.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:30.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:30.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:30.928 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:30.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:30.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:31.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:31.065 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:31.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:31.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:30 smithi118 ceph-mon[33862]: pgmap v484: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-21T14:13:31.203 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:31.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:31.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 32704 32688 157.133 141.25 0.00743039 0.00635862 2024-03-21T14:13:31.244 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:30 smithi003 ceph-mon[40510]: pgmap v484: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-21T14:13:31.244 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:30 smithi003 ceph-mon[29576]: pgmap v484: 148 pgs: 148 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-03-21T14:13:31.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:31.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:31.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:31.477 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:31.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:31.579 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:31.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:31.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.752 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:31.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:31.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:31.902 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:31.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:31.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:31.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:31.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:32.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:32.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:32.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:32.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.200 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:32.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:32.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 34764 34748 155.105 128.75 0.00383403 0.00644209 2024-03-21T14:13:32.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:32.337 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:32.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:32.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:32.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:32.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:32.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:32.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.755 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:32.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:13:32.801 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:13:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:13:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:13:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:13:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:13:32.802 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:13:32.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:32.892 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:32.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:32.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:32.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.037 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:33.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:33.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:32 smithi118 ceph-mon[33862]: pgmap v485: 148 pgs: 148 active+clean; 1.5 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 141 MiB/s wr, 2.25k op/s 2024-03-21T14:13:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:33.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:33.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 36611 36595 152.46 115.438 0.0117402 0.00655202 2024-03-21T14:13:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:32 smithi003 ceph-mon[29576]: pgmap v485: 148 pgs: 148 active+clean; 1.5 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 141 MiB/s wr, 2.25k op/s 2024-03-21T14:13:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:32 smithi003 ceph-mon[40510]: pgmap v485: 148 pgs: 148 active+clean; 1.5 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 141 MiB/s wr, 2.25k op/s 2024-03-21T14:13:33.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:33.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:33.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:33.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:33.469 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:33.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:33.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:33.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:33.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:13:33.740 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:13:33.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.7 'trim stale osdmaps' 2024-03-21T14:13:33.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.752 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:33.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:33.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:33.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:33.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:33.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:33.968 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:13:34.006 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.038 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:34.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:34.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:34.172 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:34.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 38491 38475 150.274 117.5 0.00925337 0.00664922 2024-03-21T14:13:34.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:33 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2319935028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:13:34.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:33 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:13:34.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:33 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2319935028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:13:34.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:33 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:13:34.238 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:13:34.238 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:13:34.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.238 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25232.0:38483 9.9 e252/244 9:99b040f6:::benchmark_data_smithi003_135814_object38482:head v 252'2438, mlcod=252'2435)", 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:13:34.235679+0000", 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001277584, 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013093239999999999, 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235679+0000", 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:34.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235679+0000", 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235688+0000", 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7010000000000008e-06 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235766+0000", 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7169000000000006e-05 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235777+0000", 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1015e-05 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235783+0000", 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9900000000000002e-06 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235829+0000", 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6285000000000002e-05 2024-03-21T14:13:34.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:34.235858+0000", 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8966e-05 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 1 2024-03-21T14:13:34.242 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:13:34.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:34.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:34.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:34.486 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:33 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2319935028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:13:34.486 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:33 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:13:34.521 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:34.579 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:34.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:34.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:34.766 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:34.803 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:34.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:34.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:34.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:35.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:35.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:35.181 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:35.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.206 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:35.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:35.238 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 40347 40331 148.257 116 0.00305993 0.00670828 2024-03-21T14:13:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[29576]: pgmap v486: 148 pgs: 148 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 883 B/s rd, 144 MiB/s wr, 2.30k op/s 2024-03-21T14:13:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:13:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[29576]: osdmap e252: 8 total, 8 up, 8 in 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[40510]: pgmap v486: 148 pgs: 148 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 883 B/s rd, 144 MiB/s wr, 2.30k op/s 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[40510]: osdmap e252: 8 total, 8 up, 8 in 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:13:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:34 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:13:35.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:35.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:35.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:34 smithi118 ceph-mon[33862]: pgmap v486: 148 pgs: 148 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 883 B/s rd, 144 MiB/s wr, 2.30k op/s 2024-03-21T14:13:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:34 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:13:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:13:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:34 smithi118 ceph-mon[33862]: osdmap e252: 8 total, 8 up, 8 in 2024-03-21T14:13:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:34 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:13:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:34 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:13:35.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:35.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:35.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:35.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:35.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:35.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:35.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:35.756 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:35.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:35.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:35.881 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:35.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:35.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:36.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:36.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:36.161 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:36.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:13:36.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:13:36.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:13:36.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:35 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:13:36.236 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 41886 41870 145.364 96.1875 0.00853424 0.00687404 2024-03-21T14:13:36.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.297 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:36.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:36.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:35 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:13:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:35 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:13:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:35 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:35 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:35 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:13:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:35 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:13:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:36.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:36.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:36.581 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:36.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:36.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:13:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:13:36.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:36.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:36.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:36.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:36.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:36.998 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:36.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:37.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:37.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:37.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:13:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: pgmap v488: 148 pgs: 148 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.31k op/s 2024-03-21T14:13:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: osdmap e253: 8 total, 8 up, 8 in 2024-03-21T14:13:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:13:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:13:37.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:13:37.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:13:37.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:13:37.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:13:37.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:13:37.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:36 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:13:37.237 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 43445 43429 142.841 97.4375 0.00241581 0.00698777 2024-03-21T14:13:37.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: pgmap v488: 148 pgs: 148 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.31k op/s 2024-03-21T14:13:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: osdmap e253: 8 total, 8 up, 8 in 2024-03-21T14:13:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:13:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:13:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:13:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: pgmap v488: 148 pgs: 148 active+clean; 2.0 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.31k op/s 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: osdmap e253: 8 total, 8 up, 8 in 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:13:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:13:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:13:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:13:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:13:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:13:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:36 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:13:37.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:37.382 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:37.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:37.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:37.558 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:37.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.701 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:37.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:37.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.840 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:37.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:37.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:37.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:37.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:37.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:38.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:38.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:38.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:38.237 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:13:38.238110+0000 min lat: 0.00113893 max lat: 0.152179 avg lat: 0.00705661 2024-03-21T14:13:38.237 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:13:38.237 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 45338 45322 141.614 118.312 0.0107066 0.00705661 2024-03-21T14:13:38.252 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:13:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:13:38.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:38.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:13:38.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:13:38.354 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:38.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:38.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:38.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:38.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:37 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:13:38.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:37 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:13:38.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:38.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.533 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:38.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:38.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:38.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:38.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:38.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:38.809 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:38.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:38.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:38.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:38.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:38.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:39.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.073 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:39.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:39.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:39.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:39.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:39.237 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 47033 47017 139.915 105.938 0.0100947 0.00714252 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: pgmap v490: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 158 MiB/s wr, 2.53k op/s 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:13:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: pgmap v490: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 158 MiB/s wr, 2.53k op/s 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:13:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:38 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:13:39.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:39.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: pgmap v490: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 158 MiB/s wr, 2.53k op/s 2024-03-21T14:13:39.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:13:39.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:13:39.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:13:39.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:13:39.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:13:39.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:13:39.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:13:39.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:38 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:13:39.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:39.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:39.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:39.483 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:39.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:39.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:39.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:39.759 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:39.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:39.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:39.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:39.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:40.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:40.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.038 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:40.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:40.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:40.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:40.239 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 15 48850 48835 138.709 113.625 0.0162541 0.00720376 2024-03-21T14:13:40.260 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:13:40.260 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:13:40.260 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: osdmap e254: 8 total, 8 up, 8 in 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:13:40.261 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:13:40.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:13:40.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:13:40.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:40.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:40.262 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: osdmap e254: 8 total, 8 up, 8 in 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:13:40.263 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:40 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:13:40.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:40.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:40.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: osdmap e254: 8 total, 8 up, 8 in 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:13:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:13:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:13:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:13:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:13:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:13:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:40 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:13:40.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:40.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:40.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:40.702 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:40.731 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:40.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:40.794 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:40.799 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:40.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:40.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:40.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:40.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:40.993 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:41.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:41.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:41.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:41.239 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 50591 50575 137.406 108.75 0.00578823 0.00727256 2024-03-21T14:13:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:41 smithi003 ceph-mon[29576]: pgmap v492: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-21T14:13:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:41 smithi003 ceph-mon[40510]: pgmap v492: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-21T14:13:41.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.316 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:41.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:41.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:41 smithi118 ceph-mon[33862]: pgmap v492: 148 pgs: 148 active+clean; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-21T14:13:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:41.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:41.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:41.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:41.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:41.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:41.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:41.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:41.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:41.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:42.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:42.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:42.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:42.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:42.239 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 52267 52251 136.045 104.75 0.00913594 0.00734558 2024-03-21T14:13:42.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:13:42.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[29576]: pgmap v493: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:13:42.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[40510]: pgmap v493: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-21T14:13:42.307 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:13:42.307 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:42 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:13:42.312 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:42.414 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:42 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:13:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:42 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:13:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:42 smithi118 ceph-mon[33862]: pgmap v493: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-21T14:13:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:42 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:13:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:42 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:13:42.446 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:42.548 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:42.583 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:42.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:42.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:42.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.719 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:42.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:42.820 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:42.857 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:42.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:42.995 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:42.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:43.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:43.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:43.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:13:43.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:13:43.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:13:43.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:43 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:13:43.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:43.240 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 53872 53856 134.615 100.312 0.00748711 0.00742367 2024-03-21T14:13:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.277 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:43.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:43.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:43.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:43.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:43 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:13:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:43 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:13:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:43 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:13:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:43 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:13:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:43 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:13:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:43 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:13:43.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:43.553 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:43.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:43.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:43.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:43.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:43.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:43.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:43.984 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:43.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:44.086 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:44.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:44.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:44.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: osdmap e255: 8 total, 8 up, 8 in 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: pgmap v495: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: osdmap e255: 8 total, 8 up, 8 in 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:13:44.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:13:44.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:13:44.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:13:44.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: pgmap v495: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-21T14:13:44.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:13:44.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:44 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:13:44.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:44.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:44.239 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 55451 55435 133.233 98.6875 0.0109234 0.00750019 2024-03-21T14:13:44.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:44.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:44.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:44.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: osdmap e255: 8 total, 8 up, 8 in 2024-03-21T14:13:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:13:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:13:44.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:13:44.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:13:44.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: pgmap v495: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.3 KiB/s rd, 148 MiB/s wr, 2.37k op/s 2024-03-21T14:13:44.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:13:44.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:44 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:13:44.489 INFO:teuthology.orchestra.run.smithi003.stderr:no valid command found; 10 closest matches: 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:0 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:1 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:2 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:abort 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:assert 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:bench [] [] [] [] 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs debug_inject_read_zeros 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs files list 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:bluefs stats 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:bluestore allocator dump block 2024-03-21T14:13:44.490 INFO:teuthology.orchestra.run.smithi003.stderr:Error EINVAL: invalid command 2024-03-21T14:13:44.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:44.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:44.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:44.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:44.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:44.772 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:44.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.808 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:44.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:44.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:44.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:44.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:44.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:45.024 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:45.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:45.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:45.137 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:13:45.184 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:13:45.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:45.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:13:45.240 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 57001 56985 131.884 96.875 0.00613671 0.00757593 2024-03-21T14:13:45.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:45.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:45.269 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:13:45.312 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:13:45.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:45.353 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:13:45.382 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:45.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:13:45.435 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:13:45.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:13:45.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:13:45.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:13:45.439 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:13:45.440 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:13:45.440 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:13:45.440 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:13:45.440 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:13:45.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:45.517 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:45.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:45.642 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:45.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:45.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:45 smithi003 ceph-mon[29576]: osdmap e256: 8 total, 8 up, 8 in 2024-03-21T14:13:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4089416634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:45 smithi003 ceph-mon[40510]: osdmap e256: 8 total, 8 up, 8 in 2024-03-21T14:13:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4089416634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:45.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:45.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:45.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:45.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:45 smithi118 ceph-mon[33862]: osdmap e256: 8 total, 8 up, 8 in 2024-03-21T14:13:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4089416634' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:46.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:46.060 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:46.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:46.198 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:46.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:46.240 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 58481 58465 130.477 92.5 0.00666086 0.00765914 2024-03-21T14:13:46.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:46.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:46.474 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:46.504 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:13:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:13:46.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:46.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:46.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:46.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: pgmap v497: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 794 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: osdmap e257: 8 total, 8 up, 8 in 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:13:46.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: pgmap v497: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 794 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: osdmap e257: 8 total, 8 up, 8 in 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:13:46.875 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:13:46.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:46.876 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:46 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: pgmap v497: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 794 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: osdmap e257: 8 total, 8 up, 8 in 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:13:46.901 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:46.902 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:46.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:46.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:47.006 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:47.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:47.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:47.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:47.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:47.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:47.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:13:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:13:47.240 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 15 59979 59964 129.208 93.6875 0.00742995 0.00773417 2024-03-21T14:13:47.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.305 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:47.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:47.406 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:47.442 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:47.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:47.578 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:47.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:47.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:47.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:47.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:47.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:47.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:47.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[29576]: osdmap e258: 8 total, 8 up, 8 in 2024-03-21T14:13:47.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:47.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:13:47.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:13:47.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:47.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[40510]: osdmap e258: 8 total, 8 up, 8 in 2024-03-21T14:13:47.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:47.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:13:47.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:47 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:13:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:13:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:47 smithi118 ceph-mon[33862]: osdmap e258: 8 total, 8 up, 8 in 2024-03-21T14:13:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:47 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:13:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:47 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:13:47.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:47.993 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:47.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:48.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.129 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:48.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:48.230 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:48.241 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 15 61517 61502 128.105 96.125 0.0143557 0.00780088 2024-03-21T14:13:48.268 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:13:48.268 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:13:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.270 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:48.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:48.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:48.406 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:48.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:48.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:48.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:48.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:48.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:13:48.801 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-03-21T14:13:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:48.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:48.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:48 smithi003 ceph-mon[29576]: pgmap v500: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T14:13:48.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:48 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:13:48.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:48 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:13:48.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:48 smithi003 ceph-mon[40510]: pgmap v500: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T14:13:48.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:48 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:13:48.914 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:48 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:13:48.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:48 smithi118 ceph-mon[33862]: pgmap v500: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-21T14:13:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:48 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:13:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:48 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:13:48.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:48.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:49.064 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:49.097 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:49.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:49.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:13:49.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:49.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:49.240 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 63097 63081 127.156 98.6875 0.0100114 0.00785911 2024-03-21T14:13:49.340 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:49.374 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:49.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:49.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:49.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:13:49.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:49.653 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:49.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:49.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:49.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:13:49.847 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[40510]: osdmap e259: 8 total, 8 up, 8 in 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:13:49.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:49 smithi003 ceph-mon[29576]: osdmap e259: 8 total, 8 up, 8 in 2024-03-21T14:13:49.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:49.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:49.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:49 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:13:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:49 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:13:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:49 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:13:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:49 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:13:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:49 smithi118 ceph-mon[33862]: osdmap e259: 8 total, 8 up, 8 in 2024-03-21T14:13:50.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:50.042 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:50.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:50.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:50.104 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:50.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:50.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:50.241 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 64411 64395 125.747 82.125 0.00770275 0.00794693 2024-03-21T14:13:50.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:50.341 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:50.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:50.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:50.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:50.584 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:50.611 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:13:50.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:50.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[29576]: pgmap v501: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T14:13:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:13:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:13:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:13:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:13:50.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/254014361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[40510]: pgmap v501: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T14:13:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:13:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:13:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:13:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:13:50.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/254014361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.758 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:50.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:50.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:50.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:50.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:50 smithi118 ceph-mon[33862]: pgmap v501: 148 pgs: 148 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-03-21T14:13:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:50 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:13:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:50 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:13:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:50 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:13:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:50 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:13:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/254014361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:13:50.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:51.032 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:51.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:51.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:51.171 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:51.241 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 65921 65905 124.796 94.375 0.0170518 0.00800862 2024-03-21T14:13:51.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.307 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:51.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:51.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:51.452 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:51.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:51.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:51.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:51.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:51.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/254014361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:13:51.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: osdmap e260: 8 total, 8 up, 8 in 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/254014361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: osdmap e260: 8 total, 8 up, 8 in 2024-03-21T14:13:51.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:51 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:13:51.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:51.877 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:51.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:13:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:13:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:13:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:13:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:13:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:13:51.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/254014361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:13:51.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: osdmap e260: 8 total, 8 up, 8 in 2024-03-21T14:13:51.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:51 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:13:51.978 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:52.017 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:52.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:52.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:52.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:52.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:52.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 15 67382 67367 123.813 91.375 0.00926774 0.00807188 2024-03-21T14:13:52.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:52.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:52.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:52.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:52.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:52.598 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:52.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:52.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:52.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:52.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:52.844 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:52.876 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:52.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:52.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:52.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[29576]: pgmap v504: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-21T14:13:52.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:13:52.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:13:52.902 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:13:52.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[40510]: pgmap v504: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-21T14:13:52.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:13:52.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:13:52.903 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:52 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:13:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:52 smithi118 ceph-mon[33862]: pgmap v504: 148 pgs: 148 active+clean; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 103 MiB/s wr, 1.64k op/s 2024-03-21T14:13:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:52 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:13:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:52 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:13:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:52 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:13:52.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:53.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:53.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:53.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:53.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.170 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:53.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:53.241 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 68659 68643 122.554 79.75 0.00643681 0.00815477 2024-03-21T14:13:53.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:53.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:53.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:53.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:53.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:53.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:53.597 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:53.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:53.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:53.801 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:13:53.801 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:13:53.802 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T14:13:53.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T14:13:53.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:53.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:53 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:13:53.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:53 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:13:53.839 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:53 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:13:53.839 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:53 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:13:53.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:53.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:53 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:13:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:53 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:13:53.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:54.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:54.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:54.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:54.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 69879 69863 121.264 76.25 0.00951216 0.00824142 2024-03-21T14:13:54.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:54.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:54.420 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:13:54.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:54.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:54.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:54.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:13:54.507 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:13:54.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:13:54.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:54.595 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:13:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:54.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:54.636 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:13:54.677 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:13:54.677 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: pgmap v505: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 744 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/609006459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: 5.c deep-scrub starts 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: 5.c deep-scrub ok 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[29576]: osdmap e261: 8 total, 8 up, 8 in 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: pgmap v505: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 744 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/609006459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:54.678 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: 5.c deep-scrub starts 2024-03-21T14:13:54.679 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: 5.c deep-scrub ok 2024-03-21T14:13:54.679 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:13:54.679 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:13:54.679 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:54 smithi003 ceph-mon[40510]: osdmap e261: 8 total, 8 up, 8 in 2024-03-21T14:13:54.719 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:13:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:13:54.727 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:13:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:13:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:13:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:13:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:13:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:13:54.728 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:13:54.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:54.768 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:54.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:54.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:54.906 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:54.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: pgmap v505: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 744 B/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/609006459' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: 5.c deep-scrub starts 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: 5.c deep-scrub ok 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:13:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:54 smithi118 ceph-mon[33862]: osdmap e261: 8 total, 8 up, 8 in 2024-03-21T14:13:55.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:55.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:55.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:55.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:55.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:55.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 71103 71087 120.055 76.5 0.00757348 0.00832511 2024-03-21T14:13:55.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:55.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:141958 7.6 7:666570b1:::benchmark_data_smithi003_80362_object20941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:13:55.422 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:31.918588+0000", 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 443.05121956099998, 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16439579000000001, 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141958 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.918588+0000", 2024-03-21T14:13:55.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.918588+0000", 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.918589+0000", 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6400000000000002e-07 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.918592+0000", 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.2839999999999998e-06 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.918599+0000", 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0870000000000001e-06 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.918606+0000", 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.973e-06 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.960954+0000", 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.042347691999999999 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.961077+0000", 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012354099999999999 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.961196+0000", 2024-03-21T14:13:55.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011834599999999999 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.970457+0000", 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0092616569999999995 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978413+0000", 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0079555479999999998 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.427 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.082861+0000", 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.104448693 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.082917+0000", 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5649000000000002e-05 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.082984+0000", 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6755999999999996e-05 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.428 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:141968 7.6 7:6f27aae3:::benchmark_data_smithi003_80362_object20951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:31.924353+0000", 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 443.04545491800002, 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.158897293, 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141968 2024-03-21T14:13:55.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.924353+0000", 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.924353+0000", 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.924354+0000", 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3999999999999999e-06 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.924359+0000", 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5329999999999996e-06 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.924374+0000", 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5514000000000001e-05 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.924383+0000", 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5690000000000005e-06 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.964182+0000", 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039799355000000002 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.964311+0000", 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012940200000000001 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.964447+0000", 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013593200000000001 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978359+0000", 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013911566 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978418+0000", 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9135000000000002e-05 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.083149+0000", 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.104730489 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.083197+0000", 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8828000000000001e-05 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.083250+0000", 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.257e-05 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:141973 7.6 7:69b0d3d5:::benchmark_data_smithi003_80362_object20956:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:31.929095+0000", 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 443.04071253799998, 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16136488399999999, 2024-03-21T14:13:55.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141973 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.929095+0000", 2024-03-21T14:13:55.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.929095+0000", 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.929096+0000", 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1629999999999999e-06 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.929102+0000", 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4210000000000003e-06 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.929115+0000", 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.361e-05 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.929123+0000", 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3329999999999999e-06 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.967385+0000", 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.038262299999999999 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.967507+0000", 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000121741 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.967624+0000", 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117422 2024-03-21T14:13:55.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978423+0000", 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010798657999999999 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978467+0000", 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4641999999999997e-05 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.090281+0000", 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.111813356 2024-03-21T14:13:55.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.090389+0000", 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107996 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.090460+0000", 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1242000000000007e-05 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:141975 7.6 7:69e8bc21:::benchmark_data_smithi003_80362_object20958:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:31.931483+0000", 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 443.03832504100001, 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.159391901, 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 141975 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.442 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.931483+0000", 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.931483+0000", 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.931484+0000", 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.11e-06 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.931488+0000", 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5199999999999999e-06 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.931499+0000", 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0635999999999999e-05 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.931506+0000", 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3459999999999996e-06 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.970491+0000", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.038984726999999997 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.970601+0000", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010967399999999999 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.970716+0000", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115043 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978428+0000", 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077122579999999996 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:31.978505+0000", 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7070000000000006e-05 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.090724+0000", 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.112218937 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.090813+0000", 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9128000000000005e-05 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:32.090874+0000", 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1451999999999994e-05 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151132 7.6 7:6c627678:::benchmark_data_smithi003_80362_object30115:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:46.084612+0000", 2024-03-21T14:13:55.447 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 428.885196049, 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.20849891200000001, 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151132 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.084612+0000", 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.084612+0000", 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.084613+0000", 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7880000000000001e-06 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.084619+0000", 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8459999999999996e-06 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.084631+0000", 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1698999999999999e-05 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.084639+0000", 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8639999999999996e-06 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.147771+0000", 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.063131807999999998 2024-03-21T14:13:55.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.147961+0000", 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019059200000000001 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.148117+0000", 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000155699 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.151866+0000", 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037492290000000002 2024-03-21T14:13:55.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.159078+0000", 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072120250000000004 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.292957+0000", 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13387875799999999 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.293025+0000", 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7669999999999994e-05 2024-03-21T14:13:55.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.293110+0000", 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5933999999999994e-05 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151196 7.6 7:630e18b2:::benchmark_data_smithi003_80362_object30179:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:46.208116+0000", 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 428.76169187400001, 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16352362400000001, 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.453 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151196 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.208116+0000", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.208116+0000", 2024-03-21T14:13:55.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.208117+0000", 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.474e-06 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.208123+0000", 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8839999999999999e-06 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.208141+0000", 2024-03-21T14:13:55.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7504e-05 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.208149+0000", 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1489999999999994e-06 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.301459+0000", 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093310702999999995 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.301573+0000", 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113442 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.301698+0000", 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012522999999999999 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.304915+0000", 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003216857 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.305016+0000", 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100633 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.371510+0000", 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.066494761999999999 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.371575+0000", 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4953e-05 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:46.371639+0000", 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4033000000000002e-05 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:151779 7.6 7:6908a292:::benchmark_data_smithi003_80362_object30762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:47.123287+0000", 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 427.84652068899999, 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16573644100000001, 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.459 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 151779 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.123287+0000", 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.123287+0000", 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.123288+0000", 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6700000000000003e-07 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.123292+0000", 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2869999999999998e-06 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.123304+0000", 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1868999999999999e-05 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.123311+0000", 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9720000000000003e-06 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.183729+0000", 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.060417744000000002 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.183849+0000", 2024-03-21T14:13:55.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120538 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.183974+0000", 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012483000000000001 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.187213+0000", 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032387090000000002 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.207713+0000", 2024-03-21T14:13:55.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020500438999999999 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.288901+0000", 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.081187715999999993 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.288966+0000", 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4901999999999996e-05 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:47.289023+0000", 2024-03-21T14:13:55.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7667999999999997e-05 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153227 7.6 7:6f0cbf16:::benchmark_data_smithi003_80362_object32210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.390411+0000", 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 425.57939657200001, 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17522652599999999, 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.465 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153227 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.390411+0000", 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.390411+0000", 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.390412+0000", 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3650000000000001e-06 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.390418+0000", 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6879999999999996e-06 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.390433+0000", 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4975e-05 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.390441+0000", 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9740000000000002e-06 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.445277+0000", 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.054835463000000001 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.445394+0000", 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117413 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.445508+0000", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011445799999999999 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.453290+0000", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077820340000000002 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.457010+0000", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037194910000000001 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.565506+0000", 2024-03-21T14:13:55.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10849621600000001 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.565565+0000", 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8949000000000001e-05 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.565638+0000", 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.25e-05 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153367 7.6 7:69548cc7:::benchmark_data_smithi003_80362_object32350:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.602199+0000", 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 425.36760844899999, 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15359149599999999, 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153367 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.602199+0000", 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.602199+0000", 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.602200+0000", 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7199999999999998e-07 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.602205+0000", 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6110000000000004e-06 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.602218+0000", 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3926e-05 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.602227+0000", 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3669999999999997e-06 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.624480+0000", 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.022252854999999998 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.624638+0000", 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015838899999999999 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.624784+0000", 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014618699999999999 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.634773+0000", 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0099883239999999998 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.755191+0000", 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.120418384 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.755668+0000", 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00047686899999999999 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.755724+0000", 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5943999999999998e-05 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.755791+0000", 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6867999999999997e-05 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153431 7.6 7:66931924:::benchmark_data_smithi003_80362_object32414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.476 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.727277+0000", 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 425.24253075399997, 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.179091888, 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153431 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.727277+0000", 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.727277+0000", 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.727278+0000", 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3230000000000001e-06 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.727284+0000", 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0290000000000002e-06 2024-03-21T14:13:55.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.727529+0000", 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024449699999999999 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.727540+0000", 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0899e-05 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.781952+0000", 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.054412288000000003 2024-03-21T14:13:55.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.782165+0000", 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000213107 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.782363+0000", 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000197758 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.789642+0000", 2024-03-21T14:13:55.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072792339999999999 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.814780+0000", 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025137788000000001 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.906246+0000", 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091466262000000007 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.906301+0000", 2024-03-21T14:13:55.481 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4712000000000001e-05 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.906369+0000", 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7990999999999993e-05 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153436 7.6 7:611e0f87:::benchmark_data_smithi003_80362_object32419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:49.738552+0000", 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 425.23125604500001, 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.167966965, 2024-03-21T14:13:55.482 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153436 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.738552+0000", 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.738552+0000", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.738553+0000", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.141e-06 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.738559+0000", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8139999999999997e-06 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.738569+0000", 2024-03-21T14:13:55.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9930000000000007e-06 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.738576+0000", 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5399999999999998e-06 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.785311+0000", 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.046734713999999997 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.785435+0000", 2024-03-21T14:13:55.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012448400000000001 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.785585+0000", 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014966900000000001 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.789679+0000", 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040945390000000003 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.815104+0000", 2024-03-21T14:13:55.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.025425005000000001 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.906415+0000", 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.091310817000000002 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.906464+0000", 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8242000000000003e-05 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:49.906519+0000", 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5006999999999997e-05 2024-03-21T14:13:55.487 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153691 7.6 7:65dd3ae0:::benchmark_data_smithi003_80362_object32674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.104765+0000", 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.86504265799999, 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15923103, 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153691 2024-03-21T14:13:55.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.104765+0000", 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.104765+0000", 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.104766+0000", 2024-03-21T14:13:55.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.153e-06 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.104771+0000", 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1560000000000003e-06 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.104780+0000", 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1789999999999997e-06 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.104788+0000", 2024-03-21T14:13:55.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4630000000000004e-06 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.138404+0000", 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.033616003999999998 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.138535+0000", 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013094100000000001 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.138685+0000", 2024-03-21T14:13:55.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015028900000000001 2024-03-21T14:13:55.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.142703+0000", 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0040179880000000001 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.142797+0000", 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3372e-05 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.263780+0000", 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12098393 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.263880+0000", 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9468000000000004e-05 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.263996+0000", 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116087 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153937 7.f 7:fe142000:::benchmark_data_smithi003_80362_object32920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.459166+0000", 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.51064196700003, 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16093449900000001, 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.495 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153937 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.459166+0000", 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.459166+0000", 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.459167+0000", 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.136e-06 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.459170+0000", 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4740000000000001e-06 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.459177+0000", 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.99e-06 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.459183+0000", 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.767e-06 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.552805+0000", 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.093621791999999995 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.553142+0000", 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033743800000000002 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.553671+0000", 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052836500000000004 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.557276+0000", 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003605516 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.564697+0000", 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074206649999999999 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.619988+0000", 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.055291607 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.620052+0000", 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3137000000000002e-05 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.620100+0000", 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8612000000000001e-05 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.500 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:153938 7.f 7:f0ea2b9e:::benchmark_data_smithi003_80362_object32921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.461942+0000", 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.50786576899998, 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.158293199, 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 153938 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.461942+0000", 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.461942+0000", 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.461943+0000", 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1000000000000001e-06 2024-03-21T14:13:55.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.461948+0000", 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5870000000000003e-06 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.461961+0000", 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3414000000000001e-05 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.461968+0000", 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1770000000000001e-06 2024-03-21T14:13:55.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.557317+0000", 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095349014999999995 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.557632+0000", 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031478099999999998 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.557804+0000", 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000172051 2024-03-21T14:13:55.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.564743+0000", 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069386969999999997 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.564754+0000", 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1311999999999999e-05 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.620139+0000", 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.055384646000000003 2024-03-21T14:13:55.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.620192+0000", 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3557999999999998e-05 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.620235+0000", 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2861000000000002e-05 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.506 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154041 7.6 7:697049a8:::benchmark_data_smithi003_80362_object33024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.636682+0000", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.33312542499999, 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.158113684, 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154041 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636682+0000", 2024-03-21T14:13:55.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636682+0000", 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636683+0000", 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.096e-06 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636687+0000", 2024-03-21T14:13:55.508 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8859999999999997e-06 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636697+0000", 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0003000000000001e-05 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636705+0000", 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8960000000000003e-06 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636745+0000", 2024-03-21T14:13:55.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0021999999999997e-05 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636825+0000", 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.941e-05 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.636964+0000", 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000139901 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.640289+0000", 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033249389999999998 2024-03-21T14:13:55.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.642961+0000", 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026712939999999998 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.794668+0000", 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15170689500000001 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.794723+0000", 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5943000000000003e-05 2024-03-21T14:13:55.511 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.794796+0000", 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2398999999999996e-05 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.24692.0:154042 7.e e120/89 7:7e3a347b:::benchmark_data_smithi003_80362_object33025:head v 120'9700, mlcod=120'9698)", 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.644627+0000", 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.32518016699998, 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.163706509, 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.512 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644627+0000", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644627+0000", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644628+0000", 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7599999999999997e-07 2024-03-21T14:13:55.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644633+0000", 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1780000000000002e-06 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644641+0000", 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9279999999999993e-06 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644647+0000", 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7780000000000004e-06 2024-03-21T14:13:55.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644766+0000", 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119459 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.644785+0000", 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9114999999999999e-05 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.808290+0000", 2024-03-21T14:13:55.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16350476999999999 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.808334+0000", 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3705000000000003e-05 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154049 7.6 7:693e030c:::benchmark_data_smithi003_80362_object33032:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.645950+0000", 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.32385805299998, 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.189696169, 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.516 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154049 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.645950+0000", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.645950+0000", 2024-03-21T14:13:55.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.645951+0000", 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4160000000000001e-06 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.645956+0000", 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4970000000000001e-06 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699555+0000", 2024-03-21T14:13:55.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.053598229999999997 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699568+0000", 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3443e-05 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.819513+0000", 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11994476799999999 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.819651+0000", 2024-03-21T14:13:55.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000138319 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.819767+0000", 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115902 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.825741+0000", 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0059739959999999996 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.825798+0000", 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6611999999999999e-05 2024-03-21T14:13:55.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.835526+0000", 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0097286939999999995 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.835580+0000", 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3751000000000001e-05 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.835646+0000", 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5541000000000001e-05 2024-03-21T14:13:55.521 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154050 7.6 7:6ae4f36c:::benchmark_data_smithi003_80362_object33033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.699596+0000", 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.27021203800001, 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.160427608, 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.522 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154050 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699596+0000", 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699596+0000", 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699597+0000", 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3769999999999999e-06 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699602+0000", 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.31e-06 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699611+0000", 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8119999999999997e-06 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.699617+0000", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1550000000000004e-06 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.825843+0000", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12622550499999999 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.825943+0000", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010029199999999999 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.826024+0000", 2024-03-21T14:13:55.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0950000000000003e-05 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.831917+0000", 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0058928330000000001 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.839772+0000", 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.007855038 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.859882+0000", 2024-03-21T14:13:55.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.020110593999999999 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.859951+0000", 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8292000000000003e-05 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.860023+0000", 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2449999999999999e-05 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:154107 7.6 7:69234449:::benchmark_data_smithi003_80362_object33090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:06:50.782224+0000", 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 424.187583964, 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17822795899999999, 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 154107 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.782224+0000", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.782224+0000", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.782224+0000", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1200000000000002e-07 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.782229+0000", 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8250000000000004e-06 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.782238+0000", 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6240000000000008e-06 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.782245+0000", 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0099999999999998e-06 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.839908+0000", 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.057662956000000001 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.840028+0000", 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120115 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,7", 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.840137+0000", 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010887099999999999 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.843771+0000", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036340410000000002 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.860492+0000", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.016721393000000001 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.960374+0000", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.099881365 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:13:55.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.960410+0000", 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6050000000000002e-05 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:06:50.960452+0000", 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1996999999999998e-05 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:228274 8.6 8.e78a0f06 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:19.639563+0000", 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 95.330245031000004, 2024-03-21T14:13:55.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92258471799999997, 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 228274 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639563+0000", 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639563+0000", 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639564+0000", 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4780000000000001e-06 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639571+0000", 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4520000000000002e-06 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639585+0000", 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4477999999999999e-05 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639594+0000", 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1519999999999998e-06 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639673+0000", 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9060000000000005e-05 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639694+0000", 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1234999999999999e-05 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:20.562121+0000", 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92242685000000002 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:20.562147+0000", 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6013e-05 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.537 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:13:55.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:13:55.538 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:13:55.538 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:55.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:55.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:55.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:13:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:13:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:13:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:55 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:13:55.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:55.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:55 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:55 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:55 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:13:55.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:55 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:13:55.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:55.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:55.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:56.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:13:56.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:56.133 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:56.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:56.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:56.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 72544 72528 119.265 90.0625 0.00631422 0.00837935 2024-03-21T14:13:56.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:56.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:56.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:56.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:56.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:56.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:56.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:56.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:13:56.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:56.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:56.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:56.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:13:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:13:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: pgmap v507: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:13:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 4.11 deep-scrub starts 2024-03-21T14:13:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 4.11 deep-scrub ok 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: osdmap e262: 8 total, 8 up, 8 in 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 4.0 deep-scrub starts 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[29576]: 4.0 deep-scrub ok 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: pgmap v507: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 4.11 deep-scrub starts 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 4.11 deep-scrub ok 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:13:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:13:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: osdmap e262: 8 total, 8 up, 8 in 2024-03-21T14:13:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:13:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:13:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 4.0 deep-scrub starts 2024-03-21T14:13:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:56 smithi003 ceph-mon[40510]: 4.0 deep-scrub ok 2024-03-21T14:13:56.770 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:56.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:56.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:56.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: pgmap v507: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 4.11 deep-scrub starts 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 4.11 deep-scrub ok 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: osdmap e262: 8 total, 8 up, 8 in 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 4.0 deep-scrub starts 2024-03-21T14:13:56.913 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-mon[33862]: 4.0 deep-scrub ok 2024-03-21T14:13:56.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:56.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:57.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:57.054 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:57.069 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:57.084 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:57.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:57.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:13:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:13:57.191 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:57.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 73838 73822 118.281 80.875 0.00889707 0.00844897 2024-03-21T14:13:57.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:57.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:57.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:57.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:57.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:57.616 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:57.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:57.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:57.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:57.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:57.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:57.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:57 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:13:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:57 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:13:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:57 smithi118 ceph-mon[33862]: osdmap e263: 8 total, 8 up, 8 in 2024-03-21T14:13:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:57 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:13:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:57 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:13:57.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:13:57.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[29576]: osdmap e263: 8 total, 8 up, 8 in 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[40510]: osdmap e263: 8 total, 8 up, 8 in 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:13:57.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:57 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:13:57.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:58.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:58.126 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:58.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.162 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:58.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:58.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:13:58.243588+0000 min lat: 0.00113893 max lat: 0.152179 avg lat: 0.00851577 2024-03-21T14:13:58.242 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:13:58.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 75142 75126 117.361 81.5 0.0146188 0.00851577 2024-03-21T14:13:58.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:13:58.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:58.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:13:58.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:58.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:58.547 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:13:58.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.582 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:58.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:13:58.642 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:13:58.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:13:58.720 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:13:58.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:58.858 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:58.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:13:58 smithi118 ceph-mon[33862]: pgmap v510: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-21T14:13:58.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:58.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:58.998 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:58.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:13:58 smithi003 ceph-mon[29576]: pgmap v510: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-21T14:13:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:13:58 smithi003 ceph-mon[40510]: pgmap v510: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-21T14:13:59.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:13:59.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:13:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:59.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:59.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:13:59.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 76551 76535 116.646 88.0625 0.0118775 0.00856778 2024-03-21T14:13:59.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:13:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:59.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:59.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:13:59.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:59.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:59.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:59.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:13:59.578 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:59.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:59.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:13:59.680 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:13:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:59.713 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:13:59.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:13:59.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:13:59.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:13:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:13:59.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:13:59.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:00.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:00.170 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:00.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:00.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:00.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 77888 77872 115.858 83.5625 0.00745836 0.00862549 2024-03-21T14:14:00.317 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.359 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:00.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:00.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.496 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:00.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:00.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:00.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:00.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:00.733 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:00.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:00.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:00.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:00.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:00.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:00 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:14:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:00 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:14:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:00 smithi118 ceph-mon[33862]: pgmap v511: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-21T14:14:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:00 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:14:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:00 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:14:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:00 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:14:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:14:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:14:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[29576]: pgmap v511: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-21T14:14:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:14:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:14:01.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:14:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:14:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:14:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[40510]: pgmap v511: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-21T14:14:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:14:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:14:01.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:00 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:14:01.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:01.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:01.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:01.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:01.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 79187 79171 115.052 81.1875 0.010417 0.00868698 2024-03-21T14:14:01.292 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:01.328 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:01.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:01.464 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:01.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:01.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:01.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:01.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.745 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:01.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:01.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:01.881 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:01.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:01.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 6.1c deep-scrub starts 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 6.1c deep-scrub ok 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:14:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:01 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:14:01.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:01.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 6.1c deep-scrub starts 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 6.1c deep-scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 6.1c deep-scrub starts 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 6.1c deep-scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:14:01.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:14:01.998 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:01 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:14:02.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:02.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:02.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:02.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 80488 80472 114.285 81.3125 0.0184161 0.00874497 2024-03-21T14:14:02.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:02.310 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:02.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:02.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.448 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:02.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:02.549 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.581 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:02.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:02.648 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:14:02.648 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:14:02.648 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 263, 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-03-21T13:56:54.530125+0000", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-03-21T14:13:56.652299+0000", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2024-03-21T14:02:19.321454+0000", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2024-03-21T14:02:01.236118+0000", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5799936, 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:14:02.649 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 18, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 9, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "luminous", 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "luminous", 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "squid", 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_crimson": false, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2024-03-21T14:14:02.650 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".mgr", 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:00:12.925535+0000", 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 1, 2024-03-21T14:14:02.651 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 1, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 1, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 1, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 1, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "122", 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.652 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.653 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.654 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_max": 32, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": {} 2024-03-21T14:14:02.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 6.9800000190734863, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 6.9800000190734863, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.43000000715255737, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 3, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 3, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.656 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "datapool", 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.373659+0000", 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.657 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 3, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 3, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 3, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 3, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 3, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.658 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 2, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 68, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.659 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.660 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.3299999237060547, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.3299999237060547, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.86000001430511475, 2024-03-21T14:14:02.661 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".rgw.root", 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.794831+0000", 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.662 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.663 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "66", 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.665 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.666 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.667 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5399999618530273, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5399999618530273, 2024-03-21T14:14:02.668 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.75, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.75, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 4, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.log", 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:33.750394+0000", 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:14:02.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:14:02.670 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.671 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.672 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.673 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.190000057220459, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.190000057220459, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.5, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.5, 2024-03-21T14:14:02.674 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 5, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.control", 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:35.711498+0000", 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.675 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.676 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "70", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:14:02.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5399999618530273, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5399999618530273, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.75, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.75, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.680 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 6, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.meta", 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:37.763124+0000", 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.681 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "72", 2024-03-21T14:14:02.682 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.683 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.684 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.685 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.9700000286102295, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.9700000286102295, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 9, 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "unique_pool_2", 2024-03-21T14:14:02.686 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:13:07.510511+0000", 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 16, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 16, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 16, 2024-03-21T14:14:02.687 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 16, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 16, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "248", 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:14:02.688 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:14:02.689 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:14:02.690 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "rados": {} 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 3.059999942779541, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 3.059999942779541, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 3.059999942779541, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 3.059999942779541, 2024-03-21T14:14:02.691 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.875, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 1, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb", 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.73016357421875, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.692 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 8, 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 224, 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6801/930096433", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6802/930096433", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6804/930096433", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6803/930096433", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.695 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d", 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 13, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 249, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.696 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2024-03-21T14:14:02.697 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6805/1526485482", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6806/1526485482", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6808/1526485482", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/1526485482", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c", 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 18, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 224, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.700 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.701 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.702 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3751378277 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6809/3751378277", 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6810/3751378277", 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6812/3751378277", 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6811/3751378277", 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.703 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c", 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.6562042236328125, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 25, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 244, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.705 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.706 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6813/1918209225", 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6814/1918209225", 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6816/1918209225", 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6815/1918209225", 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:14:02.707 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5", 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 30, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 249, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6800", 2024-03-21T14:14:02.708 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6801", 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6803", 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.710 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.711 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6802", 2024-03-21T14:14:02.711 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6800/296472230", 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6801/296472230", 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6803/296472230", 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6802/296472230", 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:14:02.712 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "bff24605-38ce-47ce-9142-4ced3766476e", 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 40, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 249, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6804", 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6805", 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.714 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6807", 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6806", 2024-03-21T14:14:02.715 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6804/326385780", 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6805/326385780", 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6807/326385780", 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6806/326385780", 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:14:02.716 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "73a781e7-8660-4885-b976-2aeb8515109a", 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.2286529541015625, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 48, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 224, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6808", 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.718 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6809", 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.719 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6811", 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6810", 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1322208109 2024-03-21T14:14:02.720 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6808/1322208109", 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6809/1322208109", 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6811/1322208109", 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6810/1322208109", 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.721 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0", 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 58, 2024-03-21T14:14:02.722 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 244, 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6812", 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6813", 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6815", 2024-03-21T14:14:02.724 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6814", 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6812/2128678671", 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6813/2128678671", 2024-03-21T14:14:02.725 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6815/2128678671", 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6814/2128678671", 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.726 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:54.926862+0000", 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:45.233321+0000", 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.727 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:38.793891+0000", 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.728 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:56.241779+0000", 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:45.818519+0000", 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.729 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:35.117113+0000", 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.730 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:13:42.808843+0000", 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:14:02.731 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:12:37.469201+0000", 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [ 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.13", 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:14:02.732 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.0", 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.4", 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.733 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 5 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.b", 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.734 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.13", 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.1e", 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.735 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.0", 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.2", 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:14:02.736 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.4", 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 5, 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.737 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.a", 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.d", 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.738 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.e", 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 6 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.739 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.13", 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_primaries": [], 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklist": { 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/45234419": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/384540368": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:14:02.740 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3883474825": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/1598547836": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3687155565": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3884352348": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2319227492": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2154949954": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/4162970769": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2292819669": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3411869253": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/520312891": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2267426691": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3211426112": "2024-03-22T13:57:56.576834+0000" 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "range_blocklist": {}, 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2024-03-21T14:14:02.741 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {}, 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": { 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_enabled": false, 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_bucket_count": 0, 2024-03-21T14:14:02.742 INFO:teuthology.orchestra.run.smithi003.stdout: "degraded_stretch_mode": 0, 2024-03-21T14:14:02.743 INFO:teuthology.orchestra.run.smithi003.stdout: "recovering_stretch_mode": 0, 2024-03-21T14:14:02.743 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_bucket": 0 2024-03-21T14:14:02.743 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:02.743 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:14:02.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:02.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:02.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:02.917 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:02.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:02 smithi118 ceph-mon[33862]: pgmap v512: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 741 B/s rd, 60 MiB/s wr, 952 op/s 2024-03-21T14:14:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:02 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:14:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:02 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:14:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2121077496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:14:02.998 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[29576]: pgmap v512: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 741 B/s rd, 60 MiB/s wr, 952 op/s 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2121077496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[40510]: pgmap v512: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 741 B/s rd, 60 MiB/s wr, 952 op/s 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:02.999 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:14:03.000 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:14:03.000 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2121077496' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:14:03.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:03.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:03.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:03.192 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:03.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 81790 81774 113.554 81.375 0.0157427 0.00880025 2024-03-21T14:14:03.294 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.332 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:03.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:03.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:03.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:03.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:03.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:03.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:03.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:03.738 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:03.839 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:03.875 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:03.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:03.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:04.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:04.243 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 83084 83068 112.843 80.875 0.0114754 0.00885709 2024-03-21T14:14:04.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:04.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:04.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:04.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:04.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:04.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:04.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:04.675 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:04.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:04.811 INFO:tasks.thrashosds.thrasher:j is {'epoch': 263, 'fsid': 'c60e082a-e78a-11ee-95cd-87774f69a715', 'created': '2024-03-21T13:56:54.530125+0000', 'modified': '2024-03-21T14:13:56.652299+0000', 'last_up_change': '2024-03-21T14:02:19.321454+0000', 'last_in_change': '2024-03-21T14:02:01.236118+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T14:00:12.925535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.980000019073486, 'score_stable': 6.980000019073486, 'optimal_score': 0.4300000071525574, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T14:02:31.373659+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T14:02:31.794831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T14:02:33.750394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T14:02:35.711498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T14:02:37.763124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-21T14:13:07.510511+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '248', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.059999942779541, 'score_stable': 3.059999942779541, 'optimal_score': 1, 'raw_score_acting': 3.059999942779541, 'raw_score_stable': 3.059999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'daf9bc3c-4e73-4379-afb6-7ee181b686fb', 'up': 1, 'in': 1, 'weight': 0.73016357421875, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 224, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6801', 'nonce': 930096433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6802', 'nonce': 930096433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6804', 'nonce': 930096433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6803', 'nonce': 930096433}]}, 'public_addr': '172.21.15.3:6801/930096433', 'cluster_addr': '172.21.15.3:6802/930096433', 'heartbeat_back_addr': '172.21.15.3:6804/930096433', 'heartbeat_front_addr': '172.21.15.3:6803/930096433', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1652f24d-aa0e-471c-bf54-03814b1ee11d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 249, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6805', 'nonce': 1526485482}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6806', 'nonce': 1526485482}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6808', 'nonce': 1526485482}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6807', 'nonce': 1526485482}]}, 'public_addr': '172.21.15.3:6805/1526485482', 'cluster_addr': '172.21.15.3:6806/1526485482', 'heartbeat_back_addr': '172.21.15.3:6808/1526485482', 'heartbeat_front_addr': '172.21.15.3:6807/1526485482', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5e67d907-f8d1-4ecc-85d3-df88da5c364c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 224, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6809', 'nonce': 3751378277}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6810', 'nonce': 3751378277}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6812', 'nonce': 3751378277}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6811', 'nonce': 3751378277}]}, 'public_addr': '172.21.15.3:6809/3751378277', 'cluster_addr': '172.21.15.3:6810/3751378277', 'heartbeat_back_addr': '172.21.15.3:6812/3751378277', 'heartbeat_front_addr': '172.21.15.3:6811/3751378277', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'ad8dabdf-fce5-4312-9e80-dbbb6ba2947c', 'up': 1, 'in': 1, 'weight': 0.6562042236328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6813', 'nonce': 1918209225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6814', 'nonce': 1918209225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6816', 'nonce': 1918209225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6815', 'nonce': 1918209225}]}, 'public_addr': '172.21.15.3:6813/1918209225', 'cluster_addr': '172.21.15.3:6814/1918209225', 'heartbeat_back_addr': '172.21.15.3:6816/1918209225', 'heartbeat_front_addr': '172.21.15.3:6815/1918209225', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b5d486b1-2cb6-46c7-8664-994af07ff7b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 249, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 296472230}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 296472230}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 296472230}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 296472230}]}, 'public_addr': '172.21.15.118:6800/296472230', 'cluster_addr': '172.21.15.118:6801/296472230', 'heartbeat_back_addr': '172.21.15.118:6803/296472230', 'heartbeat_front_addr': '172.21.15.118:6802/296472230', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bff24605-38ce-47ce-9142-4ced3766476e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 249, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 326385780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 326385780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 326385780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 326385780}]}, 'public_addr': '172.21.15.118:6804/326385780', 'cluster_addr': '172.21.15.118:6805/326385780', 'heartbeat_back_addr': '172.21.15.118:6807/326385780', 'heartbeat_front_addr': '172.21.15.118:6806/326385780', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '73a781e7-8660-4885-b976-2aeb8515109a', 'up': 1, 'in': 1, 'weight': 0.2286529541015625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 224, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 1322208109}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 1322208109}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 1322208109}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 1322208109}]}, 'public_addr': '172.21.15.118:6808/1322208109', 'cluster_addr': '172.21.15.118:6809/1322208109', 'heartbeat_back_addr': '172.21.15.118:6811/1322208109', 'heartbeat_front_addr': '172.21.15.118:6810/1322208109', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3e95559b-79f6-40ba-9d98-d350bcdd60e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 2128678671}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 2128678671}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 2128678671}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 2128678671}]}, 'public_addr': '172.21.15.118:6812/2128678671', 'cluster_addr': '172.21.15.118:6813/2128678671', 'heartbeat_back_addr': '172.21.15.118:6815/2128678671', 'heartbeat_front_addr': '172.21.15.118:6814/2128678671', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:54.926862+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:45.233321+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:38.793891+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:56.241779+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:45.818519+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:35.117113+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:13:42.808843+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:12:37.469201+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.13', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.0', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1e', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.0', 'mappings': [{'from': 3, 'to': 6}]}, {'pgid': '6.2', 'mappings': [{'from': 2, 'to': 0}]}, {'pgid': '6.4', 'mappings': [{'from': 5, 'to': 0}, {'from': 3, 'to': 6}]}, {'pgid': '6.a', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '6.d', 'mappings': [{'from': 2, 'to': 6}]}, {'pgid': '6.e', 'mappings': [{'from': 2, 'to': 6}]}, {'pgid': '6.13', 'mappings': [{'from': 2, 'to': 1}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.3:0/45234419': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/384540368': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3883474825': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/1598547836': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:6800/3687155565': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3884352348': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2319227492': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2154949954': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/4162970769': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/2292819669': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/3411869253': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/520312891': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2267426691': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/3211426112': '2024-03-22T13:57:56.576834+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T14:14:04.811 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-03-21T14:14:04.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:04.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:04 smithi118 ceph-mon[33862]: 5.19 deep-scrub starts 2024-03-21T14:14:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:04 smithi118 ceph-mon[33862]: 5.19 deep-scrub ok 2024-03-21T14:14:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:04 smithi118 ceph-mon[33862]: pgmap v513: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-21T14:14:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:04 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:14:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:04 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:14:04.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[29576]: 5.19 deep-scrub starts 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[29576]: 5.19 deep-scrub ok 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[29576]: pgmap v513: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[40510]: 5.19 deep-scrub starts 2024-03-21T14:14:04.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[40510]: 5.19 deep-scrub ok 2024-03-21T14:14:04.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[40510]: pgmap v513: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-21T14:14:04.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:14:04.968 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:04 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:14:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:04.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:04.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:05.089 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:05.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:05.227 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:05.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 84398 84382 112.19 82.125 0.0113755 0.00890892 2024-03-21T14:14:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:05.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:05.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:05.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:05.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:05.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:05.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:05.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:05.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:05.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:05 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:14:05.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:05 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:14:05.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:05 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:14:05.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:05 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:14:05.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:05.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:05.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:05.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:05 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:14:05.935 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:05 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:14:05.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:05.994 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:05.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:05.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:06.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:06.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:06.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:06.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:06.244 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 85756 85740 111.617 84.875 0.0118967 0.00895389 2024-03-21T14:14:06.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:06.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:06.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:06.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:06.457 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:06.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:06.555 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:06.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:06.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:06.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[29576]: pgmap v514: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T14:14:06.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:14:06.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:14:06.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:14:06.728 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:14:06.728 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:14:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:14:06.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[40510]: pgmap v514: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T14:14:06.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:14:06.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:14:06.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:14:06.729 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:06 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:14:06.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:06.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:06.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:06.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:06 smithi118 ceph-mon[33862]: pgmap v514: 148 pgs: 148 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.0 KiB/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-21T14:14:06.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:06 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:14:06.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:06 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:14:06.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:06 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:14:06.923 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:06 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:14:06.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:06.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:06.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:06.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:07.058 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:14:07.058 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:14:07.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:86632 9.0 9:02ad8776:::benchmark_data_smithi003_135814_object86631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:07.043108+0000", 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.014059563000000001, 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014096864000000001, 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86632 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.043108+0000", 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.043108+0000", 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.043110+0000", 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9250000000000002e-06 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.060 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.043133+0000", 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3129000000000002e-05 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.043145+0000", 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1980999999999999e-05 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.043152+0000", 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7510000000000008e-06 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.046793+0000", 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003640061 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.046962+0000", 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000169591 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.047131+0000", 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016867 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:07.062 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:86637 9.2 9:472836d8:::benchmark_data_smithi003_135814_object86636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:07.049688+0000", 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0074788069999999996, 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076034639999999999, 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86637 2024-03-21T14:14:07.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049688+0000", 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049688+0000", 2024-03-21T14:14:07.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049690+0000", 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5009999999999999e-06 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049711+0000", 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1367000000000001e-05 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049724+0000", 2024-03-21T14:14:07.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2642e-05 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049732+0000", 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9549999999999992e-06 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.052306+0000", 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002574439 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.052439+0000", 2024-03-21T14:14:07.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013282499999999999 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,7", 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.052582+0000", 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000143292 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:86638 9.4 9:2c5fc176:::benchmark_data_smithi003_135814_object86637:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:07.049748+0000", 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0074194600000000001, 2024-03-21T14:14:07.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076235729999999998, 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86638 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049748+0000", 2024-03-21T14:14:07.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:07.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049748+0000", 2024-03-21T14:14:07.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:07.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049748+0000", 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.99e-07 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049770+0000", 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1624000000000002e-05 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049779+0000", 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5059999999999995e-06 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049785+0000", 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0399999999999998e-06 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.050156+0000", 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037161600000000001 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.050321+0000", 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016500999999999999 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.050481+0000", 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015955 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:86641 9.4 9:2ba89110:::benchmark_data_smithi003_135814_object86640:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:07.049796+0000", 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0073714219999999999, 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076539440000000002, 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:14:07.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 86641 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049796+0000", 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049796+0000", 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049796+0000", 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9399999999999995e-07 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049815+0000", 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8970999999999998e-05 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049823+0000", 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6070000000000002e-06 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049829+0000", 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0970000000000002e-06 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.052707+0000", 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0028777550000000001 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.052840+0000", 2024-03-21T14:14:07.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013290399999999999 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 3,4", 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.052988+0000", 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014842800000000001 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25232.0:86634 9.6 e263/244 9:638eda0e:::benchmark_data_smithi003_135814_object86633:head v 263'5432, mlcod=263'5429)", 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:07.049871+0000", 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0072967070000000004, 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076532129999999999, 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:14:07.079 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049871+0000", 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.080 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049871+0000", 2024-03-21T14:14:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.049871+0000", 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4000000000000002e-07 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.050021+0000", 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015028799999999999 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.050030+0000", 2024-03-21T14:14:07.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9069999999999996e-06 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.050037+0000", 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3029999999999996e-06 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.055162+0000", 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051257150000000003 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:07.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.055182+0000", 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9355999999999999e-05 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25232.0:86639 9.d e263/244 9:be280dfb:::benchmark_data_smithi003_135814_object86638:head v 263'5444, mlcod=263'5442)", 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:07.051103+0000", 2024-03-21T14:14:07.084 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0060639120000000003, 2024-03-21T14:14:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064809810000000002, 2024-03-21T14:14:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:14:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051103+0000", 2024-03-21T14:14:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:07.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051103+0000", 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051104+0000", 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.034e-06 2024-03-21T14:14:07.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051142+0000", 2024-03-21T14:14:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7632999999999997e-05 2024-03-21T14:14:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051152+0000", 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0358e-05 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:07.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051159+0000", 2024-03-21T14:14:07.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1439999999999997e-06 2024-03-21T14:14:07.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:07.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051235+0000", 2024-03-21T14:14:07.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5647000000000003e-05 2024-03-21T14:14:07.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:07.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:07.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:07.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:07.051255+0000", 2024-03-21T14:14:07.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0185000000000001e-05 2024-03-21T14:14:07.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.092 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:07.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:07.093 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:14:07.093 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 6 2024-03-21T14:14:07.093 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:14:07.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:07.121 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:07.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:14:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:14:07.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:07.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 86841 86825 110.723 67.8125 0.00795304 0.00902683 2024-03-21T14:14:07.254 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:07.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:07.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:07.489 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:07.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:07.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:07.636 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:14:07.656 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:07.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:07.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:07.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:07.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: 5.18 scrub starts 2024-03-21T14:14:07.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: 5.18 scrub ok 2024-03-21T14:14:07.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4961499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: 5.18 scrub starts 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: 5.18 scrub ok 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:07.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:14:07.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:14:07.897 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4961499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:14:07.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: 5.18 scrub starts 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: 5.18 scrub ok 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:14:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4961499' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:14:07.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:07.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:07.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:08.040 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:08.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:08.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:08.218 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:08.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 88064 88048 110.037 76.4375 0.00445924 0.00908312 2024-03-21T14:14:08.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:08.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:08.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:08.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:08.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:08.629 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:08.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:08.731 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[29576]: pgmap v515: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 949 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-21T14:14:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:14:08.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4961499' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[29576]: osdmap e264: 8 total, 8 up, 8 in 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[40510]: pgmap v515: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 949 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4961499' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:14:08.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:08 smithi003 ceph-mon[40510]: osdmap e264: 8 total, 8 up, 8 in 2024-03-21T14:14:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.771 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:08.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:08.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:08.909 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:08.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:08 smithi118 ceph-mon[33862]: pgmap v515: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 949 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-21T14:14:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:08 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:14:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:08 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:14:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:08 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:08 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4961499' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:14:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:08 smithi118 ceph-mon[33862]: osdmap e264: 8 total, 8 up, 8 in 2024-03-21T14:14:09.011 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:09.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:09.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:09.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:09.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 89434 89418 109.558 85.625 0.0113393 0.00912278 2024-03-21T14:14:09.293 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.333 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:09.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:09.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:09.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:09.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:09.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:09.603 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:09.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:09.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:09.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:14:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.748 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:09.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:09.813 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:14:09.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:14:09.814 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-21T14:14:09.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:14:09.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:09.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:09.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:09.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:10.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:10.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:10.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:10.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:10.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 90715 90699 108.991 80.0625 0.0186547 0.00916988 2024-03-21T14:14:10.333 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:10.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:10.350 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:10.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.376 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:10.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:10.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:14:10.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:10.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:10.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:10.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:10 smithi003 ceph-mon[29576]: pgmap v517: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:10.712 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:10 smithi003 ceph-mon[29576]: osdmap e265: 8 total, 8 up, 8 in 2024-03-21T14:14:10.713 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:10 smithi003 ceph-mon[40510]: pgmap v517: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:10.713 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:10 smithi003 ceph-mon[40510]: osdmap e265: 8 total, 8 up, 8 in 2024-03-21T14:14:10.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:10.829 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:10 smithi118 ceph-mon[33862]: pgmap v517: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:10.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:10 smithi118 ceph-mon[33862]: osdmap e265: 8 total, 8 up, 8 in 2024-03-21T14:14:10.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:10.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:10.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:11.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:11.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:11.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:11.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:11.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 92013 91997 108.465 81.125 0.0117752 0.009215 2024-03-21T14:14:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:11.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:11.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:11.394 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:11.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:11.633 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:11.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:11.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:11.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:11.821 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:11.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:11.943 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:14:11.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:11.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:11.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:11.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:14:12.031 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:14:12.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:12.074 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:14:12.111 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:12.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:12.116 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:14:12.156 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:14:12.199 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:14:12.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:12.242 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:14:12.244 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:14:12.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 93343 93327 107.996 83.125 0.0112224 0.00925416 2024-03-21T14:14:12.248 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:12.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:12.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:12.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:12.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:12.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:12.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:12.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:12.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:12.796 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:12.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:12.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:12 smithi118 ceph-mon[33862]: pgmap v519: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 816 op/s 2024-03-21T14:14:12.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4259672490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:12.937 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:12.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:12 smithi003 ceph-mon[29576]: pgmap v519: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 816 op/s 2024-03-21T14:14:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4259672490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:12 smithi003 ceph-mon[40510]: pgmap v519: 148 pgs: 148 active+clean; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 816 op/s 2024-03-21T14:14:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4259672490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:13.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:13.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:13.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:13.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:13.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 94672 94656 107.542 83.0625 0.0109728 0.00929396 2024-03-21T14:14:13.329 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:13.365 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:13.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:13.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:13.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:13.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:13.681 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:14:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=cleanup t=2024-03-21T14:14:13.424060556Z level=info msg="Completed cleanup jobs" duration=38.079061ms 2024-03-21T14:14:13.681 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:14:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=plugins.update.checker t=2024-03-21T14:14:13.536696282Z level=info msg="Update check succeeded" duration=66.903625ms 2024-03-21T14:14:13.681 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:14:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafana.update.checker t=2024-03-21T14:14:13.539998406Z level=info msg="Update check succeeded" duration=69.500659ms 2024-03-21T14:14:13.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:13.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:13.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:13.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:13.933 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:13.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:14.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:14.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:14.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:14.245 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 96068 96052 107.18 87.25 0.0122861 0.00932584 2024-03-21T14:14:14.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.354 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:14.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:14.408 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:14.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:14.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:14.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:14.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:14.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:14.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:14.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:14.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:14.836 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:14 smithi003 ceph-mon[29576]: pgmap v520: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-21T14:14:14.837 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:14 smithi003 ceph-mon[40510]: pgmap v520: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-21T14:14:14.860 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:14:14.875 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:14:14.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:14.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:14.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:14.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:14 smithi118 ceph-mon[33862]: pgmap v520: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-21T14:14:15.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:15.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:15.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:15.196 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:15.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:15.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 97316 97300 106.667 78 0.0115597 0.00937047 2024-03-21T14:14:15.298 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:15.332 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:15.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:15.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:15.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:15.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:14:15.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:15 smithi003 ceph-mon[29576]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:14:15.701 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:15 smithi003 ceph-mon[40510]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:14:15.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:15.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:15.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:15.891 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:15.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:15.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:15 smithi118 ceph-mon[33862]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:14:15.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:16.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:16.139 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:16.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:16.246 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 98669 98653 106.286 84.5625 0.00523534 0.00940371 2024-03-21T14:14:16.256 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":520,"stamp":"2024-03-21T14:14:13.465500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6071327144,"num_objects":92830,"num_object_clones":0,"num_object_copies":278490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92830,"num_whiteouts":0,"num_read":2047,"num_read_kb":1893,"num_write":93533,"num_write_kb":5930909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":2954599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74900,"ondisk_log_size":74900,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20869296,"kb_used_data":17664220,"kb_used_omap":12,"kb_used_meta":3204979,"kb_avail":729091920,"statfs":{"total":767960285184,"available":746590126080,"internally_reserved":0,"allocated":18088161280,"data_stored":18084458608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3281899056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,3,6,9,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":844824576,"num_objects":12891,"num_object_clones":0,"num_object_copies":38673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12891,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12891,"num_write_kb":825024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002457"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.559964+0000","last_change":"2024-03-21T14:14:03.789055+0000","last_active":"2024-03-21T14:14:09.559964+0000","last_peered":"2024-03-21T14:14:09.559964+0000","last_clean":"2024-03-21T14:14:09.559964+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:14:09.559964+0000","last_undegraded":"2024-03-21T14:14:09.559964+0000","last_fullsized":"2024-03-21T14:14:09.559964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:03.788946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:03.788946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:10.852590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":454,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566274+0000","last_change":"2024-03-21T14:14:03.429724+0000","last_active":"2024-03-21T14:14:09.566274+0000","last_peered":"2024-03-21T14:14:09.566274+0000","last_clean":"2024-03-21T14:14:09.566274+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:14:09.566274+0000","last_undegraded":"2024-03-21T14:14:09.566274+0000","last_fullsized":"2024-03-21T14:14:09.566274+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:14:03.429627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:14.891870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":314,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566864+0000","last_change":"2024-03-21T14:13:07.249337+0000","last_active":"2024-03-21T14:14:09.566864+0000","last_peered":"2024-03-21T14:14:09.566864+0000","last_clean":"2024-03-21T14:14:09.566864+0000","last_became_active":"2024-03-21T14:12:00.528378+0000","last_became_peered":"2024-03-21T14:12:00.528378+0000","last_unstale":"2024-03-21T14:14:09.566864+0000","last_undegraded":"2024-03-21T14:14:09.566864+0000","last_fullsized":"2024-03-21T14:14:09.566864+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:32.472312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":483,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558001+0000","last_change":"2024-03-21T14:12:52.533831+0000","last_active":"2024-03-21T14:14:09.558001+0000","last_peered":"2024-03-21T14:14:09.558001+0000","last_clean":"2024-03-21T14:14:09.558001+0000","last_became_active":"2024-03-21T14:12:00.530310+0000","last_became_peered":"2024-03-21T14:12:00.530310+0000","last_unstale":"2024-03-21T14:14:09.558001+0000","last_undegraded":"2024-03-21T14:14:09.558001+0000","last_fullsized":"2024-03-21T14:14:09.558001+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:12:52.533687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:52.533687+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":314,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562122+0000","last_change":"2024-03-21T14:12:49.783355+0000","last_active":"2024-03-21T14:14:09.562122+0000","last_peered":"2024-03-21T14:14:09.562122+0000","last_clean":"2024-03-21T14:14:09.562122+0000","last_became_active":"2024-03-21T14:12:00.531208+0000","last_became_peered":"2024-03-21T14:12:00.531208+0000","last_unstale":"2024-03-21T14:14:09.562122+0000","last_undegraded":"2024-03-21T14:14:09.562122+0000","last_fullsized":"2024-03-21T14:14:09.562122+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:49.783238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:49.783238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:17.052191+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":452,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.557875+0000","last_change":"2024-03-21T14:14:06.781676+0000","last_active":"2024-03-21T14:14:09.557875+0000","last_peered":"2024-03-21T14:14:09.557875+0000","last_clean":"2024-03-21T14:14:09.557875+0000","last_became_active":"2024-03-21T14:12:00.529555+0000","last_became_peered":"2024-03-21T14:12:00.529555+0000","last_unstale":"2024-03-21T14:14:09.557875+0000","last_undegraded":"2024-03-21T14:14:09.557875+0000","last_fullsized":"2024-03-21T14:14:09.557875+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:17.064470+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563665+0000","last_change":"2024-03-21T14:12:55.147543+0000","last_active":"2024-03-21T14:14:09.563665+0000","last_peered":"2024-03-21T14:14:09.563665+0000","last_clean":"2024-03-21T14:14:09.563665+0000","last_became_active":"2024-03-21T14:12:00.534344+0000","last_became_peered":"2024-03-21T14:12:00.534344+0000","last_unstale":"2024-03-21T14:14:09.563665+0000","last_undegraded":"2024-03-21T14:14:09.563665+0000","last_fullsized":"2024-03-21T14:14:09.563665+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.147407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.147407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:16.590432+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":501,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561894+0000","last_change":"2024-03-21T14:12:48.244384+0000","last_active":"2024-03-21T14:14:09.561894+0000","last_peered":"2024-03-21T14:14:09.561894+0000","last_clean":"2024-03-21T14:14:09.561894+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:14:09.561894+0000","last_undegraded":"2024-03-21T14:14:09.561894+0000","last_fullsized":"2024-03-21T14:14:09.561894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:12:48.244225+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:12:48.244225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":307,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566770+0000","last_change":"2024-03-21T14:13:39.512231+0000","last_active":"2024-03-21T14:14:09.566770+0000","last_peered":"2024-03-21T14:14:09.566770+0000","last_clean":"2024-03-21T14:14:09.566770+0000","last_became_active":"2024-03-21T14:07:23.360834+0000","last_became_peered":"2024-03-21T14:07:23.360834+0000","last_unstale":"2024-03-21T14:14:09.566770+0000","last_undegraded":"2024-03-21T14:14:09.566770+0000","last_fullsized":"2024-03-21T14:14:09.566770+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.510830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.510830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:40.102365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566191+0000","last_change":"2024-03-21T14:13:53.257162+0000","last_active":"2024-03-21T14:14:09.566191+0000","last_peered":"2024-03-21T14:14:09.566191+0000","last_clean":"2024-03-21T14:14:09.566191+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:14:09.566191+0000","last_undegraded":"2024-03-21T14:14:09.566191+0000","last_fullsized":"2024-03-21T14:14:09.566191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:53.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:13:53.256723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:21.995626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566587+0000","last_change":"2024-03-21T14:13:10.251700+0000","last_active":"2024-03-21T14:14:09.566587+0000","last_peered":"2024-03-21T14:14:09.566587+0000","last_clean":"2024-03-21T14:14:09.566587+0000","last_became_active":"2024-03-21T14:12:00.528355+0000","last_became_peered":"2024-03-21T14:12:00.528355+0000","last_unstale":"2024-03-21T14:14:09.566587+0000","last_undegraded":"2024-03-21T14:14:09.566587+0000","last_fullsized":"2024-03-21T14:14:09.566587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:13:10.251587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:16.842388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":459,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.560218+0000","last_change":"2024-03-21T14:10:46.814006+0000","last_active":"2024-03-21T14:14:09.560218+0000","last_peered":"2024-03-21T14:14:09.560218+0000","last_clean":"2024-03-21T14:14:09.560218+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:14:09.560218+0000","last_undegraded":"2024-03-21T14:14:09.560218+0000","last_fullsized":"2024-03-21T14:14:09.560218+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:10:46.813854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:46.813854+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565962+0000","last_change":"2024-03-21T14:13:41.341788+0000","last_active":"2024-03-21T14:14:09.565962+0000","last_peered":"2024-03-21T14:14:09.565962+0000","last_clean":"2024-03-21T14:14:09.565962+0000","last_became_active":"2024-03-21T14:07:23.350761+0000","last_became_peered":"2024-03-21T14:07:23.350761+0000","last_unstale":"2024-03-21T14:14:09.565962+0000","last_undegraded":"2024-03-21T14:14:09.565962+0000","last_fullsized":"2024-03-21T14:14:09.565962+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:41.341673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.341673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:44.213517+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":439,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562443+0000","last_change":"2024-03-21T14:11:08.999188+0000","last_active":"2024-03-21T14:14:09.562443+0000","last_peered":"2024-03-21T14:14:09.562443+0000","last_clean":"2024-03-21T14:14:09.562443+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:14:09.562443+0000","last_undegraded":"2024-03-21T14:14:09.562443+0000","last_fullsized":"2024-03-21T14:14:09.562443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:08.999081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:11:08.999081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.562176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":454,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566093+0000","last_change":"2024-03-21T14:12:58.205869+0000","last_active":"2024-03-21T14:14:09.566093+0000","last_peered":"2024-03-21T14:14:09.566093+0000","last_clean":"2024-03-21T14:14:09.566093+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:14:09.566093+0000","last_undegraded":"2024-03-21T14:14:09.566093+0000","last_fullsized":"2024-03-21T14:14:09.566093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.205794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.205794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.257184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558360+0000","last_change":"2024-03-21T14:13:08.544451+0000","last_active":"2024-03-21T14:14:09.558360+0000","last_peered":"2024-03-21T14:14:09.558360+0000","last_clean":"2024-03-21T14:14:09.558360+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:14:09.558360+0000","last_undegraded":"2024-03-21T14:14:09.558360+0000","last_fullsized":"2024-03-21T14:14:09.558360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:13:08.544355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:08.544355+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:30.648033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":433,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558867+0000","last_change":"2024-03-21T14:12:00.533587+0000","last_active":"2024-03-21T14:14:09.558867+0000","last_peered":"2024-03-21T14:14:09.558867+0000","last_clean":"2024-03-21T14:14:09.558867+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:14:09.558867+0000","last_undegraded":"2024-03-21T14:14:09.558867+0000","last_fullsized":"2024-03-21T14:14:09.558867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:11.357287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:10:11.357287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567512+0000","last_change":"2024-03-21T14:13:36.553244+0000","last_active":"2024-03-21T14:14:09.567512+0000","last_peered":"2024-03-21T14:14:09.567512+0000","last_clean":"2024-03-21T14:14:09.567512+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:14:09.567512+0000","last_undegraded":"2024-03-21T14:14:09.567512+0000","last_fullsized":"2024-03-21T14:14:09.567512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:36.553120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.553120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.765113+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":299,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558272+0000","last_change":"2024-03-21T14:13:10.588040+0000","last_active":"2024-03-21T14:14:09.558272+0000","last_peered":"2024-03-21T14:14:09.558272+0000","last_clean":"2024-03-21T14:14:09.558272+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:14:09.558272+0000","last_undegraded":"2024-03-21T14:14:09.558272+0000","last_fullsized":"2024-03-21T14:14:09.558272+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:10.587951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:10.587951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:16.314570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'27","reported_seq":518,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.572869+0000","last_change":"2024-03-21T14:13:42.196025+0000","last_active":"2024-03-21T14:14:09.572869+0000","last_peered":"2024-03-21T14:14:09.572869+0000","last_clean":"2024-03-21T14:14:09.572869+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:14:09.572869+0000","last_undegraded":"2024-03-21T14:14:09.572869+0000","last_fullsized":"2024-03-21T14:14:09.572869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:13:42.195878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:42.195878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:42.977636+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563746+0000","last_change":"2024-03-21T14:13:02.282371+0000","last_active":"2024-03-21T14:14:09.563746+0000","last_peered":"2024-03-21T14:14:09.563746+0000","last_clean":"2024-03-21T14:14:09.563746+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:14:09.563746+0000","last_undegraded":"2024-03-21T14:14:09.563746+0000","last_fullsized":"2024-03-21T14:14:09.563746+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:02.282247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:02.282247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.670173+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":438,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558571+0000","last_change":"2024-03-21T14:14:00.679668+0000","last_active":"2024-03-21T14:14:09.558571+0000","last_peered":"2024-03-21T14:14:09.558571+0000","last_clean":"2024-03-21T14:14:09.558571+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:14:09.558571+0000","last_undegraded":"2024-03-21T14:14:09.558571+0000","last_fullsized":"2024-03-21T14:14:09.558571+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.679529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.679529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:01.166486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.560298+0000","last_change":"2024-03-21T14:10:49.867066+0000","last_active":"2024-03-21T14:14:09.560298+0000","last_peered":"2024-03-21T14:14:09.560298+0000","last_clean":"2024-03-21T14:14:09.560298+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:14:09.560298+0000","last_undegraded":"2024-03-21T14:14:09.560298+0000","last_fullsized":"2024-03-21T14:14:09.560298+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:49.866989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.866989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":483,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563411+0000","last_change":"2024-03-21T14:13:39.533070+0000","last_active":"2024-03-21T14:14:09.563411+0000","last_peered":"2024-03-21T14:14:09.563411+0000","last_clean":"2024-03-21T14:14:09.563411+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:14:09.563411+0000","last_undegraded":"2024-03-21T14:14:09.563411+0000","last_fullsized":"2024-03-21T14:14:09.563411+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:13:39.532904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.532904+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:50.781737+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":453,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561959+0000","last_change":"2024-03-21T14:14:00.547921+0000","last_active":"2024-03-21T14:14:09.561959+0000","last_peered":"2024-03-21T14:14:09.561959+0000","last_clean":"2024-03-21T14:14:09.561959+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:14:09.561959+0000","last_undegraded":"2024-03-21T14:14:09.561959+0000","last_fullsized":"2024-03-21T14:14:09.561959+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.547775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:27.636509+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":609,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.757498+0000","last_change":"2024-03-21T14:13:38.032421+0000","last_active":"2024-03-21T14:14:09.757498+0000","last_peered":"2024-03-21T14:14:09.757498+0000","last_clean":"2024-03-21T14:14:09.757498+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:14:09.757498+0000","last_undegraded":"2024-03-21T14:14:09.757498+0000","last_fullsized":"2024-03-21T14:14:09.757498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:13:38.032095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.032095+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:40.723676+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566171+0000","last_change":"2024-03-21T14:12:52.744870+0000","last_active":"2024-03-21T14:14:09.566171+0000","last_peered":"2024-03-21T14:14:09.566171+0000","last_clean":"2024-03-21T14:14:09.566171+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:14:09.566171+0000","last_undegraded":"2024-03-21T14:14:09.566171+0000","last_fullsized":"2024-03-21T14:14:09.566171+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:12:52.744765+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:12:52.744765+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.314968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":149,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565506+0000","last_change":"2024-03-21T14:13:41.601952+0000","last_active":"2024-03-21T14:14:09.565506+0000","last_peered":"2024-03-21T14:14:09.565506+0000","last_clean":"2024-03-21T14:14:09.565506+0000","last_became_active":"2024-03-21T14:12:20.564175+0000","last_became_peered":"2024-03-21T14:12:20.564175+0000","last_unstale":"2024-03-21T14:14:09.565506+0000","last_undegraded":"2024-03-21T14:14:09.565506+0000","last_fullsized":"2024-03-21T14:14:09.565506+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:13:41.601807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.601807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.904689+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":442,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567216+0000","last_change":"2024-03-21T14:13:10.633497+0000","last_active":"2024-03-21T14:14:09.567216+0000","last_peered":"2024-03-21T14:14:09.567216+0000","last_clean":"2024-03-21T14:14:09.567216+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:14:09.567216+0000","last_undegraded":"2024-03-21T14:14:09.567216+0000","last_fullsized":"2024-03-21T14:14:09.567216+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:10.633398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:10.633398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:34.869598+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":101,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565413+0000","last_change":"2024-03-21T14:12:59.820203+0000","last_active":"2024-03-21T14:14:09.565413+0000","last_peered":"2024-03-21T14:14:09.565413+0000","last_clean":"2024-03-21T14:14:09.565413+0000","last_became_active":"2024-03-21T14:12:20.562837+0000","last_became_peered":"2024-03-21T14:12:20.562837+0000","last_unstale":"2024-03-21T14:14:09.565413+0000","last_undegraded":"2024-03-21T14:14:09.565413+0000","last_fullsized":"2024-03-21T14:14:09.565413+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.819350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.819350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:20.114979+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":469,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.560859+0000","last_change":"2024-03-21T14:13:59.621648+0000","last_active":"2024-03-21T14:14:09.560859+0000","last_peered":"2024-03-21T14:14:09.560859+0000","last_clean":"2024-03-21T14:14:09.560859+0000","last_became_active":"2024-03-21T14:07:23.364680+0000","last_became_peered":"2024-03-21T14:07:23.364680+0000","last_unstale":"2024-03-21T14:14:09.560859+0000","last_undegraded":"2024-03-21T14:14:09.560859+0000","last_fullsized":"2024-03-21T14:14:09.560859+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:59.621530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.621530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:29.389896+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":324,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567050+0000","last_change":"2024-03-21T14:13:09.203461+0000","last_active":"2024-03-21T14:14:09.567050+0000","last_peered":"2024-03-21T14:14:09.567050+0000","last_clean":"2024-03-21T14:14:09.567050+0000","last_became_active":"2024-03-21T14:12:00.536809+0000","last_became_peered":"2024-03-21T14:12:00.536809+0000","last_unstale":"2024-03-21T14:14:09.567050+0000","last_undegraded":"2024-03-21T14:14:09.567050+0000","last_fullsized":"2024-03-21T14:14:09.567050+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:13:09.203367+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:13:09.203367+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:17.295555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":443,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562599+0000","last_change":"2024-03-21T14:12:48.717566+0000","last_active":"2024-03-21T14:14:09.562599+0000","last_peered":"2024-03-21T14:14:09.562599+0000","last_clean":"2024-03-21T14:14:09.562599+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:14:09.562599+0000","last_undegraded":"2024-03-21T14:14:09.562599+0000","last_fullsized":"2024-03-21T14:14:09.562599+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:48.717461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:48.717461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.562312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":128,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558317+0000","last_change":"2024-03-21T14:13:55.540387+0000","last_active":"2024-03-21T14:14:09.558317+0000","last_peered":"2024-03-21T14:14:09.558317+0000","last_clean":"2024-03-21T14:14:09.558317+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:14:09.558317+0000","last_undegraded":"2024-03-21T14:14:09.558317+0000","last_fullsized":"2024-03-21T14:14:09.558317+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:55.540139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.540139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:24.763371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":100,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566968+0000","last_change":"2024-03-21T14:13:00.210250+0000","last_active":"2024-03-21T14:14:09.566968+0000","last_peered":"2024-03-21T14:14:09.566968+0000","last_clean":"2024-03-21T14:14:09.566968+0000","last_became_active":"2024-03-21T14:12:20.560066+0000","last_became_peered":"2024-03-21T14:12:20.560066+0000","last_unstale":"2024-03-21T14:14:09.566968+0000","last_undegraded":"2024-03-21T14:14:09.566968+0000","last_fullsized":"2024-03-21T14:14:09.566968+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:00.210172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:00.210172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:17.634940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":475,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.560978+0000","last_change":"2024-03-21T14:12:07.940160+0000","last_active":"2024-03-21T14:14:09.560978+0000","last_peered":"2024-03-21T14:14:09.560978+0000","last_clean":"2024-03-21T14:14:09.560978+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:14:09.560978+0000","last_undegraded":"2024-03-21T14:14:09.560978+0000","last_fullsized":"2024-03-21T14:14:09.560978+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:12:07.940015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:07.940015+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.670105+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563352+0000","last_change":"2024-03-21T14:12:51.294104+0000","last_active":"2024-03-21T14:14:09.563352+0000","last_peered":"2024-03-21T14:14:09.563352+0000","last_clean":"2024-03-21T14:14:09.563352+0000","last_became_active":"2024-03-21T14:12:00.536387+0000","last_became_peered":"2024-03-21T14:12:00.536387+0000","last_unstale":"2024-03-21T14:14:09.563352+0000","last_undegraded":"2024-03-21T14:14:09.563352+0000","last_fullsized":"2024-03-21T14:14:09.563352+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:51.293955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:12:51.293955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:20.487194+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562988+0000","last_change":"2024-03-21T14:13:54.518871+0000","last_active":"2024-03-21T14:14:09.562988+0000","last_peered":"2024-03-21T14:14:09.562988+0000","last_clean":"2024-03-21T14:14:09.562988+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:14:09.562988+0000","last_undegraded":"2024-03-21T14:14:09.562988+0000","last_fullsized":"2024-03-21T14:14:09.562988+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:54.518724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:54.518724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:15.950041+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566385+0000","last_change":"2024-03-21T14:12:54.264971+0000","last_active":"2024-03-21T14:14:09.566385+0000","last_peered":"2024-03-21T14:14:09.566385+0000","last_clean":"2024-03-21T14:14:09.566385+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:14:09.566385+0000","last_undegraded":"2024-03-21T14:14:09.566385+0000","last_fullsized":"2024-03-21T14:14:09.566385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:12:54.264874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.264874+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:14.905283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":513,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561788+0000","last_change":"2024-03-21T14:13:55.514810+0000","last_active":"2024-03-21T14:14:09.561788+0000","last_peered":"2024-03-21T14:14:09.561788+0000","last_clean":"2024-03-21T14:14:09.561788+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:14:09.561788+0000","last_undegraded":"2024-03-21T14:14:09.561788+0000","last_fullsized":"2024-03-21T14:14:09.561788+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.514646+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:19.498523+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":100,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567024+0000","last_change":"2024-03-21T14:13:06.269597+0000","last_active":"2024-03-21T14:14:09.567024+0000","last_peered":"2024-03-21T14:14:09.567024+0000","last_clean":"2024-03-21T14:14:09.567024+0000","last_became_active":"2024-03-21T14:12:20.560601+0000","last_became_peered":"2024-03-21T14:12:20.560601+0000","last_unstale":"2024-03-21T14:14:09.567024+0000","last_undegraded":"2024-03-21T14:14:09.567024+0000","last_fullsized":"2024-03-21T14:14:09.567024+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:06.269490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:06.269490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563263+0000","last_change":"2024-03-21T14:13:48.467696+0000","last_active":"2024-03-21T14:14:09.563263+0000","last_peered":"2024-03-21T14:14:09.563263+0000","last_clean":"2024-03-21T14:14:09.563263+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:14:09.563263+0000","last_undegraded":"2024-03-21T14:14:09.563263+0000","last_fullsized":"2024-03-21T14:14:09.563263+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:48.467442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:13:48.467442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:03.673244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562415+0000","last_change":"2024-03-21T14:13:04.662406+0000","last_active":"2024-03-21T14:14:09.562415+0000","last_peered":"2024-03-21T14:14:09.562415+0000","last_clean":"2024-03-21T14:14:09.562415+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:14:09.562415+0000","last_undegraded":"2024-03-21T14:14:09.562415+0000","last_fullsized":"2024-03-21T14:14:09.562415+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:04.662327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:04.662327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":499,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.568042+0000","last_change":"2024-03-21T14:13:50.493055+0000","last_active":"2024-03-21T14:14:09.568042+0000","last_peered":"2024-03-21T14:14:09.568042+0000","last_clean":"2024-03-21T14:14:09.568042+0000","last_became_active":"2024-03-21T14:12:00.536162+0000","last_became_peered":"2024-03-21T14:12:00.536162+0000","last_unstale":"2024-03-21T14:14:09.568042+0000","last_undegraded":"2024-03-21T14:14:09.568042+0000","last_fullsized":"2024-03-21T14:14:09.568042+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.492830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.492830+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:18.194658+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":102,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566059+0000","last_change":"2024-03-21T14:13:01.256791+0000","last_active":"2024-03-21T14:14:09.566059+0000","last_peered":"2024-03-21T14:14:09.566059+0000","last_clean":"2024-03-21T14:14:09.566059+0000","last_became_active":"2024-03-21T14:12:20.561889+0000","last_became_peered":"2024-03-21T14:12:20.561889+0000","last_unstale":"2024-03-21T14:14:09.566059+0000","last_undegraded":"2024-03-21T14:14:09.566059+0000","last_fullsized":"2024-03-21T14:14:09.566059+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:01.256711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:01.256711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:28.461256+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567836+0000","last_change":"2024-03-21T14:13:56.430898+0000","last_active":"2024-03-21T14:14:09.567836+0000","last_peered":"2024-03-21T14:14:09.567836+0000","last_clean":"2024-03-21T14:14:09.567836+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:14:09.567836+0000","last_undegraded":"2024-03-21T14:14:09.567836+0000","last_fullsized":"2024-03-21T14:14:09.567836+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:56.430507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:56.430507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:05.852233+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":460,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558327+0000","last_change":"2024-03-21T14:12:00.531790+0000","last_active":"2024-03-21T14:14:09.558327+0000","last_peered":"2024-03-21T14:14:09.558327+0000","last_clean":"2024-03-21T14:14:09.558327+0000","last_became_active":"2024-03-21T14:12:00.531406+0000","last_became_peered":"2024-03-21T14:12:00.531406+0000","last_unstale":"2024-03-21T14:14:09.558327+0000","last_undegraded":"2024-03-21T14:14:09.558327+0000","last_fullsized":"2024-03-21T14:14:09.558327+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:42.706023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:42.706023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":496,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558455+0000","last_change":"2024-03-21T14:10:54.800377+0000","last_active":"2024-03-21T14:14:09.558455+0000","last_peered":"2024-03-21T14:14:09.558455+0000","last_clean":"2024-03-21T14:14:09.558455+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:14:09.558455+0000","last_undegraded":"2024-03-21T14:14:09.558455+0000","last_fullsized":"2024-03-21T14:14:09.558455+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-03-21T14:10:54.800107+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:10:54.800107+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":300,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565143+0000","last_change":"2024-03-21T14:11:06.009897+0000","last_active":"2024-03-21T14:14:09.565143+0000","last_peered":"2024-03-21T14:14:09.565143+0000","last_clean":"2024-03-21T14:14:09.565143+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:14:09.565143+0000","last_undegraded":"2024-03-21T14:14:09.565143+0000","last_fullsized":"2024-03-21T14:14:09.565143+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:06.009710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:11:06.009710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.562205+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":602,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566670+0000","last_change":"2024-03-21T14:13:34.317612+0000","last_active":"2024-03-21T14:14:09.566670+0000","last_peered":"2024-03-21T14:14:09.566670+0000","last_clean":"2024-03-21T14:14:09.566670+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:14:09.566670+0000","last_undegraded":"2024-03-21T14:14:09.566670+0000","last_fullsized":"2024-03-21T14:14:09.566670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:13:34.317492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:34.317492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:47.112660+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":464,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562040+0000","last_change":"2024-03-21T14:13:34.724463+0000","last_active":"2024-03-21T14:14:09.562040+0000","last_peered":"2024-03-21T14:14:09.562040+0000","last_clean":"2024-03-21T14:14:09.562040+0000","last_became_active":"2024-03-21T14:08:17.142668+0000","last_became_peered":"2024-03-21T14:08:17.142668+0000","last_unstale":"2024-03-21T14:14:09.562040+0000","last_undegraded":"2024-03-21T14:14:09.562040+0000","last_fullsized":"2024-03-21T14:14:09.562040+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:34.724192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:13:34.724192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:45.746660+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":478,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567633+0000","last_change":"2024-03-21T14:12:08.873079+0000","last_active":"2024-03-21T14:14:09.567633+0000","last_peered":"2024-03-21T14:14:09.567633+0000","last_clean":"2024-03-21T14:14:09.567633+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:14:09.567633+0000","last_undegraded":"2024-03-21T14:14:09.567633+0000","last_fullsized":"2024-03-21T14:14:09.567633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'15","last_scrub_stamp":"2024-03-21T14:12:08.872861+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:12:08.872861+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.670079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":434,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.559026+0000","last_change":"2024-03-21T14:12:58.643025+0000","last_active":"2024-03-21T14:14:09.559026+0000","last_peered":"2024-03-21T14:14:09.559026+0000","last_clean":"2024-03-21T14:14:09.559026+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:14:09.559026+0000","last_undegraded":"2024-03-21T14:14:09.559026+0000","last_fullsized":"2024-03-21T14:14:09.559026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:23.989215+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"130'9","reported_seq":419,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561860+0000","last_change":"2024-03-21T14:13:03.639121+0000","last_active":"2024-03-21T14:14:09.561860+0000","last_peered":"2024-03-21T14:14:09.561860+0000","last_clean":"2024-03-21T14:14:09.561860+0000","last_became_active":"2024-03-21T14:07:23.354945+0000","last_became_peered":"2024-03-21T14:07:23.354945+0000","last_unstale":"2024-03-21T14:14:09.561860+0000","last_undegraded":"2024-03-21T14:14:09.561860+0000","last_fullsized":"2024-03-21T14:14:09.561860+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:13:03.638324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.638324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.770723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":256,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562264+0000","last_change":"2024-03-21T14:13:44.667633+0000","last_active":"2024-03-21T14:14:09.562264+0000","last_peered":"2024-03-21T14:14:09.562264+0000","last_clean":"2024-03-21T14:14:09.562264+0000","last_became_active":"2024-03-21T14:12:20.562532+0000","last_became_peered":"2024-03-21T14:12:20.562532+0000","last_unstale":"2024-03-21T14:14:09.562264+0000","last_undegraded":"2024-03-21T14:14:09.562264+0000","last_fullsized":"2024-03-21T14:14:09.562264+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:44.667480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:44.667480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:46.142833+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":475,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566913+0000","last_change":"2024-03-21T14:13:52.240889+0000","last_active":"2024-03-21T14:14:09.566913+0000","last_peered":"2024-03-21T14:14:09.566913+0000","last_clean":"2024-03-21T14:14:09.566913+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:14:09.566913+0000","last_undegraded":"2024-03-21T14:14:09.566913+0000","last_fullsized":"2024-03-21T14:14:09.566913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:52.240603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:52.240603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:58.617545+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":451,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.573162+0000","last_change":"2024-03-21T14:12:53.155160+0000","last_active":"2024-03-21T14:14:09.573162+0000","last_peered":"2024-03-21T14:14:09.573162+0000","last_clean":"2024-03-21T14:14:09.573162+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:14:09.573162+0000","last_undegraded":"2024-03-21T14:14:09.573162+0000","last_fullsized":"2024-03-21T14:14:09.573162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:53.155075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.155075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:20.958924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":299,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.569358+0000","last_change":"2024-03-21T14:12:56.663622+0000","last_active":"2024-03-21T14:14:09.569358+0000","last_peered":"2024-03-21T14:14:09.569358+0000","last_clean":"2024-03-21T14:14:09.569358+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:14:09.569358+0000","last_undegraded":"2024-03-21T14:14:09.569358+0000","last_fullsized":"2024-03-21T14:14:09.569358+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:56.663519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:12:56.663519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561840+0000","last_change":"2024-03-21T14:12:00.761561+0000","last_active":"2024-03-21T14:14:09.561840+0000","last_peered":"2024-03-21T14:14:09.561840+0000","last_clean":"2024-03-21T14:14:09.561840+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:14:09.561840+0000","last_undegraded":"2024-03-21T14:14:09.561840+0000","last_fullsized":"2024-03-21T14:14:09.561840+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:00.761483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:00.761483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.562261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":144,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563804+0000","last_change":"2024-03-21T14:13:37.488188+0000","last_active":"2024-03-21T14:14:09.563804+0000","last_peered":"2024-03-21T14:14:09.563804+0000","last_clean":"2024-03-21T14:14:09.563804+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:14:09.563804+0000","last_undegraded":"2024-03-21T14:14:09.563804+0000","last_fullsized":"2024-03-21T14:14:09.563804+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:37.488036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:37.488036+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:47.071726+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":449,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562316+0000","last_change":"2024-03-21T14:13:38.740375+0000","last_active":"2024-03-21T14:14:09.562316+0000","last_peered":"2024-03-21T14:14:09.562316+0000","last_clean":"2024-03-21T14:14:09.562316+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:14:09.562316+0000","last_undegraded":"2024-03-21T14:14:09.562316+0000","last_fullsized":"2024-03-21T14:14:09.562316+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:38.740248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.740248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:40.988073+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":446,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566286+0000","last_change":"2024-03-21T14:12:57.241462+0000","last_active":"2024-03-21T14:14:09.566286+0000","last_peered":"2024-03-21T14:14:09.566286+0000","last_clean":"2024-03-21T14:14:09.566286+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:14:09.566286+0000","last_undegraded":"2024-03-21T14:14:09.566286+0000","last_fullsized":"2024-03-21T14:14:09.566286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:57.241377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:12:57.241377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:18.647747+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":477,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.757759+0000","last_change":"2024-03-21T14:13:39.023162+0000","last_active":"2024-03-21T14:14:09.757759+0000","last_peered":"2024-03-21T14:14:09.757759+0000","last_clean":"2024-03-21T14:14:09.757759+0000","last_became_active":"2024-03-21T14:07:23.354457+0000","last_became_peered":"2024-03-21T14:07:23.354457+0000","last_unstale":"2024-03-21T14:14:09.757759+0000","last_undegraded":"2024-03-21T14:14:09.757759+0000","last_fullsized":"2024-03-21T14:14:09.757759+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.023049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.023049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:04.473366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":353,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561832+0000","last_change":"2024-03-21T14:13:59.568156+0000","last_active":"2024-03-21T14:14:09.561832+0000","last_peered":"2024-03-21T14:14:09.561832+0000","last_clean":"2024-03-21T14:14:09.561832+0000","last_became_active":"2024-03-21T14:07:23.365278+0000","last_became_peered":"2024-03-21T14:07:23.365278+0000","last_unstale":"2024-03-21T14:14:09.561832+0000","last_undegraded":"2024-03-21T14:14:09.561832+0000","last_fullsized":"2024-03-21T14:14:09.561832+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:13:59.567870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.567870+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:59.574445+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"265'5719","reported_seq":5771,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.527182+0000","last_change":"2024-03-21T14:13:09.117421+0000","last_active":"2024-03-21T14:14:11.527182+0000","last_peered":"2024-03-21T14:14:11.527182+0000","last_clean":"2024-03-21T14:14:11.527182+0000","last_became_active":"2024-03-21T14:13:09.117000+0000","last_became_peered":"2024-03-21T14:13:09.117000+0000","last_unstale":"2024-03-21T14:14:11.527182+0000","last_undegraded":"2024-03-21T14:14:11.527182+0000","last_fullsized":"2024-03-21T14:14:11.527182+0000","mapping_epoch":244,"log_start":"251'800","ondisk_log_start":"251'800","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4919,"log_dups_size":0,"ondisk_log_size":4919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:22.059734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374800384,"num_objects":5719,"num_object_clones":0,"num_object_copies":17157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5719,"num_write_kb":366016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566143+0000","last_change":"2024-03-21T14:13:36.400555+0000","last_active":"2024-03-21T14:14:09.566143+0000","last_peered":"2024-03-21T14:14:09.566143+0000","last_clean":"2024-03-21T14:14:09.566143+0000","last_became_active":"2024-03-21T14:13:18.544191+0000","last_became_peered":"2024-03-21T14:13:18.544191+0000","last_unstale":"2024-03-21T14:14:09.566143+0000","last_undegraded":"2024-03-21T14:14:09.566143+0000","last_fullsized":"2024-03-21T14:14:09.566143+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:36.400445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.400445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:58.782401+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":480,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561811+0000","last_change":"2024-03-21T14:12:57.742705+0000","last_active":"2024-03-21T14:14:09.561811+0000","last_peered":"2024-03-21T14:14:09.561811+0000","last_clean":"2024-03-21T14:14:09.561811+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:14:09.561811+0000","last_undegraded":"2024-03-21T14:14:09.561811+0000","last_fullsized":"2024-03-21T14:14:09.561811+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:12:57.742582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:57.742582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:19.570362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":516,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566867+0000","last_change":"2024-03-21T14:13:05.267472+0000","last_active":"2024-03-21T14:14:09.566867+0000","last_peered":"2024-03-21T14:14:09.566867+0000","last_clean":"2024-03-21T14:14:09.566867+0000","last_became_active":"2024-03-21T14:12:00.537606+0000","last_became_peered":"2024-03-21T14:12:00.537606+0000","last_unstale":"2024-03-21T14:14:09.566867+0000","last_undegraded":"2024-03-21T14:14:09.566867+0000","last_fullsized":"2024-03-21T14:14:09.566867+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:13:05.267363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:05.267363+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:19.943184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"214'9","reported_seq":606,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562056+0000","last_change":"2024-03-21T14:14:07.574814+0000","last_active":"2024-03-21T14:14:09.562056+0000","last_peered":"2024-03-21T14:14:09.562056+0000","last_clean":"2024-03-21T14:14:09.562056+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:14:09.562056+0000","last_undegraded":"2024-03-21T14:14:09.562056+0000","last_fullsized":"2024-03-21T14:14:09.562056+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'9","last_scrub_stamp":"2024-03-21T14:14:07.574692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.574692+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:24.335788+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"265'5754","reported_seq":5806,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:12.151595+0000","last_change":"2024-03-21T14:13:09.392712+0000","last_active":"2024-03-21T14:14:12.151595+0000","last_peered":"2024-03-21T14:14:12.151595+0000","last_clean":"2024-03-21T14:14:12.151595+0000","last_became_active":"2024-03-21T14:13:09.392115+0000","last_became_peered":"2024-03-21T14:13:09.392115+0000","last_unstale":"2024-03-21T14:14:12.151595+0000","last_undegraded":"2024-03-21T14:14:12.151595+0000","last_fullsized":"2024-03-21T14:14:12.151595+0000","mapping_epoch":244,"log_start":"251'2000","ondisk_log_start":"251'2000","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3754,"log_dups_size":0,"ondisk_log_size":3754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:15.514518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377094144,"num_objects":5754,"num_object_clones":0,"num_object_copies":17262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5754,"num_write_kb":368256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":186,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563027+0000","last_change":"2024-03-21T14:14:01.739227+0000","last_active":"2024-03-21T14:14:09.563027+0000","last_peered":"2024-03-21T14:14:09.563027+0000","last_clean":"2024-03-21T14:14:09.563027+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:14:09.563027+0000","last_undegraded":"2024-03-21T14:14:09.563027+0000","last_fullsized":"2024-03-21T14:14:09.563027+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:01.739104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:01.739104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:30.893064+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":464,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565316+0000","last_change":"2024-03-21T14:13:45.695342+0000","last_active":"2024-03-21T14:14:09.565316+0000","last_peered":"2024-03-21T14:14:09.565316+0000","last_clean":"2024-03-21T14:14:09.565316+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:14:09.565316+0000","last_undegraded":"2024-03-21T14:14:09.565316+0000","last_fullsized":"2024-03-21T14:14:09.565316+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:45.695201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:45.695201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:05.891374+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":511,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558930+0000","last_change":"2024-03-21T14:13:36.652349+0000","last_active":"2024-03-21T14:14:09.558930+0000","last_peered":"2024-03-21T14:14:09.558930+0000","last_clean":"2024-03-21T14:14:09.558930+0000","last_became_active":"2024-03-21T14:12:00.540768+0000","last_became_peered":"2024-03-21T14:12:00.540768+0000","last_unstale":"2024-03-21T14:14:09.558930+0000","last_undegraded":"2024-03-21T14:14:09.558930+0000","last_fullsized":"2024-03-21T14:14:09.558930+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:13:36.652162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.652162+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:00.176123+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":464,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.570431+0000","last_change":"2024-03-21T14:12:54.200661+0000","last_active":"2024-03-21T14:14:09.570431+0000","last_peered":"2024-03-21T14:14:09.570431+0000","last_clean":"2024-03-21T14:14:09.570431+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:14:09.570431+0000","last_undegraded":"2024-03-21T14:14:09.570431+0000","last_fullsized":"2024-03-21T14:14:09.570431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:54.200554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.200554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.813039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"265'5823","reported_seq":5875,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.524411+0000","last_change":"2024-03-21T14:13:08.815145+0000","last_active":"2024-03-21T14:14:11.524411+0000","last_peered":"2024-03-21T14:14:11.524411+0000","last_clean":"2024-03-21T14:14:11.524411+0000","last_became_active":"2024-03-21T14:13:08.814832+0000","last_became_peered":"2024-03-21T14:13:08.814832+0000","last_unstale":"2024-03-21T14:14:11.524411+0000","last_undegraded":"2024-03-21T14:14:11.524411+0000","last_fullsized":"2024-03-21T14:14:11.524411+0000","mapping_epoch":244,"log_start":"251'900","ondisk_log_start":"251'900","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4923,"log_dups_size":0,"ondisk_log_size":4923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:11.996325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381616128,"num_objects":5823,"num_object_clones":0,"num_object_copies":17469,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5823,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5823,"num_write_kb":372672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":444,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562679+0000","last_change":"2024-03-21T14:12:54.779559+0000","last_active":"2024-03-21T14:14:09.562679+0000","last_peered":"2024-03-21T14:14:09.562679+0000","last_clean":"2024-03-21T14:14:09.562679+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:14:09.562679+0000","last_undegraded":"2024-03-21T14:14:09.562679+0000","last_fullsized":"2024-03-21T14:14:09.562679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:54.779479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.779479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:17.053564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":464,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.573391+0000","last_change":"2024-03-21T14:13:43.154627+0000","last_active":"2024-03-21T14:14:09.573391+0000","last_peered":"2024-03-21T14:14:09.573391+0000","last_clean":"2024-03-21T14:14:09.573391+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:14:09.573391+0000","last_undegraded":"2024-03-21T14:14:09.573391+0000","last_fullsized":"2024-03-21T14:14:09.573391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:43.154506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.154506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:57.654214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":456,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562785+0000","last_change":"2024-03-21T14:13:48.637245+0000","last_active":"2024-03-21T14:14:09.562785+0000","last_peered":"2024-03-21T14:14:09.562785+0000","last_clean":"2024-03-21T14:14:09.562785+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:14:09.562785+0000","last_undegraded":"2024-03-21T14:14:09.562785+0000","last_fullsized":"2024-03-21T14:14:09.562785+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:48.637120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:48.637120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:08.082382+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":404,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567621+0000","last_change":"2024-03-21T14:13:57.469144+0000","last_active":"2024-03-21T14:14:09.567621+0000","last_peered":"2024-03-21T14:14:09.567621+0000","last_clean":"2024-03-21T14:14:09.567621+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:14:09.567621+0000","last_undegraded":"2024-03-21T14:14:09.567621+0000","last_fullsized":"2024-03-21T14:14:09.567621+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:13:57.468072+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:11:10.962498+0000","last_clean_scrub_stamp":"2024-03-21T14:13:57.468072+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:05.076240+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"265'5830","reported_seq":5882,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.521585+0000","last_change":"2024-03-21T14:13:08.812607+0000","last_active":"2024-03-21T14:14:11.521585+0000","last_peered":"2024-03-21T14:14:11.521585+0000","last_clean":"2024-03-21T14:14:11.521585+0000","last_became_active":"2024-03-21T14:13:08.812224+0000","last_became_peered":"2024-03-21T14:13:08.812224+0000","last_unstale":"2024-03-21T14:14:11.521585+0000","last_undegraded":"2024-03-21T14:14:11.521585+0000","last_fullsized":"2024-03-21T14:14:11.521585+0000","mapping_epoch":244,"log_start":"251'900","ondisk_log_start":"251'900","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4930,"log_dups_size":0,"ondisk_log_size":4930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:31.084518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382074880,"num_objects":5830,"num_object_clones":0,"num_object_copies":17490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5830,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5830,"num_write_kb":373120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":301,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561471+0000","last_change":"2024-03-21T14:12:08.632071+0000","last_active":"2024-03-21T14:14:09.561471+0000","last_peered":"2024-03-21T14:14:09.561471+0000","last_clean":"2024-03-21T14:14:09.561471+0000","last_became_active":"2024-03-21T14:07:23.354443+0000","last_became_peered":"2024-03-21T14:07:23.354443+0000","last_unstale":"2024-03-21T14:14:09.561471+0000","last_undegraded":"2024-03-21T14:14:09.561471+0000","last_fullsized":"2024-03-21T14:14:09.561471+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:08.631976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:12:08.631976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.562287+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":464,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.572299+0000","last_change":"2024-03-21T14:13:00.119409+0000","last_active":"2024-03-21T14:14:09.572299+0000","last_peered":"2024-03-21T14:14:09.572299+0000","last_clean":"2024-03-21T14:14:09.572299+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:14:09.572299+0000","last_undegraded":"2024-03-21T14:14:09.572299+0000","last_fullsized":"2024-03-21T14:14:09.572299+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:00.119325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:13:00.119325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:22.659756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566182+0000","last_change":"2024-03-21T14:13:03.296024+0000","last_active":"2024-03-21T14:14:09.566182+0000","last_peered":"2024-03-21T14:14:09.566182+0000","last_clean":"2024-03-21T14:14:09.566182+0000","last_became_active":"2024-03-21T14:12:00.530707+0000","last_became_peered":"2024-03-21T14:12:00.530707+0000","last_unstale":"2024-03-21T14:14:09.566182+0000","last_undegraded":"2024-03-21T14:14:09.566182+0000","last_fullsized":"2024-03-21T14:14:09.566182+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:03.295889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.295889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:22.550796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":157,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.573634+0000","last_change":"2024-03-21T14:13:50.159521+0000","last_active":"2024-03-21T14:14:09.573634+0000","last_peered":"2024-03-21T14:14:09.573634+0000","last_clean":"2024-03-21T14:14:09.573634+0000","last_became_active":"2024-03-21T14:12:20.560843+0000","last_became_peered":"2024-03-21T14:12:20.560843+0000","last_unstale":"2024-03-21T14:14:09.573634+0000","last_undegraded":"2024-03-21T14:14:09.573634+0000","last_fullsized":"2024-03-21T14:14:09.573634+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.159386+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:54.974885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.9","version":"265'5842","reported_seq":5894,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.453957+0000","last_change":"2024-03-21T14:13:09.117304+0000","last_active":"2024-03-21T14:14:11.453957+0000","last_peered":"2024-03-21T14:14:11.453957+0000","last_clean":"2024-03-21T14:14:11.453957+0000","last_became_active":"2024-03-21T14:13:09.116976+0000","last_became_peered":"2024-03-21T14:13:09.116976+0000","last_unstale":"2024-03-21T14:14:11.453957+0000","last_undegraded":"2024-03-21T14:14:11.453957+0000","last_fullsized":"2024-03-21T14:14:11.453957+0000","mapping_epoch":244,"log_start":"251'1600","ondisk_log_start":"251'1600","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:24.568782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382861312,"num_objects":5842,"num_object_clones":0,"num_object_copies":17526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5842,"num_write_kb":373888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":436,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563892+0000","last_change":"2024-03-21T14:12:09.871728+0000","last_active":"2024-03-21T14:14:09.563892+0000","last_peered":"2024-03-21T14:14:09.563892+0000","last_clean":"2024-03-21T14:14:09.563892+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:14:09.563892+0000","last_undegraded":"2024-03-21T14:14:09.563892+0000","last_fullsized":"2024-03-21T14:14:09.563892+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:09.871639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:09.871639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.670050+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":484,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565824+0000","last_change":"2024-03-21T14:13:48.229671+0000","last_active":"2024-03-21T14:14:09.565824+0000","last_peered":"2024-03-21T14:14:09.565824+0000","last_clean":"2024-03-21T14:14:09.565824+0000","last_became_active":"2024-03-21T14:12:00.538240+0000","last_became_peered":"2024-03-21T14:12:00.538240+0000","last_unstale":"2024-03-21T14:14:09.565824+0000","last_undegraded":"2024-03-21T14:14:09.565824+0000","last_fullsized":"2024-03-21T14:14:09.565824+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:13:48.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:13:48.229489+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:14.531882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":453,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558174+0000","last_change":"2024-03-21T14:14:05.796577+0000","last_active":"2024-03-21T14:14:09.558174+0000","last_peered":"2024-03-21T14:14:09.558174+0000","last_clean":"2024-03-21T14:14:09.558174+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:14:09.558174+0000","last_undegraded":"2024-03-21T14:14:09.558174+0000","last_fullsized":"2024-03-21T14:14:09.558174+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:05.796461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:05.796461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:33.125930+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":621,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565827+0000","last_change":"2024-03-21T14:12:51.291020+0000","last_active":"2024-03-21T14:14:09.565827+0000","last_peered":"2024-03-21T14:14:09.565827+0000","last_clean":"2024-03-21T14:14:09.565827+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:14:09.565827+0000","last_undegraded":"2024-03-21T14:14:09.565827+0000","last_fullsized":"2024-03-21T14:14:09.565827+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-03-21T14:12:51.290920+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:12:51.290920+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:16.010448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":315,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567604+0000","last_change":"2024-03-21T14:13:41.480464+0000","last_active":"2024-03-21T14:14:09.567604+0000","last_peered":"2024-03-21T14:14:09.567604+0000","last_clean":"2024-03-21T14:14:09.567604+0000","last_became_active":"2024-03-21T14:12:00.526762+0000","last_became_peered":"2024-03-21T14:12:00.526762+0000","last_unstale":"2024-03-21T14:14:09.567604+0000","last_undegraded":"2024-03-21T14:14:09.567604+0000","last_fullsized":"2024-03-21T14:14:09.567604+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:41.480136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.480136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:01.599929+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.a","version":"265'5847","reported_seq":5897,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.538805+0000","last_change":"2024-03-21T14:13:08.814099+0000","last_active":"2024-03-21T14:14:11.538805+0000","last_peered":"2024-03-21T14:14:11.538805+0000","last_clean":"2024-03-21T14:14:11.538805+0000","last_became_active":"2024-03-21T14:13:08.813800+0000","last_became_peered":"2024-03-21T14:13:08.813800+0000","last_unstale":"2024-03-21T14:14:11.538805+0000","last_undegraded":"2024-03-21T14:14:11.538805+0000","last_fullsized":"2024-03-21T14:14:11.538805+0000","mapping_epoch":244,"log_start":"251'1000","ondisk_log_start":"251'1000","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4847,"log_dups_size":0,"ondisk_log_size":4847,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:18.317243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383188992,"num_objects":5847,"num_object_clones":0,"num_object_copies":17541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5847,"num_write_kb":374208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562863+0000","last_change":"2024-03-21T14:13:37.711491+0000","last_active":"2024-03-21T14:14:09.562863+0000","last_peered":"2024-03-21T14:14:09.562863+0000","last_clean":"2024-03-21T14:14:09.562863+0000","last_became_active":"2024-03-21T14:12:00.544093+0000","last_became_peered":"2024-03-21T14:12:00.544093+0000","last_unstale":"2024-03-21T14:14:09.562863+0000","last_undegraded":"2024-03-21T14:14:09.562863+0000","last_fullsized":"2024-03-21T14:14:09.562863+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:37.711376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:37.711376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:57.687359+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.569241+0000","last_change":"2024-03-21T14:12:58.667273+0000","last_active":"2024-03-21T14:14:09.569241+0000","last_peered":"2024-03-21T14:14:09.569241+0000","last_clean":"2024-03-21T14:14:09.569241+0000","last_became_active":"2024-03-21T14:12:00.535428+0000","last_became_peered":"2024-03-21T14:12:00.535428+0000","last_unstale":"2024-03-21T14:14:09.569241+0000","last_undegraded":"2024-03-21T14:14:09.569241+0000","last_fullsized":"2024-03-21T14:14:09.569241+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.667132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.667132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:19.150029+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":489,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.572200+0000","last_change":"2024-03-21T14:14:06.212889+0000","last_active":"2024-03-21T14:14:09.572200+0000","last_peered":"2024-03-21T14:14:09.572200+0000","last_clean":"2024-03-21T14:14:09.572200+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:14:09.572200+0000","last_undegraded":"2024-03-21T14:14:09.572200+0000","last_fullsized":"2024-03-21T14:14:09.572200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.212763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.212763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:35.487106+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":523,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.568127+0000","last_change":"2024-03-21T14:13:59.418320+0000","last_active":"2024-03-21T14:14:09.568127+0000","last_peered":"2024-03-21T14:14:09.568127+0000","last_clean":"2024-03-21T14:14:09.568127+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:14:09.568127+0000","last_undegraded":"2024-03-21T14:14:09.568127+0000","last_fullsized":"2024-03-21T14:14:09.568127+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:13:59.418054+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.418054+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:09.213398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":472,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.573848+0000","last_change":"2024-03-21T14:14:07.244767+0000","last_active":"2024-03-21T14:14:09.573848+0000","last_peered":"2024-03-21T14:14:09.573848+0000","last_clean":"2024-03-21T14:14:09.573848+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:14:09.573848+0000","last_undegraded":"2024-03-21T14:14:09.573848+0000","last_fullsized":"2024-03-21T14:14:09.573848+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:07.244640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.244640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:30.259905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"265'5798","reported_seq":5850,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.472064+0000","last_change":"2024-03-21T14:13:09.117302+0000","last_active":"2024-03-21T14:14:11.472064+0000","last_peered":"2024-03-21T14:14:11.472064+0000","last_clean":"2024-03-21T14:14:11.472064+0000","last_became_active":"2024-03-21T14:13:09.116993+0000","last_became_peered":"2024-03-21T14:13:09.116993+0000","last_unstale":"2024-03-21T14:14:11.472064+0000","last_undegraded":"2024-03-21T14:14:11.472064+0000","last_fullsized":"2024-03-21T14:14:11.472064+0000","mapping_epoch":244,"log_start":"251'900","ondisk_log_start":"251'900","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:16.279715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379977728,"num_objects":5798,"num_object_clones":0,"num_object_copies":17394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5798,"num_write_kb":371072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":443,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566843+0000","last_change":"2024-03-21T14:13:43.362540+0000","last_active":"2024-03-21T14:14:09.566843+0000","last_peered":"2024-03-21T14:14:09.566843+0000","last_clean":"2024-03-21T14:14:09.566843+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:14:09.566843+0000","last_undegraded":"2024-03-21T14:14:09.566843+0000","last_fullsized":"2024-03-21T14:14:09.566843+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:43.362326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.362326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:09.710464+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":742,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562883+0000","last_change":"2024-03-21T14:12:03.542289+0000","last_active":"2024-03-21T14:14:09.562883+0000","last_peered":"2024-03-21T14:14:09.562883+0000","last_clean":"2024-03-21T14:14:09.562883+0000","last_became_active":"2024-03-21T14:12:03.541331+0000","last_became_peered":"2024-03-21T14:12:03.541331+0000","last_unstale":"2024-03-21T14:14:09.562883+0000","last_undegraded":"2024-03-21T14:14:09.562883+0000","last_fullsized":"2024-03-21T14:14:09.562883+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:11:09.977051+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:11:09.977051+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.562235+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561269+0000","last_change":"2024-03-21T14:12:53.182676+0000","last_active":"2024-03-21T14:14:09.561269+0000","last_peered":"2024-03-21T14:14:09.561269+0000","last_clean":"2024-03-21T14:14:09.561269+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:14:09.561269+0000","last_undegraded":"2024-03-21T14:14:09.561269+0000","last_fullsized":"2024-03-21T14:14:09.561269+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:53.182591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.182591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.670150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":226,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558576+0000","last_change":"2024-03-21T14:12:00.597903+0000","last_active":"2024-03-21T14:14:09.558576+0000","last_peered":"2024-03-21T14:14:09.558576+0000","last_clean":"2024-03-21T14:14:09.558576+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:14:09.558576+0000","last_undegraded":"2024-03-21T14:14:09.558576+0000","last_fullsized":"2024-03-21T14:14:09.558576+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:10:55.042070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.042070+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":448,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562017+0000","last_change":"2024-03-21T14:12:55.820482+0000","last_active":"2024-03-21T14:14:09.562017+0000","last_peered":"2024-03-21T14:14:09.562017+0000","last_clean":"2024-03-21T14:14:09.562017+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:14:09.562017+0000","last_undegraded":"2024-03-21T14:14:09.562017+0000","last_fullsized":"2024-03-21T14:14:09.562017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.820408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.820408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"265'5741","reported_seq":5793,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.486543+0000","last_change":"2024-03-21T14:13:08.815820+0000","last_active":"2024-03-21T14:14:11.486543+0000","last_peered":"2024-03-21T14:14:11.486543+0000","last_clean":"2024-03-21T14:14:11.486543+0000","last_became_active":"2024-03-21T14:13:08.815075+0000","last_became_peered":"2024-03-21T14:13:08.815075+0000","last_unstale":"2024-03-21T14:14:11.486543+0000","last_undegraded":"2024-03-21T14:14:11.486543+0000","last_fullsized":"2024-03-21T14:14:11.486543+0000","mapping_epoch":244,"log_start":"251'900","ondisk_log_start":"251'900","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4841,"log_dups_size":0,"ondisk_log_size":4841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:21.390950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376242176,"num_objects":5741,"num_object_clones":0,"num_object_copies":17223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5741,"num_write_kb":367424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.570005+0000","last_change":"2024-03-21T14:13:18.721382+0000","last_active":"2024-03-21T14:14:09.570005+0000","last_peered":"2024-03-21T14:14:09.570005+0000","last_clean":"2024-03-21T14:14:09.570005+0000","last_became_active":"2024-03-21T14:13:18.721013+0000","last_became_peered":"2024-03-21T14:13:18.721013+0000","last_unstale":"2024-03-21T14:14:09.570005+0000","last_undegraded":"2024-03-21T14:14:09.570005+0000","last_fullsized":"2024-03-21T14:14:09.570005+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:13:04.716819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:23.006548+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":248,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558782+0000","last_change":"2024-03-21T14:12:20.560104+0000","last_active":"2024-03-21T14:14:09.558782+0000","last_peered":"2024-03-21T14:14:09.558782+0000","last_clean":"2024-03-21T14:14:09.558782+0000","last_became_active":"2024-03-21T14:12:20.559683+0000","last_became_peered":"2024-03-21T14:12:20.559683+0000","last_unstale":"2024-03-21T14:14:09.558782+0000","last_undegraded":"2024-03-21T14:14:09.558782+0000","last_fullsized":"2024-03-21T14:14:09.558782+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:11.081667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:11:11.081667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":1159,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:10.414422+0000","last_change":"2024-03-21T14:13:07.680778+0000","last_active":"2024-03-21T14:14:10.414422+0000","last_peered":"2024-03-21T14:14:10.414422+0000","last_clean":"2024-03-21T14:14:10.414422+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:14:10.414422+0000","last_undegraded":"2024-03-21T14:14:10.414422+0000","last_fullsized":"2024-03-21T14:14:10.414422+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:13:07.680672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.680672+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:33.414349+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":691,"num_read_kb":686,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":377,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.575877+0000","last_change":"2024-03-21T14:13:43.544913+0000","last_active":"2024-03-21T14:14:09.575877+0000","last_peered":"2024-03-21T14:14:09.575877+0000","last_clean":"2024-03-21T14:14:09.575877+0000","last_became_active":"2024-03-21T14:07:23.363490+0000","last_became_peered":"2024-03-21T14:07:23.363490+0000","last_unstale":"2024-03-21T14:14:09.575877+0000","last_undegraded":"2024-03-21T14:14:09.575877+0000","last_fullsized":"2024-03-21T14:14:09.575877+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:13:43.544671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.544671+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:51.992091+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567144+0000","last_change":"2024-03-21T14:13:39.314326+0000","last_active":"2024-03-21T14:14:09.567144+0000","last_peered":"2024-03-21T14:14:09.567144+0000","last_clean":"2024-03-21T14:14:09.567144+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:14:09.567144+0000","last_undegraded":"2024-03-21T14:14:09.567144+0000","last_fullsized":"2024-03-21T14:14:09.567144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.313966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.313966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:51.859924+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"265'5788","reported_seq":5840,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.459163+0000","last_change":"2024-03-21T14:13:09.392683+0000","last_active":"2024-03-21T14:14:11.459163+0000","last_peered":"2024-03-21T14:14:11.459163+0000","last_clean":"2024-03-21T14:14:11.459163+0000","last_became_active":"2024-03-21T14:13:09.392267+0000","last_became_peered":"2024-03-21T14:13:09.392267+0000","last_unstale":"2024-03-21T14:14:11.459163+0000","last_undegraded":"2024-03-21T14:14:11.459163+0000","last_fullsized":"2024-03-21T14:14:11.459163+0000","mapping_epoch":244,"log_start":"251'1600","ondisk_log_start":"251'1600","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:30.908283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379322368,"num_objects":5788,"num_object_clones":0,"num_object_copies":17364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5788,"num_write_kb":370432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":449,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566364+0000","last_change":"2024-03-21T14:12:59.668840+0000","last_active":"2024-03-21T14:14:09.566364+0000","last_peered":"2024-03-21T14:14:09.566364+0000","last_clean":"2024-03-21T14:14:09.566364+0000","last_became_active":"2024-03-21T14:12:00.533868+0000","last_became_peered":"2024-03-21T14:12:00.533868+0000","last_unstale":"2024-03-21T14:14:09.566364+0000","last_undegraded":"2024-03-21T14:14:09.566364+0000","last_fullsized":"2024-03-21T14:14:09.566364+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.663279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":463,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.570306+0000","last_change":"2024-03-21T14:13:55.416345+0000","last_active":"2024-03-21T14:14:09.570306+0000","last_peered":"2024-03-21T14:14:09.570306+0000","last_clean":"2024-03-21T14:14:09.570306+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:14:09.570306+0000","last_undegraded":"2024-03-21T14:14:09.570306+0000","last_fullsized":"2024-03-21T14:14:09.570306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:55.416068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.416068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:59.940103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":170,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565950+0000","last_change":"2024-03-21T14:12:53.298944+0000","last_active":"2024-03-21T14:14:09.565950+0000","last_peered":"2024-03-21T14:14:09.565950+0000","last_clean":"2024-03-21T14:14:09.565950+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:14:09.565950+0000","last_undegraded":"2024-03-21T14:14:09.565950+0000","last_fullsized":"2024-03-21T14:14:09.565950+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:12:53.298845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.298845+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:13.337221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":102,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565944+0000","last_change":"2024-03-21T14:12:55.222431+0000","last_active":"2024-03-21T14:14:09.565944+0000","last_peered":"2024-03-21T14:14:09.565944+0000","last_clean":"2024-03-21T14:14:09.565944+0000","last_became_active":"2024-03-21T14:12:20.561996+0000","last_became_peered":"2024-03-21T14:12:20.561996+0000","last_unstale":"2024-03-21T14:14:09.565944+0000","last_undegraded":"2024-03-21T14:14:09.565944+0000","last_fullsized":"2024-03-21T14:14:09.565944+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.222353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.222353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:22.891771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.f","version":"265'5935","reported_seq":5987,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.499948+0000","last_change":"2024-03-21T14:13:08.815297+0000","last_active":"2024-03-21T14:14:11.499948+0000","last_peered":"2024-03-21T14:14:11.499948+0000","last_clean":"2024-03-21T14:14:11.499948+0000","last_became_active":"2024-03-21T14:13:08.815004+0000","last_became_peered":"2024-03-21T14:13:08.815004+0000","last_unstale":"2024-03-21T14:14:11.499948+0000","last_undegraded":"2024-03-21T14:14:11.499948+0000","last_fullsized":"2024-03-21T14:14:11.499948+0000","mapping_epoch":244,"log_start":"251'1000","ondisk_log_start":"251'1000","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4935,"log_dups_size":0,"ondisk_log_size":4935,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:26.836653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":388956160,"num_objects":5935,"num_object_clones":0,"num_object_copies":17805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5935,"num_write_kb":379840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":284,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.757645+0000","last_change":"2024-03-21T14:13:42.065806+0000","last_active":"2024-03-21T14:14:09.757645+0000","last_peered":"2024-03-21T14:14:09.757645+0000","last_clean":"2024-03-21T14:14:09.757645+0000","last_became_active":"2024-03-21T14:12:20.560873+0000","last_became_peered":"2024-03-21T14:12:20.560873+0000","last_unstale":"2024-03-21T14:14:09.757645+0000","last_undegraded":"2024-03-21T14:14:09.757645+0000","last_fullsized":"2024-03-21T14:14:09.757645+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:42.065682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:42.065682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.243764+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566761+0000","last_change":"2024-03-21T14:12:45.295649+0000","last_active":"2024-03-21T14:14:09.566761+0000","last_peered":"2024-03-21T14:14:09.566761+0000","last_clean":"2024-03-21T14:14:09.566761+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:14:09.566761+0000","last_undegraded":"2024-03-21T14:14:09.566761+0000","last_fullsized":"2024-03-21T14:14:09.566761+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:45.295537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:45.295537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:14.227680+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":494,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567937+0000","last_change":"2024-03-21T14:13:38.544949+0000","last_active":"2024-03-21T14:14:09.567937+0000","last_peered":"2024-03-21T14:14:09.567937+0000","last_clean":"2024-03-21T14:14:09.567937+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:14:09.567937+0000","last_undegraded":"2024-03-21T14:14:09.567937+0000","last_fullsized":"2024-03-21T14:14:09.567937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:13:38.544738+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.544738+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:07.852333+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"226'10","reported_seq":147,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.565776+0000","last_change":"2024-03-21T14:13:47.212212+0000","last_active":"2024-03-21T14:14:09.565776+0000","last_peered":"2024-03-21T14:14:09.565776+0000","last_clean":"2024-03-21T14:14:09.565776+0000","last_became_active":"2024-03-21T14:12:20.561788+0000","last_became_peered":"2024-03-21T14:12:20.561788+0000","last_unstale":"2024-03-21T14:14:09.565776+0000","last_undegraded":"2024-03-21T14:14:09.565776+0000","last_fullsized":"2024-03-21T14:14:09.565776+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T14:13:47.212098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:47.212098+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.788156+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.c","version":"265'5759","reported_seq":5811,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:12.036730+0000","last_change":"2024-03-21T14:13:09.392139+0000","last_active":"2024-03-21T14:14:12.036730+0000","last_peered":"2024-03-21T14:14:12.036730+0000","last_clean":"2024-03-21T14:14:12.036730+0000","last_became_active":"2024-03-21T14:13:09.391815+0000","last_became_peered":"2024-03-21T14:13:09.391815+0000","last_unstale":"2024-03-21T14:14:12.036730+0000","last_undegraded":"2024-03-21T14:14:12.036730+0000","last_fullsized":"2024-03-21T14:14:12.036730+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":5759,"log_dups_size":0,"ondisk_log_size":5759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.051662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377421824,"num_objects":5759,"num_object_clones":0,"num_object_copies":17277,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5759,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5759,"num_write_kb":368576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.562279+0000","last_change":"2024-03-21T14:13:51.612582+0000","last_active":"2024-03-21T14:14:09.562279+0000","last_peered":"2024-03-21T14:14:09.562279+0000","last_clean":"2024-03-21T14:14:09.562279+0000","last_became_active":"2024-03-21T14:12:00.530815+0000","last_became_peered":"2024-03-21T14:12:00.530815+0000","last_unstale":"2024-03-21T14:14:09.562279+0000","last_undegraded":"2024-03-21T14:14:09.562279+0000","last_fullsized":"2024-03-21T14:14:09.562279+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:51.612367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:02.083143+0000","last_clean_scrub_stamp":"2024-03-21T14:13:51.612367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:17.305704+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":313,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.575821+0000","last_change":"2024-03-21T14:13:02.667615+0000","last_active":"2024-03-21T14:14:09.575821+0000","last_peered":"2024-03-21T14:14:09.575821+0000","last_clean":"2024-03-21T14:14:09.575821+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:14:09.575821+0000","last_undegraded":"2024-03-21T14:14:09.575821+0000","last_fullsized":"2024-03-21T14:14:09.575821+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:13:02.667511+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:26.195393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":485,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558693+0000","last_change":"2024-03-21T14:11:02.706392+0000","last_active":"2024-03-21T14:14:09.558693+0000","last_peered":"2024-03-21T14:14:09.558693+0000","last_clean":"2024-03-21T14:14:09.558693+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:14:09.558693+0000","last_undegraded":"2024-03-21T14:14:09.558693+0000","last_fullsized":"2024-03-21T14:14:09.558693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:11:02.706261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:11:02.706261+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":596,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.561093+0000","last_change":"2024-03-21T14:13:38.528704+0000","last_active":"2024-03-21T14:14:09.561093+0000","last_peered":"2024-03-21T14:14:09.561093+0000","last_clean":"2024-03-21T14:14:09.561093+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:14:09.561093+0000","last_undegraded":"2024-03-21T14:14:09.561093+0000","last_fullsized":"2024-03-21T14:14:09.561093+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:13:38.528575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.528575+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:57.699075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"265'5738","reported_seq":5790,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.458195+0000","last_change":"2024-03-21T14:13:08.816071+0000","last_active":"2024-03-21T14:14:11.458195+0000","last_peered":"2024-03-21T14:14:11.458195+0000","last_clean":"2024-03-21T14:14:11.458195+0000","last_became_active":"2024-03-21T14:13:08.815381+0000","last_became_peered":"2024-03-21T14:13:08.815381+0000","last_unstale":"2024-03-21T14:14:11.458195+0000","last_undegraded":"2024-03-21T14:14:11.458195+0000","last_fullsized":"2024-03-21T14:14:11.458195+0000","mapping_epoch":244,"log_start":"251'1500","ondisk_log_start":"251'1500","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4238,"log_dups_size":0,"ondisk_log_size":4238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:28.266712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":376045568,"num_objects":5738,"num_object_clones":0,"num_object_copies":17214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5738,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5738,"num_write_kb":367232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":453,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.560060+0000","last_change":"2024-03-21T14:13:18.544218+0000","last_active":"2024-03-21T14:14:09.560060+0000","last_peered":"2024-03-21T14:14:09.560060+0000","last_clean":"2024-03-21T14:14:09.560060+0000","last_became_active":"2024-03-21T14:13:18.543885+0000","last_became_peered":"2024-03-21T14:13:18.543885+0000","last_unstale":"2024-03-21T14:14:09.560060+0000","last_undegraded":"2024-03-21T14:14:09.560060+0000","last_fullsized":"2024-03-21T14:14:09.560060+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:53.436026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:11:53.436026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":456,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.564141+0000","last_change":"2024-03-21T14:12:49.278950+0000","last_active":"2024-03-21T14:14:09.564141+0000","last_peered":"2024-03-21T14:14:09.564141+0000","last_clean":"2024-03-21T14:14:09.564141+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:14:09.564141+0000","last_undegraded":"2024-03-21T14:14:09.564141+0000","last_fullsized":"2024-03-21T14:14:09.564141+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:49.278077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:12:49.278077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:12.670128+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":436,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563000+0000","last_change":"2024-03-21T14:13:56.515249+0000","last_active":"2024-03-21T14:14:09.563000+0000","last_peered":"2024-03-21T14:14:09.563000+0000","last_clean":"2024-03-21T14:14:09.563000+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:14:09.563000+0000","last_undegraded":"2024-03-21T14:14:09.563000+0000","last_fullsized":"2024-03-21T14:14:09.563000+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:13:56.515093+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:21.854953+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":431,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558408+0000","last_change":"2024-03-21T14:11:01.663320+0000","last_active":"2024-03-21T14:14:09.558408+0000","last_peered":"2024-03-21T14:14:09.558408+0000","last_clean":"2024-03-21T14:14:09.558408+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:14:09.558408+0000","last_undegraded":"2024-03-21T14:14:09.558408+0000","last_fullsized":"2024-03-21T14:14:09.558408+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:01.663200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:11:01.663200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"265'5816","reported_seq":5867,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:12.166338+0000","last_change":"2024-03-21T14:13:08.814595+0000","last_active":"2024-03-21T14:14:12.166338+0000","last_peered":"2024-03-21T14:14:12.166338+0000","last_clean":"2024-03-21T14:14:12.166338+0000","last_became_active":"2024-03-21T14:13:08.813908+0000","last_became_peered":"2024-03-21T14:13:08.813908+0000","last_unstale":"2024-03-21T14:14:12.166338+0000","last_undegraded":"2024-03-21T14:14:12.166338+0000","last_fullsized":"2024-03-21T14:14:12.166338+0000","mapping_epoch":244,"log_start":"251'2100","ondisk_log_start":"251'2100","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3716,"log_dups_size":0,"ondisk_log_size":3716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:15.846281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381157376,"num_objects":5816,"num_object_clones":0,"num_object_copies":17448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5816,"num_write_kb":372224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566498+0000","last_change":"2024-03-21T14:12:59.226585+0000","last_active":"2024-03-21T14:14:09.566498+0000","last_peered":"2024-03-21T14:14:09.566498+0000","last_clean":"2024-03-21T14:14:09.566498+0000","last_became_active":"2024-03-21T14:08:17.203307+0000","last_became_peered":"2024-03-21T14:08:17.203307+0000","last_unstale":"2024-03-21T14:14:09.566498+0000","last_undegraded":"2024-03-21T14:14:09.566498+0000","last_fullsized":"2024-03-21T14:14:09.566498+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.226457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.226457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:11.605611+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":587,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558894+0000","last_change":"2024-03-21T14:10:55.804376+0000","last_active":"2024-03-21T14:14:09.558894+0000","last_peered":"2024-03-21T14:14:09.558894+0000","last_clean":"2024-03-21T14:14:09.558894+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:14:09.558894+0000","last_undegraded":"2024-03-21T14:14:09.558894+0000","last_fullsized":"2024-03-21T14:14:09.558894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:10:55.804112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.804112+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":301,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.569378+0000","last_change":"2024-03-21T14:12:47.812244+0000","last_active":"2024-03-21T14:14:09.569378+0000","last_peered":"2024-03-21T14:14:09.569378+0000","last_clean":"2024-03-21T14:14:09.569378+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:14:09.569378+0000","last_undegraded":"2024-03-21T14:14:09.569378+0000","last_fullsized":"2024-03-21T14:14:09.569378+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:47.812131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:47.812131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:10.896017+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":494,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.568550+0000","last_change":"2024-03-21T14:13:03.665475+0000","last_active":"2024-03-21T14:14:09.568550+0000","last_peered":"2024-03-21T14:14:09.568550+0000","last_clean":"2024-03-21T14:14:09.568550+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:14:09.568550+0000","last_undegraded":"2024-03-21T14:14:09.568550+0000","last_fullsized":"2024-03-21T14:14:09.568550+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:13:03.665389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.665389+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"265'5643","reported_seq":5695,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:11.468742+0000","last_change":"2024-03-21T14:13:08.815949+0000","last_active":"2024-03-21T14:14:11.468742+0000","last_peered":"2024-03-21T14:14:11.468742+0000","last_clean":"2024-03-21T14:14:11.468742+0000","last_became_active":"2024-03-21T14:13:08.815576+0000","last_became_peered":"2024-03-21T14:13:08.815576+0000","last_unstale":"2024-03-21T14:14:11.468742+0000","last_undegraded":"2024-03-21T14:14:11.468742+0000","last_fullsized":"2024-03-21T14:14:11.468742+0000","mapping_epoch":244,"log_start":"251'1400","ondisk_log_start":"251'1400","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369819648,"num_objects":5643,"num_object_clones":0,"num_object_copies":16929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5643,"num_write_kb":361152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":450,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.566499+0000","last_change":"2024-03-21T14:14:00.615422+0000","last_active":"2024-03-21T14:14:09.566499+0000","last_peered":"2024-03-21T14:14:09.566499+0000","last_clean":"2024-03-21T14:14:09.566499+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:14:09.566499+0000","last_undegraded":"2024-03-21T14:14:09.566499+0000","last_fullsized":"2024-03-21T14:14:09.566499+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.614983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.614983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:09.249329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":447,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.567109+0000","last_change":"2024-03-21T14:13:50.211771+0000","last_active":"2024-03-21T14:14:09.567109+0000","last_peered":"2024-03-21T14:14:09.567109+0000","last_clean":"2024-03-21T14:14:09.567109+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:14:09.567109+0000","last_undegraded":"2024-03-21T14:14:09.567109+0000","last_fullsized":"2024-03-21T14:14:09.567109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:50.211665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.211665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:15.947203+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558543+0000","last_change":"2024-03-21T14:13:45.603203+0000","last_active":"2024-03-21T14:14:09.558543+0000","last_peered":"2024-03-21T14:14:09.558543+0000","last_clean":"2024-03-21T14:14:09.558543+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:14:09.558543+0000","last_undegraded":"2024-03-21T14:14:09.558543+0000","last_fullsized":"2024-03-21T14:14:09.558543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:45.602994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:45.602994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:00.370011+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":492,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558791+0000","last_change":"2024-03-21T14:12:11.483612+0000","last_active":"2024-03-21T14:14:09.558791+0000","last_peered":"2024-03-21T14:14:09.558791+0000","last_clean":"2024-03-21T14:14:09.558791+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:14:09.558791+0000","last_undegraded":"2024-03-21T14:14:09.558791+0000","last_fullsized":"2024-03-21T14:14:09.558791+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:12:11.483458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:11.483458+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"265'5871","reported_seq":5923,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:12.135272+0000","last_change":"2024-03-21T14:13:09.117596+0000","last_active":"2024-03-21T14:14:12.135272+0000","last_peered":"2024-03-21T14:14:12.135272+0000","last_clean":"2024-03-21T14:14:12.135272+0000","last_became_active":"2024-03-21T14:13:09.117255+0000","last_became_peered":"2024-03-21T14:13:09.117255+0000","last_unstale":"2024-03-21T14:14:12.135272+0000","last_undegraded":"2024-03-21T14:14:12.135272+0000","last_fullsized":"2024-03-21T14:14:12.135272+0000","mapping_epoch":244,"log_start":"251'2100","ondisk_log_start":"251'2100","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3771,"log_dups_size":0,"ondisk_log_size":3771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:31.752629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384761856,"num_objects":5871,"num_object_clones":0,"num_object_copies":17613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5871,"num_write_kb":375744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":461,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.573592+0000","last_change":"2024-03-21T14:13:36.335019+0000","last_active":"2024-03-21T14:14:09.573592+0000","last_peered":"2024-03-21T14:14:09.573592+0000","last_clean":"2024-03-21T14:14:09.573592+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:14:09.573592+0000","last_undegraded":"2024-03-21T14:14:09.573592+0000","last_fullsized":"2024-03-21T14:14:09.573592+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:13:36.334881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.334881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.420200+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":455,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.570455+0000","last_change":"2024-03-21T14:13:54.426318+0000","last_active":"2024-03-21T14:14:09.570455+0000","last_peered":"2024-03-21T14:14:09.570455+0000","last_clean":"2024-03-21T14:14:09.570455+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:14:09.570455+0000","last_undegraded":"2024-03-21T14:14:09.570455+0000","last_fullsized":"2024-03-21T14:14:09.570455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_clean_scrub_stamp":"2024-03-21T14:13:54.426178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:01.191151+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":121,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.560175+0000","last_change":"2024-03-21T14:12:00.531054+0000","last_active":"2024-03-21T14:14:09.560175+0000","last_peered":"2024-03-21T14:14:09.560175+0000","last_clean":"2024-03-21T14:14:09.560175+0000","last_became_active":"2024-03-21T14:12:00.530704+0000","last_became_peered":"2024-03-21T14:12:00.530704+0000","last_unstale":"2024-03-21T14:14:09.560175+0000","last_undegraded":"2024-03-21T14:14:09.560175+0000","last_fullsized":"2024-03-21T14:14:09.560175+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:52.959604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:11:52.959604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:15.736622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":492,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558680+0000","last_change":"2024-03-21T14:12:53.529498+0000","last_active":"2024-03-21T14:14:09.558680+0000","last_peered":"2024-03-21T14:14:09.558680+0000","last_clean":"2024-03-21T14:14:09.558680+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:14:09.558680+0000","last_undegraded":"2024-03-21T14:14:09.558680+0000","last_fullsized":"2024-03-21T14:14:09.558680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:12:53.529362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.529362+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"265'5728","reported_seq":5780,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:12.023430+0000","last_change":"2024-03-21T14:13:09.391951+0000","last_active":"2024-03-21T14:14:12.023430+0000","last_peered":"2024-03-21T14:14:12.023430+0000","last_clean":"2024-03-21T14:14:12.023430+0000","last_became_active":"2024-03-21T14:13:09.391580+0000","last_became_peered":"2024-03-21T14:13:09.391580+0000","last_unstale":"2024-03-21T14:14:12.023430+0000","last_undegraded":"2024-03-21T14:14:12.023430+0000","last_fullsized":"2024-03-21T14:14:12.023430+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":5728,"log_dups_size":0,"ondisk_log_size":5728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:17.600317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375390208,"num_objects":5728,"num_object_clones":0,"num_object_copies":17184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5728,"num_write_kb":366592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":441,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558970+0000","last_change":"2024-03-21T14:13:18.721177+0000","last_active":"2024-03-21T14:14:09.558970+0000","last_peered":"2024-03-21T14:14:09.558970+0000","last_clean":"2024-03-21T14:14:09.558970+0000","last_became_active":"2024-03-21T14:13:18.720794+0000","last_became_peered":"2024-03-21T14:13:18.720794+0000","last_unstale":"2024-03-21T14:14:09.558970+0000","last_undegraded":"2024-03-21T14:14:09.558970+0000","last_fullsized":"2024-03-21T14:14:09.558970+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:50.567559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:50.567559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:09.739005+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":613,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.573368+0000","last_change":"2024-03-21T14:14:05.166407+0000","last_active":"2024-03-21T14:14:09.573368+0000","last_peered":"2024-03-21T14:14:09.573368+0000","last_clean":"2024-03-21T14:14:09.573368+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:14:09.573368+0000","last_undegraded":"2024-03-21T14:14:09.573368+0000","last_fullsized":"2024-03-21T14:14:09.573368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:14:05.166173+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:14:05.166173+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:18.265678+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":462,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.563175+0000","last_change":"2024-03-21T14:13:50.440212+0000","last_active":"2024-03-21T14:14:09.563175+0000","last_peered":"2024-03-21T14:14:09.563175+0000","last_clean":"2024-03-21T14:14:09.563175+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:14:09.563175+0000","last_undegraded":"2024-03-21T14:14:09.563175+0000","last_fullsized":"2024-03-21T14:14:09.563175+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:50.439847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.439847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:53.131705+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":496,"reported_epoch":265,"state":"active+clean","last_fresh":"2024-03-21T14:14:09.558474+0000","last_change":"2024-03-21T14:13:35.642420+0000","last_active":"2024-03-21T14:14:09.558474+0000","last_peered":"2024-03-21T14:14:09.558474+0000","last_clean":"2024-03-21T14:14:09.558474+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:14:09.558474+0000","last_undegraded":"2024-03-21T14:14:09.558474+0000","last_fullsized":"2024-03-21T14:14:09.558474+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:13:35.642272+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:13:35.642272+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:47.161685+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":6070730752,"num_objects":92632,"num_object_clones":0,"num_object_copies":277896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92632,"num_write_kb":5928448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18235195392,"data_stored":18235195392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73932,"ondisk_log_size":73932,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":691,"num_read_kb":686,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":130,"ondisk_log_size":130,"up":3,"acting":3,"num_store_stats":5},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103312,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4415484,"kb_used_data":4001140,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":89329668,"statfs":{"total":95995035648,"available":91473580032,"internally_reserved":0,"allocated":4097167360,"data_stored":4096689103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":424278474},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,3,2,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430357,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333360,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93411792,"statfs":{"total":95995035648,"available":95653675008,"internally_reserved":0,"allocated":954368,"data_stored":554205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340392390},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575024,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4193680,"kb_used_data":3702980,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":89551472,"statfs":{"total":95995035648,"available":91700707328,"internally_reserved":0,"allocated":3791851520,"data_stored":3791357186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738549,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1868588,"kb_used_data":1490336,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":91876564,"statfs":{"total":95995035648,"available":94081601536,"internally_reserved":0,"allocated":1526104064,"data_stored":1525644343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":387316165},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411498,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2258404,"kb_used_data":1848216,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":91486748,"statfs":{"total":95995035648,"available":93682429952,"internally_reserved":0,"allocated":1892573184,"data_stored":1892121718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":420018629},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182564,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2229936,"kb_used_data":1837800,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91515216,"statfs":{"total":95995035648,"available":93711581184,"internally_reserved":0,"allocated":1881907200,"data_stored":1881439358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":401537479},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019040,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1835064,"kb_used_data":1448496,"kb_used_omap":1,"kb_used_meta":386558,"kb_avail":91910088,"statfs":{"total":95995035648,"available":94115930112,"internally_reserved":0,"allocated":1483259904,"data_stored":1482784024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":395835845},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691995,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3734780,"kb_used_data":3334320,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":90010372,"statfs":{"total":95995035648,"available":92170620928,"internally_reserved":0,"allocated":3414343680,"data_stored":3413868671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1531445248,"data_stored":1531445248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3824680960,"data_stored":3824680960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1910177792,"data_stored":1910177792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1894580224,"data_stored":1894580224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1497432064,"data_stored":1497432064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3427008512,"data_stored":3427008512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4149870592,"data_stored":4149870592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:14:16.263 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-21T14:14:16.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-21T14:14:16.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.315 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:16.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:16.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:16.450 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:16.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:16.587 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:16.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:16.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:16.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:16.736 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:16 smithi003 ceph-mon[29576]: pgmap v521: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 795 op/s 2024-03-21T14:14:16.737 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:14:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:14:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:14:16.737 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:16 smithi003 ceph-mon[40510]: pgmap v521: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 795 op/s 2024-03-21T14:14:16.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:16.867 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:16.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:16.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:16 smithi118 ceph-mon[33862]: pgmap v521: 148 pgs: 148 active+clean; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 50 MiB/s wr, 795 op/s 2024-03-21T14:14:16.955 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:14:16.955 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:14:16.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.011 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:17.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:17.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:17.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:17.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:17.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 99953 99937 105.844 80.25 0.0146798 0.00944278 2024-03-21T14:14:17.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:17.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:17.425 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:17.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:17.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:14:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:14:17.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:17.562 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:17.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:17.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.700 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:17.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:17.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:14:17.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:17.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:17.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:17.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:18.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:18.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:18.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:18.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:18.110 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:18.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:18.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:18.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:18.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:14:18.247618+0000 min lat: 0.00113893 max lat: 0.152179 avg lat: 0.00948138 2024-03-21T14:14:18.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:14:18.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 101241 101225 105.422 80.5 0.00998141 0.00948138 2024-03-21T14:14:18.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:18.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:18.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:18.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:18.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:18.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:18.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:18.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:18 smithi003 ceph-mon[29576]: pgmap v522: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:18.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:18 smithi003 ceph-mon[40510]: pgmap v522: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:18.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.811 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:18.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:18.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:18 smithi118 ceph-mon[33862]: pgmap v522: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:18.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:18.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:19.025 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:19.048 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:19.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:19.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:19.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:19.246 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 102501 102485 104.984 78.75 0.0082876 0.00952091 2024-03-21T14:14:19.260 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:19.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:19.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:19.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.403 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:19.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:19.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:19.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:19.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:19.653 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.695 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:19.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:19.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:19.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:19.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:19.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:19.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:19.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:19.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:20.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:20.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.124 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:20.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:20.225 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:20.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 103809 103793 104.609 81.75 0.0071689 0.00955471 2024-03-21T14:14:20.257 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:20.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:20.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:20.497 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:20.583 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:20.655 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:14:20.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:20.717 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:20.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:20.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:20 smithi118 ceph-mon[33862]: pgmap v523: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:14:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/230600614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:20.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:20 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:20.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:20.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:20.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:20 smithi003 ceph-mon[29576]: pgmap v523: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:14:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/230600614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:20 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:21.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:20 smithi003 ceph-mon[40510]: pgmap v523: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:14:21.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/230600614' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:21.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:20 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:21.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:21.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:21.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:21.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:21.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 105176 105160 104.305 85.4375 0.00732677 0.00958273 2024-03-21T14:14:21.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:21.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:21.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:21.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:21.388 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:21.428 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:21.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:21.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:21.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:21.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:21.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:21.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:21.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:21.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:21.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:21.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:21.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:21.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:21.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:21 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:21.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:21 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:14:21.890 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:21 smithi003 ceph-mon[29576]: osdmap e266: 8 total, 8 up, 8 in 2024-03-21T14:14:21.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:21 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:21.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:21 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:14:21.891 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:21 smithi003 ceph-mon[40510]: osdmap e266: 8 total, 8 up, 8 in 2024-03-21T14:14:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:21 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:21 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:14:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:21 smithi118 ceph-mon[33862]: osdmap e266: 8 total, 8 up, 8 in 2024-03-21T14:14:21.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.004 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:22.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:22.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:22.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:22.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:22.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:22.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 106562 106546 104.028 86.625 0.00832342 0.00960832 2024-03-21T14:14:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:22.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:22.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:22.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:22.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:22.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:22.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:22.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:22.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:22.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:22.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:22 smithi003 ceph-mon[29576]: pgmap v525: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:14:22.905 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:22 smithi003 ceph-mon[40510]: pgmap v525: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:14:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:22 smithi118 ceph-mon[33862]: pgmap v525: 148 pgs: 148 active+clean; 6.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-03-21T14:14:22.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:22.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:22.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:22.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:23.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:23.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:23.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:23.247 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 107739 107723 103.559 73.5625 0.0142267 0.00965108 2024-03-21T14:14:23.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:23.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:23.308 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:14:23.347 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:14:23.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:23.394 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:14:23.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:23.436 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:14:23.477 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:14:23.518 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:14:23.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:23.556 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:23.570 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:14:23.617 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:14:23.623 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:14:23.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:23.691 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4100965931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:23.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:23.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:23.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:23.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:23.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:23.984 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:23.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:24.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:23 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4100965931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:24.086 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:24.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:23 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4100965931' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:24.236 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:24.249 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 108869 108853 103.06 70.625 0.0390352 0.00969747 2024-03-21T14:14:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:24.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:24.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.430 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:24.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:24.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:24.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:24.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:24.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:14:24.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.702 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:24.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:24.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:24.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:24.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:24.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:24.997 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:24.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:24 smithi003 ceph-mon[29576]: pgmap v526: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:24 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:14:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:24 smithi003 ceph-mon[40510]: pgmap v526: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:24 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:14:25.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:25.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:25.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:24 smithi118 ceph-mon[33862]: pgmap v526: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:24 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:14:25.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:25.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 109866 109850 102.452 62.3125 0.0146185 0.00975458 2024-03-21T14:14:25.262 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:25.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:25.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:25.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:25.555 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:25.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.693 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:25.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:25.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:25.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:25.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:25.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:25.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:26.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:14:26.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:14:26.008 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:25 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:26.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:26.110 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:26.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:14:26.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:25 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:14:26.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:26.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 15 111154 111139 102.13 80.5625 0.00799503 0.00978645 2024-03-21T14:14:26.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:26.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:26.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:26.390 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:26.492 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:26.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.518 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:26.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:26.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:26.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:26.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:26.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:26 smithi003 ceph-mon[29576]: pgmap v527: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:26.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:26 smithi003 ceph-mon[29576]: Adjusting osd_memory_target on smithi003 to 3249M 2024-03-21T14:14:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:26 smithi003 ceph-mon[40510]: pgmap v527: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:26 smithi003 ceph-mon[40510]: Adjusting osd_memory_target on smithi003 to 3249M 2024-03-21T14:14:26.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:14:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:14:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:14:26.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:26.797 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:26.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:26.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:26.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:26.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:27.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:27.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:27.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:26 smithi118 ceph-mon[33862]: pgmap v527: 148 pgs: 148 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:14:27.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:26 smithi118 ceph-mon[33862]: Adjusting osd_memory_target on smithi003 to 3249M 2024-03-21T14:14:27.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:14:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:14:27.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:27.221 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:27.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 112474 112458 101.844 82.4375 0.0176485 0.00981444 2024-03-21T14:14:27.323 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:27.358 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:27.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.497 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:27.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:27.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:27.633 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:27.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:27.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:27.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.772 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:27.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:27.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:27.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:27.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:27.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:27.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:27.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:27 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1853095536' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-21T14:14:27.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:27 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-21T14:14:27.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:27.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:27 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1853095536' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-21T14:14:27.985 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:27 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-21T14:14:28.011 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:28.044 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:28.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:28.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:27 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1853095536' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-21T14:14:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:27 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-21T14:14:28.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:28.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:28.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 113811 113795 101.583 83.5625 0.0200675 0.0098397 2024-03-21T14:14:28.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:28.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:28.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:28.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:28.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:28.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.600 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:28.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pg_num to 11 2024-03-21T14:14:28.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:28.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:28.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:28.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:28.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:28.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:28.982 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:28.993 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:28 smithi003 ceph-mon[29576]: pgmap v528: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-21T14:14:28.994 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-21T14:14:28.994 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:28 smithi003 ceph-mon[29576]: osdmap e267: 8 total, 8 up, 8 in 2024-03-21T14:14:28.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:28 smithi003 ceph-mon[40510]: pgmap v528: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-21T14:14:28.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-21T14:14:28.994 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:28 smithi003 ceph-mon[40510]: osdmap e267: 8 total, 8 up, 8 in 2024-03-21T14:14:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.019 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:29.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:29.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:29.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:28 smithi118 ceph-mon[33862]: pgmap v528: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-21T14:14:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-21T14:14:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:28 smithi118 ceph-mon[33862]: osdmap e267: 8 total, 8 up, 8 in 2024-03-21T14:14:29.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 15 115203 115188 101.378 87.0625 0.00928479 0.00985971 2024-03-21T14:14:29.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:29.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:29.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:29.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:29.440 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:29.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:14:29.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:29.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:29.680 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:29.732 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:29.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:29.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:29.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:29.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:29.984 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:29 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:29 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:29 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:29 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:30.021 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:30.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:30.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:29 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:29 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:30.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 116495 116479 101.091 80.6875 0.0102892 0.00988751 2024-03-21T14:14:30.268 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:30.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.305 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:30.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221749 7.9 7:96145550:::benchmark_data_smithi003_80362_object100732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.246771+0000", 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 381.14870712999999, 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.87310833899999996, 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.401 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221749 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246771+0000", 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246771+0000", 2024-03-21T14:14:30.402 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246772+0000", 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0699999999999996e-07 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246777+0000", 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0880000000000002e-06 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.403 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246786+0000", 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0659999999999993e-06 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246794+0000", 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8939999999999993e-06 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246855+0000", 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0977000000000002e-05 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.404 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246863+0000", 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1019999999999997e-06 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.116362+0000", 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86949848699999999 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.116503+0000", 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014111199999999999 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.116650+0000", 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014681899999999999 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119736+0000", 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030864619999999999 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119754+0000", 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8315999999999999e-05 2024-03-21T14:14:30.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119786+0000", 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.1615000000000001e-05 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119840+0000", 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4014000000000002e-05 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119880+0000", 2024-03-21T14:14:30.407 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9579999999999997e-05 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219636 8.3 8:c2dee2e9:::benchmark_data_smithi003_110242_object88719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.844721988, 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99404183000000002, 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219636 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.409 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550757+0000", 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.02e-07 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550761+0000", 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0810000000000004e-06 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550771+0000", 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4530000000000008e-06 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.410 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550778+0000", 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550827+0000", 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8690000000000003e-05 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550833+0000", 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1410000000000002e-06 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.411 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539253+0000", 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98841965700000001 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539380+0000", 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012699099999999999 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539666+0000", 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028561599999999998 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.412 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.542830+0000", 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031648290000000001 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544576+0000", 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001746039 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544691+0000", 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011476000000000001 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.413 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544746+0000", 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4617999999999998e-05 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544798+0000", 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2462e-05 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219640 8.7 8:e03a6842:::benchmark_data_smithi003_110242_object88723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:14:30.414 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.83475220599999, 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.000192481, 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219640 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.415 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560728+0000", 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.548e-06 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560734+0000", 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5189999999999998e-06 2024-03-21T14:14:30.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560752+0000", 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7683999999999999e-05 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560764+0000", 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1558e-05 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560819+0000", 2024-03-21T14:14:30.417 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5161000000000003e-05 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560826+0000", 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8390000000000001e-06 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551363+0000", 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99053792699999998 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551445+0000", 2024-03-21T14:14:30.418 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1199999999999995e-05 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551758+0000", 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031323899999999999 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558010+0000", 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062516619999999998 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558022+0000", 2024-03-21T14:14:30.419 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2751e-05 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560813+0000", 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027905650000000001 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560868+0000", 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5031999999999998e-05 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560919+0000", 2024-03-21T14:14:30.420 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0796e-05 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219647 8.7 8:e325f337:::benchmark_data_smithi003_110242_object88730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.829354408, 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0098845869999999, 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.421 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219647 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.422 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566125+0000", 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7699999999999992e-07 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566131+0000", 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5400000000000003e-06 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566142+0000", 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1029000000000001e-05 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.423 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566150+0000", 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8179999999999993e-06 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566259+0000", 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109064 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566268+0000", 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1850000000000004e-06 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.424 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554008+0000", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98774052300000004 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554113+0000", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104929 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554390+0000", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027710000000000001 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560968+0000", 2024-03-21T14:14:30.425 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065778310000000001 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569693+0000", 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087252330000000006 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575871+0000", 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061776979999999997 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.426 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575961+0000", 2024-03-21T14:14:30.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9869000000000003e-05 2024-03-21T14:14:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.428 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576009+0000", 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7790999999999997e-05 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219661 8.3 8:c78a9c88:::benchmark_data_smithi003_110242_object88744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.81775621599999, 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96728133199999999, 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.429 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219661 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577723+0000", 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.218e-06 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577729+0000", 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4469999999999997e-06 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577741+0000", 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1929e-05 2024-03-21T14:14:30.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577749+0000", 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0420000000000003e-06 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577810+0000", 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0661000000000001e-05 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577818+0000", 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5720000000000003e-06 2024-03-21T14:14:30.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541402+0000", 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96358385700000004 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541513+0000", 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110608 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541765+0000", 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025253 2024-03-21T14:14:30.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544855+0000", 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030902249999999998 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544866+0000", 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0219e-05 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544900+0000", 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4406999999999998e-05 2024-03-21T14:14:30.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544963+0000", 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3510999999999996e-05 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545004+0000", 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0105999999999999e-05 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.435 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219664 8.3 8:c1f39ff4:::benchmark_data_smithi003_110242_object88747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.815820938, 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96554897500000003, 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219664 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:14:30.436 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579659+0000", 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0979999999999999e-06 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579663+0000", 2024-03-21T14:14:30.437 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6879999999999998e-06 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579675+0000", 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1701e-05 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579684+0000", 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3670000000000003e-06 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579738+0000", 2024-03-21T14:14:30.438 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3353999999999998e-05 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579746+0000", 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1349999999999992e-06 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.542881+0000", 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96313484800000004 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.543021+0000", 2024-03-21T14:14:30.439 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014037899999999999 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.543251+0000", 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000229776 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545019+0000", 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001768541 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545064+0000", 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4567999999999997e-05 2024-03-21T14:14:30.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545127+0000", 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3037e-05 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545169+0000", 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1650000000000003e-05 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545206+0000", 2024-03-21T14:14:30.441 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7833000000000002e-05 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219669 8.7 8:e640187a:::benchmark_data_smithi003_110242_object88752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.81296897600001, 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99373624599999999, 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.442 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219669 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.443 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582510+0000", 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3500000000000005e-07 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582515+0000", 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6190000000000002e-06 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582527+0000", 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1663e-05 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.444 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582535+0000", 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9719999999999992e-06 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582589+0000", 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4258000000000001e-05 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582594+0000", 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5650000000000001e-06 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.445 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556001+0000", 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97340653200000005 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556118+0000", 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116953 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556396+0000", 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027774400000000002 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.446 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569603+0000", 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013207429 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569729+0000", 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126198 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576017+0000", 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062882509999999999 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.447 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576174+0000", 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015680300000000001 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576246+0000", 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1423999999999999e-05 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219677 8.7 8:e3d98eec:::benchmark_data_smithi003_110242_object88760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.448 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.80699775400001, 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98785003400000004, 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219677 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.449 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588482+0000", 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7999999999999993e-07 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588487+0000", 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2270000000000001e-06 2024-03-21T14:14:30.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588498+0000", 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1511e-05 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588507+0000", 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6759999999999996e-06 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588580+0000", 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3046000000000005e-05 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.451 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588589+0000", 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8140000000000007e-06 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558336+0000", 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96974661200000001 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558447+0000", 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111644 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.452 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558737+0000", 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028978899999999999 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569659+0000", 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010921759 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569764+0000", 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105388 2024-03-21T14:14:30.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576252+0000", 2024-03-21T14:14:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064880720000000001 2024-03-21T14:14:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.454 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576293+0000", 2024-03-21T14:14:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1300000000000001e-05 2024-03-21T14:14:30.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576331+0000", 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7215999999999997e-05 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219708 8.7 8:e065f095:::benchmark_data_smithi003_110242_object88791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.778043419, 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96333364799999999, 2024-03-21T14:14:30.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219708 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.457 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617436+0000", 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2160000000000001e-06 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617442+0000", 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4369999999999998e-06 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.458 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617453+0000", 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1109e-05 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617460+0000", 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6179999999999997e-06 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617520+0000", 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9939000000000003e-05 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.459 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617530+0000", 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1679999999999993e-06 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561003+0000", 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94347382000000002 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561126+0000", 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012228799999999999 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.460 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561456+0000", 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033023800000000001 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575680+0000", 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014224647999999999 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575752+0000", 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1209000000000002e-05 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.461 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580677+0000", 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049251019999999998 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580721+0000", 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4159000000000002e-05 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580769+0000", 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7697000000000001e-05 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.462 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219718 8.3 8:c6a93420:::benchmark_data_smithi003_110242_object88801:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.626521+0000", 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.768957728, 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92249251799999998, 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219718 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.463 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626521+0000", 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626521+0000", 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626523+0000", 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.018e-06 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.464 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626528+0000", 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3360000000000002e-06 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626542+0000", 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4e-05 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626550+0000", 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8779999999999998e-06 2024-03-21T14:14:30.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626613+0000", 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2685e-05 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.626622+0000", 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6250000000000002e-06 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545249+0000", 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91862714099999998 2024-03-21T14:14:30.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545356+0000", 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010654600000000001 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545594+0000", 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000237588 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.548844+0000", 2024-03-21T14:14:30.467 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032503900000000001 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.548880+0000", 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6492000000000002e-05 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.548915+0000", 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4770000000000001e-05 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.548973+0000", 2024-03-21T14:14:30.468 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7447000000000001e-05 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.549013+0000", 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0602000000000002e-05 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219729 8.7 8:e575955d:::benchmark_data_smithi003_110242_object88812:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.638155+0000", 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.75732379799999, 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94269391599999997, 2024-03-21T14:14:30.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219729 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638155+0000", 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.470 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638155+0000", 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638156+0000", 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2190000000000001e-06 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638161+0000", 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5929999999999996e-06 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.471 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638170+0000", 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5089999999999993e-06 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638179+0000", 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1619999999999997e-06 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638229+0000", 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9645999999999999e-05 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.472 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.638236+0000", 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7750000000000004e-06 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563523+0000", 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92528721300000005 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563634+0000", 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111172 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.473 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563905+0000", 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000271119 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575719+0000", 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011813947 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575784+0000", 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4568999999999998e-05 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.474 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580780+0000", 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049961609999999998 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580816+0000", 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6230999999999999e-05 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580849+0000", 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.26e-05 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219742 8.7 8:e6755dee:::benchmark_data_smithi003_110242_object88825:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.646415+0000", 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.749063277, 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93471664499999996, 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219742 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.476 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646415+0000", 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646415+0000", 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646416+0000", 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4499999999999995e-07 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.477 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646421+0000", 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3589999999999999e-06 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646433+0000", 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1080999999999999e-05 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646441+0000", 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0490000000000004e-06 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.478 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646502+0000", 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1008999999999998e-05 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.646510+0000", 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3410000000000003e-06 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.567295+0000", 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92078535500000003 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.479 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.567415+0000", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120127 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.567749+0000", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033401600000000002 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580854+0000", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.01310469 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580898+0000", 2024-03-21T14:14:30.480 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4129999999999999e-05 2024-03-21T14:14:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.481 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.581055+0000", 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015669100000000001 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.581097+0000", 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.214e-05 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.482 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.581132+0000", 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4712000000000002e-05 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219811 8.3 8:c6f4fb22:::benchmark_data_smithi003_110242_object88894:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.744621+0000", 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.65085784600001, 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.81173121199999998, 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.483 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219811 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.744621+0000", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.744621+0000", 2024-03-21T14:14:30.484 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.744622+0000", 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.744630+0000", 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2090000000000005e-06 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784210+0000", 2024-03-21T14:14:30.485 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.039579521999999999 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784220+0000", 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8800000000000003e-06 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784280+0000", 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9688999999999997e-05 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784289+0000", 2024-03-21T14:14:30.486 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2350000000000005e-06 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.547033+0000", 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.762744015 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.547137+0000", 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104341 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.547371+0000", 2024-03-21T14:14:30.487 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023343800000000001 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551249+0000", 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038787050000000001 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.553401+0000", 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002152092 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556114+0000", 2024-03-21T14:14:30.488 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027130050000000001 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556298+0000", 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018336700000000001 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556352+0000", 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4014999999999997e-05 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219815 8.3 8:c7f61a46:::benchmark_data_smithi003_110242_object88898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.784264+0000", 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.611214461, 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.77235340799999996, 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219815 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.490 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784264+0000", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784264+0000", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784265+0000", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1799999999999999e-06 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.491 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784270+0000", 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.045e-06 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784279+0000", 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0440000000000001e-06 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784285+0000", 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1199999999999999e-06 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.492 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784498+0000", 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021279900000000001 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.784506+0000", 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6699999999999994e-06 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.549052+0000", 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.764546536 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.493 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.549168+0000", 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011541599999999999 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.549438+0000", 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027048300000000002 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.553414+0000", 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039756669999999996 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.494 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.553448+0000", 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.3590999999999997e-05 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556393+0000", 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029454429999999998 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556574+0000", 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018114499999999999 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.495 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556617+0000", 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3269000000000002e-05 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219851 8.7 8:e07f1e37:::benchmark_data_smithi003_110242_object88934:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.827661+0000", 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.56781758400001, 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.75418554000000004, 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.496 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219851 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827661+0000", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827661+0000", 2024-03-21T14:14:30.497 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827662+0000", 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9900000000000009e-07 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827666+0000", 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1509999999999997e-06 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.498 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827677+0000", 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0912e-05 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827683+0000", 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.494e-06 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827741+0000", 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7911999999999997e-05 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.499 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.827751+0000", 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3870000000000001e-06 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569794+0000", 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.74204293600000004 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569901+0000", 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107724 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.500 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.570329+0000", 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0004281 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580859+0000", 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010529606 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580948+0000", 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9427999999999998e-05 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.501 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.581760+0000", 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00081143199999999995 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.581807+0000", 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7522999999999999e-05 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.581846+0000", 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8936000000000001e-05 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.502 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219853 8.3 8:c39ff726:::benchmark_data_smithi003_110242_object88936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.830100+0000", 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.56537854999999, 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.73332554699999997, 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219853 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.503 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830100+0000", 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830100+0000", 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830101+0000", 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9699999999999995e-07 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.504 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830105+0000", 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1559999999999997e-06 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830124+0000", 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9403999999999999e-05 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830133+0000", 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3319999999999992e-06 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.505 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830204+0000", 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1212999999999997e-05 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.830214+0000", 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0456e-05 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551280+0000", 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.72106527600000003 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.506 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551393+0000", 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113068 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551666+0000", 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027369599999999998 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556639+0000", 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049725719999999998 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.507 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.508 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:30.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560442+0000", 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0038026560000000002 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563121+0000", 2024-03-21T14:14:30.509 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026792119999999998 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563190+0000", 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9208999999999994e-05 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.510 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563425+0000", 2024-03-21T14:14:30.513 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023550000000000001 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219876 8.3 8:c6bb4298:::benchmark_data_smithi003_110242_object88959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.912588+0000", 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 139.482890341, 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.65147798099999998, 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.514 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219876 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912588+0000", 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912588+0000", 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.515 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912589+0000", 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2500000000000005e-07 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912592+0000", 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4910000000000001e-06 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912601+0000", 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3000000000000002e-06 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.516 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912606+0000", 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4389999999999999e-06 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912654+0000", 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7546000000000002e-05 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.912660+0000", 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9059999999999996e-06 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.517 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.553535+0000", 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.64087422199999999 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.553641+0000", 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105987 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.553918+0000", 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000276803 2024-03-21T14:14:30.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560480+0000", 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065629449999999997 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561796+0000", 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0013157500000000001 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.563871+0000", 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020745099999999999 2024-03-21T14:14:30.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.564025+0000", 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015429899999999999 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.564066+0000", 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1057999999999999e-05 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:223385 8.3 8:c36677c6:::benchmark_data_smithi003_110242_object92468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-21T14:14:30.520 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:14.877457+0000", 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 135.518021917, 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18080795899999999, 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 223385 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.877457+0000", 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.877457+0000", 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.877458+0000", 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4950000000000001e-06 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.877463+0000", 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.181e-06 2024-03-21T14:14:30.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.877474+0000", 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1090000000000001e-05 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.877484+0000", 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4229999999999994e-06 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.051535+0000", 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17405153900000001 2024-03-21T14:14:30.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.051654+0000", 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118904 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.052014+0000", 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035952899999999998 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.055638+0000", 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036245930000000002 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.055690+0000", 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2046999999999998e-05 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.058151+0000", 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024604409999999998 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.058203+0000", 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2141000000000001e-05 2024-03-21T14:14:30.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.058264+0000", 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1575999999999995e-05 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:223394 8.13 8:ccea0a4f:::benchmark_data_smithi003_110242_object92477:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:14.883792+0000", 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 135.511686107, 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.186397438, 2024-03-21T14:14:30.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 223394 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.883792+0000", 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.883792+0000", 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.883793+0000", 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4899999999999995e-07 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.883796+0000", 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.875e-06 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.883804+0000", 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0409999999999998e-06 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.883810+0000", 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3879999999999997e-06 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.056066+0000", 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.172255932 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.056237+0000", 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000170972 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3,5,6", 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.056590+0000", 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000353412 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.060558+0000", 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039672650000000002 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.062814+0000", 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022561640000000002 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.062834+0000", 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0183e-05 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.063370+0000", 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00053593399999999995 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.070029+0000", 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066584620000000004 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.070095+0000", 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6513000000000005e-05 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.070190+0000", 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4747999999999995e-05 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:223401 8.3 8:c600d190:::benchmark_data_smithi003_110242_object92484:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:14.890352+0000", 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 135.50512665100001, 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.17979935, 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 223401 2024-03-21T14:14:30.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.890352+0000", 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.890352+0000", 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.890353+0000", 2024-03-21T14:14:30.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.235e-06 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.890359+0000", 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.212e-06 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.890371+0000", 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.137e-05 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:14.890379+0000", 2024-03-21T14:14:30.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9729999999999997e-06 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.055717+0000", 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16533815700000001 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.055827+0000", 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110467 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.056199+0000", 2024-03-21T14:14:30.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00037160399999999999 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.062828+0000", 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066289779999999998 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.063679+0000", 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00085137400000000001 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.070021+0000", 2024-03-21T14:14:30.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0063417619999999999 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.070083+0000", 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2217000000000004e-05 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:15.070151+0000", 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8001000000000002e-05 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:14:30.538 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:14:30.539 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:14:30.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:30.647 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:30.748 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:30.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:30.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:30.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:30.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:30.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:30 smithi003 ceph-mon[29576]: pgmap v530: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-21T14:14:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:30 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-21T14:14:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:30 smithi003 ceph-mon[29576]: osdmap e268: 8 total, 8 up, 8 in 2024-03-21T14:14:31.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:30 smithi003 ceph-mon[40510]: pgmap v530: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-21T14:14:31.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:30 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-21T14:14:31.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:30 smithi003 ceph-mon[40510]: osdmap e268: 8 total, 8 up, 8 in 2024-03-21T14:14:31.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:31.067 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:31.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:30 smithi118 ceph-mon[33862]: pgmap v530: 148 pgs: 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-21T14:14:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:30 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-21T14:14:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:30 smithi118 ceph-mon[33862]: osdmap e268: 8 total, 8 up, 8 in 2024-03-21T14:14:31.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:31.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:31.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 117844 117828 100.861 84.3125 0.0058656 0.00991008 2024-03-21T14:14:31.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:31.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:31.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:31.494 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:31.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:31.635 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:31.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:31.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:31.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:31.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:31.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:31.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:31.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:32.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:32.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.054 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:32.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:32.155 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:31 smithi118 ceph-mon[33862]: osdmap e269: 8 total, 8 up, 8 in 2024-03-21T14:14:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:31 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:32.188 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[29576]: osdmap e269: 8 total, 8 up, 8 in 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[40510]: osdmap e269: 8 total, 8 up, 8 in 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-21T14:14:32.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:32.191 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:32.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 119240 119224 100.677 87.25 0.00591495 0.00992816 2024-03-21T14:14:32.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:32.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:32.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:32.465 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:32.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:32.601 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:32.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:32.646 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:32.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:32.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:32.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:32.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:32.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:32.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:32.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[29576]: pgmap v533: 158 pgs: 10 unknown, 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[29576]: osdmap e270: 8 total, 8 up, 8 in 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[29576]: osdmap e271: 8 total, 8 up, 8 in 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[40510]: pgmap v533: 158 pgs: 10 unknown, 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[40510]: osdmap e270: 8 total, 8 up, 8 in 2024-03-21T14:14:32.937 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:32 smithi003 ceph-mon[40510]: osdmap e271: 8 total, 8 up, 8 in 2024-03-21T14:14:32.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.031 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:33.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:33.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:33.168 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:32 smithi118 ceph-mon[33862]: pgmap v533: 158 pgs: 10 unknown, 148 active+clean; 6.8 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 994 op/s 2024-03-21T14:14:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:32 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-21T14:14:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:32 smithi118 ceph-mon[33862]: osdmap e270: 8 total, 8 up, 8 in 2024-03-21T14:14:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:32 smithi118 ceph-mon[33862]: osdmap e271: 8 total, 8 up, 8 in 2024-03-21T14:14:33.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 120650 120634 100.51 88.125 0.0215218 0.00994498 2024-03-21T14:14:33.271 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:33.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:33.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:33.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:33.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:33.558 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:33.596 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:33.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:33.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:33.738 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:33.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:33.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:33.874 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:33.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:33.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:34.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:34.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:34.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:34.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 122007 121991 100.303 84.8125 0.00859803 0.0099648 2024-03-21T14:14:34.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:34.298 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:14:34.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:34.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:34.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:34.487 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:14:34.487 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:14:34.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:14:34.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:34.576 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:34.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:34.586 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:14:34.678 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:34.713 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:34.801 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: pgmap v536: 158 pgs: 2 unknown, 156 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-21T14:14:34.801 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: osdmap e272: 8 total, 8 up, 8 in 2024-03-21T14:14:34.801 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1055355490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[29576]: osdmap e273: 8 total, 8 up, 8 in 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: pgmap v536: 158 pgs: 2 unknown, 156 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: osdmap e272: 8 total, 8 up, 8 in 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1055355490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:34.802 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:14:34.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:34 smithi003 ceph-mon[40510]: osdmap e273: 8 total, 8 up, 8 in 2024-03-21T14:14:34.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:34.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:34.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:34.989 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:34.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:35.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:35.128 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:35.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: pgmap v536: 158 pgs: 2 unknown, 156 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-21T14:14:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: osdmap e272: 8 total, 8 up, 8 in 2024-03-21T14:14:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1055355490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:14:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:14:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:14:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:14:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:34 smithi118 ceph-mon[33862]: osdmap e273: 8 total, 8 up, 8 in 2024-03-21T14:14:35.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:35.248 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 123293 123277 100.044 80.375 0.0102961 0.00999094 2024-03-21T14:14:35.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:35.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:35.406 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:35.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:35.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:35.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:35.682 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:35.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:35.780 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:35.783 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:35.844 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:35.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:35.982 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:35.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:36.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:36.123 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:36.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:36.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:36.250 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 124705 124689 99.8916 88.25 0.00537819 0.0100066 2024-03-21T14:14:36.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:36.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:36.402 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:36.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:36.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:36.544 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:36.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:36.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:14:36.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:36.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: pgmap v539: 158 pgs: 2 unknown, 156 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-21T14:14:36.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: osdmap e274: 8 total, 8 up, 8 in 2024-03-21T14:14:36.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: 1.3 scrub starts 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: 1.3 scrub ok 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: 1.4 scrub starts 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: 1.4 scrub ok 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: 1.8 scrub starts 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[29576]: 1.8 scrub ok 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:14:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: pgmap v539: 158 pgs: 2 unknown, 156 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-21T14:14:36.659 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: osdmap e274: 8 total, 8 up, 8 in 2024-03-21T14:14:36.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: 1.3 scrub starts 2024-03-21T14:14:36.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: 1.3 scrub ok 2024-03-21T14:14:36.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: 1.4 scrub starts 2024-03-21T14:14:36.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: 1.4 scrub ok 2024-03-21T14:14:36.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: 1.8 scrub starts 2024-03-21T14:14:36.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:36 smithi003 ceph-mon[40510]: 1.8 scrub ok 2024-03-21T14:14:36.677 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:14:36.684 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:36.718 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:14:36.761 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:14:36.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:36.802 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:14:36.822 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:36.845 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:14:36.884 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:14:36.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:36.927 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:14:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: pgmap v539: 158 pgs: 2 unknown, 156 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 105 MiB/s wr, 1.68k op/s 2024-03-21T14:14:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: osdmap e274: 8 total, 8 up, 8 in 2024-03-21T14:14:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: 1.3 scrub starts 2024-03-21T14:14:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: 1.3 scrub ok 2024-03-21T14:14:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: 1.4 scrub starts 2024-03-21T14:14:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: 1.4 scrub ok 2024-03-21T14:14:36.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: 1.8 scrub starts 2024-03-21T14:14:36.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-mon[33862]: 1.8 scrub ok 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:14:36.933 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:14:36.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:36.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:36.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:37.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:37.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:37.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:37.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:37.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:37.249 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 126096 126080 99.7276 86.9375 0.00560471 0.010023 2024-03-21T14:14:37.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:37.386 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:37.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:14:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:14:37.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.534 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:37.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:37.635 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:37.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.672 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:37.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:37.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:14:37.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.809 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:37.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: 1.9 scrub starts 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: 1.9 scrub ok 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: 1.5 scrub starts 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: 1.5 scrub ok 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/274580748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: 1.a scrub starts 2024-03-21T14:14:37.897 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[29576]: 1.a scrub ok 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: 1.9 scrub starts 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: 1.9 scrub ok 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: 1.5 scrub starts 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: 1.5 scrub ok 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/274580748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: 1.a scrub starts 2024-03-21T14:14:37.898 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:37 smithi003 ceph-mon[40510]: 1.a scrub ok 2024-03-21T14:14:37.911 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: 1.9 scrub starts 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: 1.9 scrub ok 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: 1.5 scrub starts 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: 1.5 scrub ok 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/274580748' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: 1.a scrub starts 2024-03-21T14:14:37.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:37 smithi118 ceph-mon[33862]: 1.a scrub ok 2024-03-21T14:14:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:37.945 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:37.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:38.046 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:38.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:38.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:38.184 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:38.219 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:38.252 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:14:38.252972+0000 min lat: 0.00113893 max lat: 0.152179 avg lat: 0.0100359 2024-03-21T14:14:38.252 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:14:38.252 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 127528 127512 99.5972 89.5 0.0129719 0.0100359 2024-03-21T14:14:38.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:38.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:38.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:38.465 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.504 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:38.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:38.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:38.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:38.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:38.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.7 scrub starts 2024-03-21T14:14:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.7 scrub ok 2024-03-21T14:14:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.6 scrub starts 2024-03-21T14:14:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.6 scrub ok 2024-03-21T14:14:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: pgmap v541: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 91 MiB/s wr, 1.44k op/s 2024-03-21T14:14:38.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.1 scrub starts 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[29576]: 1.1 scrub ok 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.7 scrub starts 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.7 scrub ok 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.6 scrub starts 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.6 scrub ok 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: pgmap v541: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 91 MiB/s wr, 1.44k op/s 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.1 scrub starts 2024-03-21T14:14:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:38 smithi003 ceph-mon[40510]: 1.1 scrub ok 2024-03-21T14:14:38.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:38.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:38.886 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:38.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.7 scrub starts 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.7 scrub ok 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.6 scrub starts 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.6 scrub ok 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: pgmap v541: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 91 MiB/s wr, 1.44k op/s 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:14:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:14:38.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.1 scrub starts 2024-03-21T14:14:38.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:38 smithi118 ceph-mon[33862]: 1.1 scrub ok 2024-03-21T14:14:39.027 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:39.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.067 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:39.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:39.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:39.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:39.252 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 15 128983 128968 99.491 91 0.0114455 0.010047 2024-03-21T14:14:39.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:39.345 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:39.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:39.480 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:39.584 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:39.617 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:39.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:39.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:39.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:39.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:39.880 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:39.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:39 smithi118 ceph-mon[33862]: 1.2 scrub starts 2024-03-21T14:14:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:39 smithi118 ceph-mon[33862]: 1.2 scrub ok 2024-03-21T14:14:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:39 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:14:39.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:39 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:14:39.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:39 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:14:39.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:39 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:14:39.981 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[29576]: 1.2 scrub starts 2024-03-21T14:14:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[29576]: 1.2 scrub ok 2024-03-21T14:14:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:14:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:14:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[40510]: 1.2 scrub starts 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[40510]: 1.2 scrub ok 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:14:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:39 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:14:40.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.017 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:40.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:40.118 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:40.157 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:40.252 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 15 130366 130351 99.3317 86.4375 0.00781056 0.0100631 2024-03-21T14:14:40.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:40.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:40.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:40.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:40.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:40.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.623 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:40.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:40.724 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:40.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[29576]: pgmap v542: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-21T14:14:40.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:14:40.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:14:40.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:14:40.739 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:14:40.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[40510]: pgmap v542: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-21T14:14:40.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:14:40.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:14:40.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:14:40.740 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:40 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:14:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.759 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:40.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:40.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:40.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:40.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:40.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:40.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:40 smithi118 ceph-mon[33862]: pgmap v542: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 71 MiB/s wr, 1.13k op/s 2024-03-21T14:14:40.983 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:40 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:14:40.983 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:40 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:14:40.984 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:40 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:14:40.984 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:40 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:14:41.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:41.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:41.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:41.159 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:14:41.159 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:14:41.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.186 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:41.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:41.252 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 131685 131669 99.1274 82.375 0.00840556 0.0100838 2024-03-21T14:14:41.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:41.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:41.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:41.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:41.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:41.594 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:41.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:41.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:41.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:41.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:41.874 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:41.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:41.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:41 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:14:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:41 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:14:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:41 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:14:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:41 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:14:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:41 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:14:41.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:41 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:14:41.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:14:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:14:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:14:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:14:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:14:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:14:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:14:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:14:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:14:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:14:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:14:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:41 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:14:42.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:42.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:42.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:42.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:14:42.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:42.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 133072 133056 98.9792 86.6875 0.00463305 0.0100989 2024-03-21T14:14:42.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:42.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:42.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.433 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:42.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:42.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T14:14:42.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:42.535 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:42.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:42.657 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:14:42.663 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:14:42.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.701 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:42.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:14:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:14:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[29576]: pgmap v543: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 880 op/s 2024-03-21T14:14:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[29576]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:14:42.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:14:42.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:14:42.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:14:42.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:14:42.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[40510]: pgmap v543: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 880 op/s 2024-03-21T14:14:42.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[40510]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:14:42.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:14:42.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:42 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:14:42.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:42.838 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:42 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:14:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:42 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:14:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:42 smithi118 ceph-mon[33862]: pgmap v543: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 56 MiB/s wr, 880 op/s 2024-03-21T14:14:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:42 smithi118 ceph-mon[33862]: from='client.25322 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:14:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:42 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:14:42.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:42 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:14:42.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:42.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:42.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:43.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:43.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:43.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:43.253 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 134391 134375 98.783 82.4375 0.0118333 0.0101187 2024-03-21T14:14:43.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:43.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:43.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.407 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:43.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:43.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:43.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:43.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:43.662 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":543,"stamp":"2024-03-21T14:14:41.477066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8217965028,"num_objects":125583,"num_object_clones":0,"num_object_copies":376749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125583,"num_whiteouts":0,"num_read":1957,"num_read_kb":1750,"num_write":126187,"num_write_kb":8025655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":1237165,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69955,"ondisk_log_size":69955,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":473,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":27501364,"kb_used_data":24072604,"kb_used_omap":12,"kb_used_meta":3428531,"kb_avail":722459852,"statfs":{"total":767960285184,"available":739798888448,"internally_reserved":0,"allocated":24650346496,"data_stored":24647276928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":3510816304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,6,3,6,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":449767906,"num_objects":6860,"num_object_clones":0,"num_object_copies":20580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6860,"num_whiteouts":0,"num_read":-91,"num_read_kb":-135,"num_write":6777,"num_write_kb":437814,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-6,"num_bytes_recovered":-1395416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.698913"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":452,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577667+0000","last_change":"2024-03-21T14:14:03.789055+0000","last_active":"2024-03-21T14:14:35.577667+0000","last_peered":"2024-03-21T14:14:35.577667+0000","last_clean":"2024-03-21T14:14:35.577667+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:14:35.577667+0000","last_undegraded":"2024-03-21T14:14:35.577667+0000","last_fullsized":"2024-03-21T14:14:35.577667+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:03.788946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:03.788946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:10.852590+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":472,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578890+0000","last_change":"2024-03-21T14:14:03.429724+0000","last_active":"2024-03-21T14:14:35.578890+0000","last_peered":"2024-03-21T14:14:35.578890+0000","last_clean":"2024-03-21T14:14:35.578890+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:14:35.578890+0000","last_undegraded":"2024-03-21T14:14:35.578890+0000","last_fullsized":"2024-03-21T14:14:35.578890+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:14:03.429627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:14.891870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":332,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586034+0000","last_change":"2024-03-21T14:13:07.249337+0000","last_active":"2024-03-21T14:14:35.586034+0000","last_peered":"2024-03-21T14:14:35.586034+0000","last_clean":"2024-03-21T14:14:35.586034+0000","last_became_active":"2024-03-21T14:12:00.528378+0000","last_became_peered":"2024-03-21T14:12:00.528378+0000","last_unstale":"2024-03-21T14:14:35.586034+0000","last_undegraded":"2024-03-21T14:14:35.586034+0000","last_fullsized":"2024-03-21T14:14:35.586034+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:39.260871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":502,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577592+0000","last_change":"2024-03-21T14:12:52.533831+0000","last_active":"2024-03-21T14:14:35.577592+0000","last_peered":"2024-03-21T14:14:35.577592+0000","last_clean":"2024-03-21T14:14:35.577592+0000","last_became_active":"2024-03-21T14:12:00.530310+0000","last_became_peered":"2024-03-21T14:12:00.530310+0000","last_unstale":"2024-03-21T14:14:35.577592+0000","last_undegraded":"2024-03-21T14:14:35.577592+0000","last_fullsized":"2024-03-21T14:14:35.577592+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:12:52.533687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:52.533687+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":332,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579367+0000","last_change":"2024-03-21T14:12:49.783355+0000","last_active":"2024-03-21T14:14:35.579367+0000","last_peered":"2024-03-21T14:14:35.579367+0000","last_clean":"2024-03-21T14:14:35.579367+0000","last_became_active":"2024-03-21T14:12:00.531208+0000","last_became_peered":"2024-03-21T14:12:00.531208+0000","last_unstale":"2024-03-21T14:14:35.579367+0000","last_undegraded":"2024-03-21T14:14:35.579367+0000","last_fullsized":"2024-03-21T14:14:35.579367+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:49.783238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:49.783238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":470,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577318+0000","last_change":"2024-03-21T14:14:06.781676+0000","last_active":"2024-03-21T14:14:35.577318+0000","last_peered":"2024-03-21T14:14:35.577318+0000","last_clean":"2024-03-21T14:14:35.577318+0000","last_became_active":"2024-03-21T14:12:00.529555+0000","last_became_peered":"2024-03-21T14:12:00.529555+0000","last_unstale":"2024-03-21T14:14:35.577318+0000","last_undegraded":"2024-03-21T14:14:35.577318+0000","last_fullsized":"2024-03-21T14:14:35.577318+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:17.064470+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":481,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578644+0000","last_change":"2024-03-21T14:12:55.147543+0000","last_active":"2024-03-21T14:14:35.578644+0000","last_peered":"2024-03-21T14:14:35.578644+0000","last_clean":"2024-03-21T14:14:35.578644+0000","last_became_active":"2024-03-21T14:12:00.534344+0000","last_became_peered":"2024-03-21T14:12:00.534344+0000","last_unstale":"2024-03-21T14:14:35.578644+0000","last_undegraded":"2024-03-21T14:14:35.578644+0000","last_fullsized":"2024-03-21T14:14:35.578644+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.147407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.147407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":519,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579077+0000","last_change":"2024-03-21T14:12:48.244384+0000","last_active":"2024-03-21T14:14:35.579077+0000","last_peered":"2024-03-21T14:14:35.579077+0000","last_clean":"2024-03-21T14:14:35.579077+0000","last_became_active":"2024-03-21T14:02:35.442696+0000","last_became_peered":"2024-03-21T14:02:35.442696+0000","last_unstale":"2024-03-21T14:14:35.579077+0000","last_undegraded":"2024-03-21T14:14:35.579077+0000","last_fullsized":"2024-03-21T14:14:35.579077+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:12:48.244225+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:12:48.244225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700256+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":325,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578159+0000","last_change":"2024-03-21T14:13:39.512231+0000","last_active":"2024-03-21T14:14:35.578159+0000","last_peered":"2024-03-21T14:14:35.578159+0000","last_clean":"2024-03-21T14:14:35.578159+0000","last_became_active":"2024-03-21T14:07:23.360834+0000","last_became_peered":"2024-03-21T14:07:23.360834+0000","last_unstale":"2024-03-21T14:14:35.578159+0000","last_undegraded":"2024-03-21T14:14:35.578159+0000","last_fullsized":"2024-03-21T14:14:35.578159+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.510830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.510830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:40.102365+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583467+0000","last_change":"2024-03-21T14:13:53.257162+0000","last_active":"2024-03-21T14:14:35.583467+0000","last_peered":"2024-03-21T14:14:35.583467+0000","last_clean":"2024-03-21T14:14:35.583467+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:14:35.583467+0000","last_undegraded":"2024-03-21T14:14:35.583467+0000","last_fullsized":"2024-03-21T14:14:35.583467+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:53.256723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:13:53.256723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:21.995626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":478,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586677+0000","last_change":"2024-03-21T14:13:10.251700+0000","last_active":"2024-03-21T14:14:35.586677+0000","last_peered":"2024-03-21T14:14:35.586677+0000","last_clean":"2024-03-21T14:14:35.586677+0000","last_became_active":"2024-03-21T14:12:00.528355+0000","last_became_peered":"2024-03-21T14:12:00.528355+0000","last_unstale":"2024-03-21T14:14:35.586677+0000","last_undegraded":"2024-03-21T14:14:35.586677+0000","last_fullsized":"2024-03-21T14:14:35.586677+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:13:10.251587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":477,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577935+0000","last_change":"2024-03-21T14:10:46.814006+0000","last_active":"2024-03-21T14:14:35.577935+0000","last_peered":"2024-03-21T14:14:35.577935+0000","last_clean":"2024-03-21T14:14:35.577935+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:14:35.577935+0000","last_undegraded":"2024-03-21T14:14:35.577935+0000","last_fullsized":"2024-03-21T14:14:35.577935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'9","last_scrub_stamp":"2024-03-21T14:10:46.813854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:46.813854+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974821+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":466,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.584392+0000","last_change":"2024-03-21T14:13:41.341788+0000","last_active":"2024-03-21T14:14:35.584392+0000","last_peered":"2024-03-21T14:14:35.584392+0000","last_clean":"2024-03-21T14:14:35.584392+0000","last_became_active":"2024-03-21T14:07:23.350761+0000","last_became_peered":"2024-03-21T14:07:23.350761+0000","last_unstale":"2024-03-21T14:14:35.584392+0000","last_undegraded":"2024-03-21T14:14:35.584392+0000","last_fullsized":"2024-03-21T14:14:35.584392+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:41.341673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.341673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:44.213517+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":457,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.576884+0000","last_change":"2024-03-21T14:11:08.999188+0000","last_active":"2024-03-21T14:14:35.576884+0000","last_peered":"2024-03-21T14:14:35.576884+0000","last_clean":"2024-03-21T14:14:35.576884+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:14:35.576884+0000","last_undegraded":"2024-03-21T14:14:35.576884+0000","last_fullsized":"2024-03-21T14:14:35.576884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:08.999081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:11:08.999081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.630456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":472,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.585923+0000","last_change":"2024-03-21T14:12:58.205869+0000","last_active":"2024-03-21T14:14:35.585923+0000","last_peered":"2024-03-21T14:14:35.585923+0000","last_clean":"2024-03-21T14:14:35.585923+0000","last_became_active":"2024-03-21T14:02:33.441389+0000","last_became_peered":"2024-03-21T14:02:33.441389+0000","last_unstale":"2024-03-21T14:14:35.585923+0000","last_undegraded":"2024-03-21T14:14:35.585923+0000","last_fullsized":"2024-03-21T14:14:35.585923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.205794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.205794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":473,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579874+0000","last_change":"2024-03-21T14:13:08.544451+0000","last_active":"2024-03-21T14:14:35.579874+0000","last_peered":"2024-03-21T14:14:35.579874+0000","last_clean":"2024-03-21T14:14:35.579874+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:14:35.579874+0000","last_undegraded":"2024-03-21T14:14:35.579874+0000","last_fullsized":"2024-03-21T14:14:35.579874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:13:08.544355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:08.544355+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":460,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:40.999992+0000","last_change":"2024-03-21T14:14:40.999992+0000","last_active":"2024-03-21T14:14:40.999992+0000","last_peered":"2024-03-21T14:14:40.999992+0000","last_clean":"2024-03-21T14:14:40.999992+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:14:40.999992+0000","last_undegraded":"2024-03-21T14:14:40.999992+0000","last_fullsized":"2024-03-21T14:14:40.999992+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:40.999767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:40.999767+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:16:00.788172+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581071+0000","last_change":"2024-03-21T14:13:36.553244+0000","last_active":"2024-03-21T14:14:35.581071+0000","last_peered":"2024-03-21T14:14:35.581071+0000","last_clean":"2024-03-21T14:14:35.581071+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:14:35.581071+0000","last_undegraded":"2024-03-21T14:14:35.581071+0000","last_fullsized":"2024-03-21T14:14:35.581071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:36.553120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.553120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.765113+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":317,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579781+0000","last_change":"2024-03-21T14:13:10.588040+0000","last_active":"2024-03-21T14:14:35.579781+0000","last_peered":"2024-03-21T14:14:35.579781+0000","last_clean":"2024-03-21T14:14:35.579781+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:14:35.579781+0000","last_undegraded":"2024-03-21T14:14:35.579781+0000","last_fullsized":"2024-03-21T14:14:35.579781+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:10.587951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:10.587951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'27","reported_seq":536,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580118+0000","last_change":"2024-03-21T14:13:42.196025+0000","last_active":"2024-03-21T14:14:35.580118+0000","last_peered":"2024-03-21T14:14:35.580118+0000","last_clean":"2024-03-21T14:14:35.580118+0000","last_became_active":"2024-03-21T14:02:35.453792+0000","last_became_peered":"2024-03-21T14:02:35.453792+0000","last_unstale":"2024-03-21T14:14:35.580118+0000","last_undegraded":"2024-03-21T14:14:35.580118+0000","last_fullsized":"2024-03-21T14:14:35.580118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:13:42.195878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:42.195878+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:42.977636+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":468,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579162+0000","last_change":"2024-03-21T14:13:02.282371+0000","last_active":"2024-03-21T14:14:35.579162+0000","last_peered":"2024-03-21T14:14:35.579162+0000","last_clean":"2024-03-21T14:14:35.579162+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:14:35.579162+0000","last_undegraded":"2024-03-21T14:14:35.579162+0000","last_fullsized":"2024-03-21T14:14:35.579162+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:02.282247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:02.282247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700226+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":456,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578076+0000","last_change":"2024-03-21T14:14:00.679668+0000","last_active":"2024-03-21T14:14:35.578076+0000","last_peered":"2024-03-21T14:14:35.578076+0000","last_clean":"2024-03-21T14:14:35.578076+0000","last_became_active":"2024-03-21T14:02:37.457576+0000","last_became_peered":"2024-03-21T14:02:37.457576+0000","last_unstale":"2024-03-21T14:14:35.578076+0000","last_undegraded":"2024-03-21T14:14:35.578076+0000","last_fullsized":"2024-03-21T14:14:35.578076+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.679529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.679529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:01.166486+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578027+0000","last_change":"2024-03-21T14:10:49.867066+0000","last_active":"2024-03-21T14:14:35.578027+0000","last_peered":"2024-03-21T14:14:35.578027+0000","last_clean":"2024-03-21T14:14:35.578027+0000","last_became_active":"2024-03-21T14:02:33.434209+0000","last_became_peered":"2024-03-21T14:02:33.434209+0000","last_unstale":"2024-03-21T14:14:35.578027+0000","last_undegraded":"2024-03-21T14:14:35.578027+0000","last_fullsized":"2024-03-21T14:14:35.578027+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:49.866989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.866989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974869+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":501,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578850+0000","last_change":"2024-03-21T14:13:39.533070+0000","last_active":"2024-03-21T14:14:35.578850+0000","last_peered":"2024-03-21T14:14:35.578850+0000","last_clean":"2024-03-21T14:14:35.578850+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:14:35.578850+0000","last_undegraded":"2024-03-21T14:14:35.578850+0000","last_fullsized":"2024-03-21T14:14:35.578850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:13:39.532904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.532904+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:50.781737+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":471,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580358+0000","last_change":"2024-03-21T14:14:00.547921+0000","last_active":"2024-03-21T14:14:35.580358+0000","last_peered":"2024-03-21T14:14:35.580358+0000","last_clean":"2024-03-21T14:14:35.580358+0000","last_became_active":"2024-03-21T14:02:39.465831+0000","last_became_peered":"2024-03-21T14:02:39.465831+0000","last_unstale":"2024-03-21T14:14:35.580358+0000","last_undegraded":"2024-03-21T14:14:35.580358+0000","last_fullsized":"2024-03-21T14:14:35.580358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.547775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:27.636509+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"72'8","reported_seq":633,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.571073+0000","last_change":"2024-03-21T14:13:38.032421+0000","last_active":"2024-03-21T14:14:35.571073+0000","last_peered":"2024-03-21T14:14:35.571073+0000","last_clean":"2024-03-21T14:14:35.571073+0000","last_became_active":"2024-03-21T14:02:37.455006+0000","last_became_peered":"2024-03-21T14:02:37.455006+0000","last_unstale":"2024-03-21T14:14:35.571073+0000","last_undegraded":"2024-03-21T14:14:35.571073+0000","last_fullsized":"2024-03-21T14:14:35.571073+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:13:38.032095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.032095+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:40.723676+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":481,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578781+0000","last_change":"2024-03-21T14:12:52.744870+0000","last_active":"2024-03-21T14:14:35.578781+0000","last_peered":"2024-03-21T14:14:35.578781+0000","last_clean":"2024-03-21T14:14:35.578781+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:14:35.578781+0000","last_undegraded":"2024-03-21T14:14:35.578781+0000","last_fullsized":"2024-03-21T14:14:35.578781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:12:52.744765+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:12:52.744765+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":168,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578629+0000","last_change":"2024-03-21T14:13:41.601952+0000","last_active":"2024-03-21T14:14:35.578629+0000","last_peered":"2024-03-21T14:14:35.578629+0000","last_clean":"2024-03-21T14:14:35.578629+0000","last_became_active":"2024-03-21T14:12:20.564175+0000","last_became_peered":"2024-03-21T14:12:20.564175+0000","last_unstale":"2024-03-21T14:14:35.578629+0000","last_undegraded":"2024-03-21T14:14:35.578629+0000","last_fullsized":"2024-03-21T14:14:35.578629+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:13:41.601807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.601807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.904689+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":460,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580710+0000","last_change":"2024-03-21T14:13:10.633497+0000","last_active":"2024-03-21T14:14:35.580710+0000","last_peered":"2024-03-21T14:14:35.580710+0000","last_clean":"2024-03-21T14:14:35.580710+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:14:35.580710+0000","last_undegraded":"2024-03-21T14:14:35.580710+0000","last_fullsized":"2024-03-21T14:14:35.580710+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:10.633398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:10.633398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":120,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578541+0000","last_change":"2024-03-21T14:12:59.820203+0000","last_active":"2024-03-21T14:14:35.578541+0000","last_peered":"2024-03-21T14:14:35.578541+0000","last_clean":"2024-03-21T14:14:35.578541+0000","last_became_active":"2024-03-21T14:12:20.562837+0000","last_became_peered":"2024-03-21T14:12:20.562837+0000","last_unstale":"2024-03-21T14:14:35.578541+0000","last_undegraded":"2024-03-21T14:14:35.578541+0000","last_fullsized":"2024-03-21T14:14:35.578541+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.819350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.819350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":487,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577829+0000","last_change":"2024-03-21T14:13:59.621648+0000","last_active":"2024-03-21T14:14:35.577829+0000","last_peered":"2024-03-21T14:14:35.577829+0000","last_clean":"2024-03-21T14:14:35.577829+0000","last_became_active":"2024-03-21T14:07:23.364680+0000","last_became_peered":"2024-03-21T14:07:23.364680+0000","last_unstale":"2024-03-21T14:14:35.577829+0000","last_undegraded":"2024-03-21T14:14:35.577829+0000","last_fullsized":"2024-03-21T14:14:35.577829+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:59.621530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.621530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:29.389896+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":342,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586570+0000","last_change":"2024-03-21T14:13:09.203461+0000","last_active":"2024-03-21T14:14:35.586570+0000","last_peered":"2024-03-21T14:14:35.586570+0000","last_clean":"2024-03-21T14:14:35.586570+0000","last_became_active":"2024-03-21T14:12:00.536809+0000","last_became_peered":"2024-03-21T14:12:00.536809+0000","last_unstale":"2024-03-21T14:14:35.586570+0000","last_undegraded":"2024-03-21T14:14:35.586570+0000","last_fullsized":"2024-03-21T14:14:35.586570+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:13:09.203367+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:13:09.203367+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316667+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":462,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579166+0000","last_change":"2024-03-21T14:12:48.717566+0000","last_active":"2024-03-21T14:14:35.579166+0000","last_peered":"2024-03-21T14:14:35.579166+0000","last_clean":"2024-03-21T14:14:35.579166+0000","last_became_active":"2024-03-21T14:02:39.466558+0000","last_became_peered":"2024-03-21T14:02:39.466558+0000","last_unstale":"2024-03-21T14:14:35.579166+0000","last_undegraded":"2024-03-21T14:14:35.579166+0000","last_fullsized":"2024-03-21T14:14:35.579166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:48.717461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:48.717461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.630559+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":147,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577289+0000","last_change":"2024-03-21T14:13:55.540387+0000","last_active":"2024-03-21T14:14:35.577289+0000","last_peered":"2024-03-21T14:14:35.577289+0000","last_clean":"2024-03-21T14:14:35.577289+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:14:35.577289+0000","last_undegraded":"2024-03-21T14:14:35.577289+0000","last_fullsized":"2024-03-21T14:14:35.577289+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:55.540139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.540139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:24.763371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":118,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586370+0000","last_change":"2024-03-21T14:13:00.210250+0000","last_active":"2024-03-21T14:14:35.586370+0000","last_peered":"2024-03-21T14:14:35.586370+0000","last_clean":"2024-03-21T14:14:35.586370+0000","last_became_active":"2024-03-21T14:12:20.560066+0000","last_became_peered":"2024-03-21T14:12:20.560066+0000","last_unstale":"2024-03-21T14:14:35.586370+0000","last_undegraded":"2024-03-21T14:14:35.586370+0000","last_fullsized":"2024-03-21T14:14:35.586370+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:00.210172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:00.210172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"236'8","reported_seq":494,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577944+0000","last_change":"2024-03-21T14:12:07.940160+0000","last_active":"2024-03-21T14:14:35.577944+0000","last_peered":"2024-03-21T14:14:35.577944+0000","last_clean":"2024-03-21T14:14:35.577944+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:14:35.577944+0000","last_undegraded":"2024-03-21T14:14:35.577944+0000","last_fullsized":"2024-03-21T14:14:35.577944+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'4","last_scrub_stamp":"2024-03-21T14:12:07.940015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:07.940015+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700144+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":480,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578888+0000","last_change":"2024-03-21T14:12:51.294104+0000","last_active":"2024-03-21T14:14:35.578888+0000","last_peered":"2024-03-21T14:14:35.578888+0000","last_clean":"2024-03-21T14:14:35.578888+0000","last_became_active":"2024-03-21T14:12:00.536387+0000","last_became_peered":"2024-03-21T14:12:00.536387+0000","last_unstale":"2024-03-21T14:14:35.578888+0000","last_undegraded":"2024-03-21T14:14:35.578888+0000","last_fullsized":"2024-03-21T14:14:35.578888+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:51.293955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:12:51.293955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.701411+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":480,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.576333+0000","last_change":"2024-03-21T14:13:54.518871+0000","last_active":"2024-03-21T14:14:35.576333+0000","last_peered":"2024-03-21T14:14:35.576333+0000","last_clean":"2024-03-21T14:14:35.576333+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:14:35.576333+0000","last_undegraded":"2024-03-21T14:14:35.576333+0000","last_fullsized":"2024-03-21T14:14:35.576333+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:54.518724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:54.518724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:15.950041+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":478,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.590032+0000","last_change":"2024-03-21T14:12:54.264971+0000","last_active":"2024-03-21T14:14:35.590032+0000","last_peered":"2024-03-21T14:14:35.590032+0000","last_clean":"2024-03-21T14:14:35.590032+0000","last_became_active":"2024-03-21T14:02:33.442922+0000","last_became_peered":"2024-03-21T14:02:33.442922+0000","last_unstale":"2024-03-21T14:14:35.590032+0000","last_undegraded":"2024-03-21T14:14:35.590032+0000","last_fullsized":"2024-03-21T14:14:35.590032+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:12:54.264874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.264874+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":532,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579166+0000","last_change":"2024-03-21T14:13:55.514810+0000","last_active":"2024-03-21T14:14:35.579166+0000","last_peered":"2024-03-21T14:14:35.579166+0000","last_clean":"2024-03-21T14:14:35.579166+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:14:35.579166+0000","last_undegraded":"2024-03-21T14:14:35.579166+0000","last_fullsized":"2024-03-21T14:14:35.579166+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.514646+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:19.498523+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":118,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.585429+0000","last_change":"2024-03-21T14:13:06.269597+0000","last_active":"2024-03-21T14:14:35.585429+0000","last_peered":"2024-03-21T14:14:35.585429+0000","last_clean":"2024-03-21T14:14:35.585429+0000","last_became_active":"2024-03-21T14:12:20.560601+0000","last_became_peered":"2024-03-21T14:12:20.560601+0000","last_unstale":"2024-03-21T14:14:35.585429+0000","last_undegraded":"2024-03-21T14:14:35.585429+0000","last_fullsized":"2024-03-21T14:14:35.585429+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:06.269490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:06.269490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316299+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":466,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578779+0000","last_change":"2024-03-21T14:13:48.467696+0000","last_active":"2024-03-21T14:14:35.578779+0000","last_peered":"2024-03-21T14:14:35.578779+0000","last_clean":"2024-03-21T14:14:35.578779+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:14:35.578779+0000","last_undegraded":"2024-03-21T14:14:35.578779+0000","last_fullsized":"2024-03-21T14:14:35.578779+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:48.467442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:13:48.467442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:03.673244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":474,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578059+0000","last_change":"2024-03-21T14:13:04.662406+0000","last_active":"2024-03-21T14:14:35.578059+0000","last_peered":"2024-03-21T14:14:35.578059+0000","last_clean":"2024-03-21T14:14:35.578059+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:14:35.578059+0000","last_undegraded":"2024-03-21T14:14:35.578059+0000","last_fullsized":"2024-03-21T14:14:35.578059+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:04.662327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:04.662327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":518,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583385+0000","last_change":"2024-03-21T14:13:50.493055+0000","last_active":"2024-03-21T14:14:35.583385+0000","last_peered":"2024-03-21T14:14:35.583385+0000","last_clean":"2024-03-21T14:14:35.583385+0000","last_became_active":"2024-03-21T14:12:00.536162+0000","last_became_peered":"2024-03-21T14:12:00.536162+0000","last_unstale":"2024-03-21T14:14:35.583385+0000","last_undegraded":"2024-03-21T14:14:35.583385+0000","last_fullsized":"2024-03-21T14:14:35.583385+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.492830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.492830+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:18.194658+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":120,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.584481+0000","last_change":"2024-03-21T14:13:01.256791+0000","last_active":"2024-03-21T14:14:35.584481+0000","last_peered":"2024-03-21T14:14:35.584481+0000","last_clean":"2024-03-21T14:14:35.584481+0000","last_became_active":"2024-03-21T14:12:20.561889+0000","last_became_peered":"2024-03-21T14:12:20.561889+0000","last_unstale":"2024-03-21T14:14:35.584481+0000","last_undegraded":"2024-03-21T14:14:35.584481+0000","last_fullsized":"2024-03-21T14:14:35.584481+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:01.256711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:01.256711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":479,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583230+0000","last_change":"2024-03-21T14:13:56.430898+0000","last_active":"2024-03-21T14:14:35.583230+0000","last_peered":"2024-03-21T14:14:35.583230+0000","last_clean":"2024-03-21T14:14:35.583230+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:14:35.583230+0000","last_undegraded":"2024-03-21T14:14:35.583230+0000","last_fullsized":"2024-03-21T14:14:35.583230+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:56.430507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:56.430507+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:05.852233+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":479,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577838+0000","last_change":"2024-03-21T14:12:00.531790+0000","last_active":"2024-03-21T14:14:35.577838+0000","last_peered":"2024-03-21T14:14:35.577838+0000","last_clean":"2024-03-21T14:14:35.577838+0000","last_became_active":"2024-03-21T14:12:00.531406+0000","last_became_peered":"2024-03-21T14:12:00.531406+0000","last_unstale":"2024-03-21T14:14:35.577838+0000","last_undegraded":"2024-03-21T14:14:35.577838+0000","last_fullsized":"2024-03-21T14:14:35.577838+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:42.706023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:10:42.706023+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":515,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579983+0000","last_change":"2024-03-21T14:10:54.800377+0000","last_active":"2024-03-21T14:14:35.579983+0000","last_peered":"2024-03-21T14:14:35.579983+0000","last_clean":"2024-03-21T14:14:35.579983+0000","last_became_active":"2024-03-21T14:02:58.505108+0000","last_became_peered":"2024-03-21T14:02:58.505108+0000","last_unstale":"2024-03-21T14:14:35.579983+0000","last_undegraded":"2024-03-21T14:14:35.579983+0000","last_fullsized":"2024-03-21T14:14:35.579983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"153'16","last_scrub_stamp":"2024-03-21T14:10:54.800107+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:10:54.800107+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":318,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578159+0000","last_change":"2024-03-21T14:11:06.009897+0000","last_active":"2024-03-21T14:14:35.578159+0000","last_peered":"2024-03-21T14:14:35.578159+0000","last_clean":"2024-03-21T14:14:35.578159+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:14:35.578159+0000","last_undegraded":"2024-03-21T14:14:35.578159+0000","last_fullsized":"2024-03-21T14:14:35.578159+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:06.009710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:11:06.009710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.630484+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"72'8","reported_seq":625,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586769+0000","last_change":"2024-03-21T14:13:34.317612+0000","last_active":"2024-03-21T14:14:35.586769+0000","last_peered":"2024-03-21T14:14:35.586769+0000","last_clean":"2024-03-21T14:14:35.586769+0000","last_became_active":"2024-03-21T14:02:37.457378+0000","last_became_peered":"2024-03-21T14:02:37.457378+0000","last_unstale":"2024-03-21T14:14:35.586769+0000","last_undegraded":"2024-03-21T14:14:35.586769+0000","last_fullsized":"2024-03-21T14:14:35.586769+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:13:34.317492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:34.317492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:47.112660+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":483,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581028+0000","last_change":"2024-03-21T14:13:34.724463+0000","last_active":"2024-03-21T14:14:35.581028+0000","last_peered":"2024-03-21T14:14:35.581028+0000","last_clean":"2024-03-21T14:14:35.581028+0000","last_became_active":"2024-03-21T14:08:17.142668+0000","last_became_peered":"2024-03-21T14:08:17.142668+0000","last_unstale":"2024-03-21T14:14:35.581028+0000","last_undegraded":"2024-03-21T14:14:35.581028+0000","last_fullsized":"2024-03-21T14:14:35.581028+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:34.724192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:13:34.724192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:45.746660+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":497,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580606+0000","last_change":"2024-03-21T14:12:08.873079+0000","last_active":"2024-03-21T14:14:35.580606+0000","last_peered":"2024-03-21T14:14:35.580606+0000","last_clean":"2024-03-21T14:14:35.580606+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:14:35.580606+0000","last_undegraded":"2024-03-21T14:14:35.580606+0000","last_fullsized":"2024-03-21T14:14:35.580606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'15","last_scrub_stamp":"2024-03-21T14:12:08.872861+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:12:08.872861+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700104+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":452,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579139+0000","last_change":"2024-03-21T14:12:58.643025+0000","last_active":"2024-03-21T14:14:35.579139+0000","last_peered":"2024-03-21T14:14:35.579139+0000","last_clean":"2024-03-21T14:14:35.579139+0000","last_became_active":"2024-03-21T14:02:39.465482+0000","last_became_peered":"2024-03-21T14:02:39.465482+0000","last_unstale":"2024-03-21T14:14:35.579139+0000","last_undegraded":"2024-03-21T14:14:35.579139+0000","last_fullsized":"2024-03-21T14:14:35.579139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975220+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"130'9","reported_seq":443,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580254+0000","last_change":"2024-03-21T14:13:03.639121+0000","last_active":"2024-03-21T14:14:35.580254+0000","last_peered":"2024-03-21T14:14:35.580254+0000","last_clean":"2024-03-21T14:14:35.580254+0000","last_became_active":"2024-03-21T14:07:23.354945+0000","last_became_peered":"2024-03-21T14:07:23.354945+0000","last_unstale":"2024-03-21T14:14:35.580254+0000","last_undegraded":"2024-03-21T14:14:35.580254+0000","last_fullsized":"2024-03-21T14:14:35.580254+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:13:03.638324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.638324+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.632180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":274,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.576627+0000","last_change":"2024-03-21T14:13:44.667633+0000","last_active":"2024-03-21T14:14:35.576627+0000","last_peered":"2024-03-21T14:14:35.576627+0000","last_clean":"2024-03-21T14:14:35.576627+0000","last_became_active":"2024-03-21T14:12:20.562532+0000","last_became_peered":"2024-03-21T14:12:20.562532+0000","last_unstale":"2024-03-21T14:14:35.576627+0000","last_undegraded":"2024-03-21T14:14:35.576627+0000","last_fullsized":"2024-03-21T14:14:35.576627+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:44.667480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:44.667480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:46.142833+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":493,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.588190+0000","last_change":"2024-03-21T14:13:52.240889+0000","last_active":"2024-03-21T14:14:35.588190+0000","last_peered":"2024-03-21T14:14:35.588190+0000","last_clean":"2024-03-21T14:14:35.588190+0000","last_became_active":"2024-03-21T14:02:35.775132+0000","last_became_peered":"2024-03-21T14:02:35.775132+0000","last_unstale":"2024-03-21T14:14:35.588190+0000","last_undegraded":"2024-03-21T14:14:35.588190+0000","last_fullsized":"2024-03-21T14:14:35.588190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:52.240603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:52.240603+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:58.617545+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":470,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580502+0000","last_change":"2024-03-21T14:12:53.155160+0000","last_active":"2024-03-21T14:14:35.580502+0000","last_peered":"2024-03-21T14:14:35.580502+0000","last_clean":"2024-03-21T14:14:35.580502+0000","last_became_active":"2024-03-21T14:02:39.463606+0000","last_became_peered":"2024-03-21T14:02:39.463606+0000","last_unstale":"2024-03-21T14:14:35.580502+0000","last_undegraded":"2024-03-21T14:14:35.580502+0000","last_fullsized":"2024-03-21T14:14:35.580502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:53.155075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.155075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.314764+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":317,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.582724+0000","last_change":"2024-03-21T14:12:56.663622+0000","last_active":"2024-03-21T14:14:35.582724+0000","last_peered":"2024-03-21T14:14:35.582724+0000","last_clean":"2024-03-21T14:14:35.582724+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:14:35.582724+0000","last_undegraded":"2024-03-21T14:14:35.582724+0000","last_fullsized":"2024-03-21T14:14:35.582724+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:56.663519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:12:56.663519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599255+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":474,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581091+0000","last_change":"2024-03-21T14:12:00.761561+0000","last_active":"2024-03-21T14:14:35.581091+0000","last_peered":"2024-03-21T14:14:35.581091+0000","last_clean":"2024-03-21T14:14:35.581091+0000","last_became_active":"2024-03-21T14:02:33.447047+0000","last_became_peered":"2024-03-21T14:02:33.447047+0000","last_unstale":"2024-03-21T14:14:35.581091+0000","last_undegraded":"2024-03-21T14:14:35.581091+0000","last_fullsized":"2024-03-21T14:14:35.581091+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:00.761483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:00.761483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.630509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":162,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578670+0000","last_change":"2024-03-21T14:13:37.488188+0000","last_active":"2024-03-21T14:14:35.578670+0000","last_peered":"2024-03-21T14:14:35.578670+0000","last_clean":"2024-03-21T14:14:35.578670+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:14:35.578670+0000","last_undegraded":"2024-03-21T14:14:35.578670+0000","last_fullsized":"2024-03-21T14:14:35.578670+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:37.488036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:37.488036+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:47.071726+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":467,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577926+0000","last_change":"2024-03-21T14:13:38.740375+0000","last_active":"2024-03-21T14:14:35.577926+0000","last_peered":"2024-03-21T14:14:35.577926+0000","last_clean":"2024-03-21T14:14:35.577926+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:14:35.577926+0000","last_undegraded":"2024-03-21T14:14:35.577926+0000","last_fullsized":"2024-03-21T14:14:35.577926+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:38.740248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.740248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:40.988073+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":464,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.585473+0000","last_change":"2024-03-21T14:12:57.241462+0000","last_active":"2024-03-21T14:14:35.585473+0000","last_peered":"2024-03-21T14:14:35.585473+0000","last_clean":"2024-03-21T14:14:35.585473+0000","last_became_active":"2024-03-21T14:02:37.453402+0000","last_became_peered":"2024-03-21T14:02:37.453402+0000","last_unstale":"2024-03-21T14:14:35.585473+0000","last_undegraded":"2024-03-21T14:14:35.585473+0000","last_fullsized":"2024-03-21T14:14:35.585473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:57.241377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:12:57.241377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":495,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.571427+0000","last_change":"2024-03-21T14:13:39.023162+0000","last_active":"2024-03-21T14:14:35.571427+0000","last_peered":"2024-03-21T14:14:35.571427+0000","last_clean":"2024-03-21T14:14:35.571427+0000","last_became_active":"2024-03-21T14:07:23.354457+0000","last_became_peered":"2024-03-21T14:07:23.354457+0000","last_unstale":"2024-03-21T14:14:35.571427+0000","last_undegraded":"2024-03-21T14:14:35.571427+0000","last_fullsized":"2024-03-21T14:14:35.571427+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.023049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.023049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:04.473366+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":371,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581350+0000","last_change":"2024-03-21T14:13:59.568156+0000","last_active":"2024-03-21T14:14:35.581350+0000","last_peered":"2024-03-21T14:14:35.581350+0000","last_clean":"2024-03-21T14:14:35.581350+0000","last_became_active":"2024-03-21T14:07:23.365278+0000","last_became_peered":"2024-03-21T14:07:23.365278+0000","last_unstale":"2024-03-21T14:14:35.581350+0000","last_undegraded":"2024-03-21T14:14:35.581350+0000","last_fullsized":"2024-03-21T14:14:35.581350+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:13:59.567870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.567870+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:59.574445+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"274'7762","reported_seq":7832,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.552451+0000","last_change":"2024-03-21T14:13:09.117421+0000","last_active":"2024-03-21T14:14:36.552451+0000","last_peered":"2024-03-21T14:14:36.552451+0000","last_clean":"2024-03-21T14:14:36.552451+0000","last_became_active":"2024-03-21T14:13:09.117000+0000","last_became_peered":"2024-03-21T14:13:09.117000+0000","last_unstale":"2024-03-21T14:14:36.552451+0000","last_undegraded":"2024-03-21T14:14:36.552451+0000","last_fullsized":"2024-03-21T14:14:36.552451+0000","mapping_epoch":244,"log_start":"255'3394","ondisk_log_start":"255'3394","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4368,"log_dups_size":0,"ondisk_log_size":4368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508690432,"num_objects":7762,"num_object_clones":0,"num_object_copies":23286,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7762,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7762,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":473,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.588293+0000","last_change":"2024-03-21T14:13:36.400555+0000","last_active":"2024-03-21T14:14:35.588293+0000","last_peered":"2024-03-21T14:14:35.588293+0000","last_clean":"2024-03-21T14:14:35.588293+0000","last_became_active":"2024-03-21T14:13:18.544191+0000","last_became_peered":"2024-03-21T14:13:18.544191+0000","last_unstale":"2024-03-21T14:14:35.588293+0000","last_undegraded":"2024-03-21T14:14:35.588293+0000","last_fullsized":"2024-03-21T14:14:35.588293+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:36.400445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.400445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:58.782401+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":498,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581201+0000","last_change":"2024-03-21T14:12:57.742705+0000","last_active":"2024-03-21T14:14:35.581201+0000","last_peered":"2024-03-21T14:14:35.581201+0000","last_clean":"2024-03-21T14:14:35.581201+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:14:35.581201+0000","last_undegraded":"2024-03-21T14:14:35.581201+0000","last_fullsized":"2024-03-21T14:14:35.581201+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:12:57.742582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:57.742582+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":534,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583913+0000","last_change":"2024-03-21T14:13:05.267472+0000","last_active":"2024-03-21T14:14:35.583913+0000","last_peered":"2024-03-21T14:14:35.583913+0000","last_clean":"2024-03-21T14:14:35.583913+0000","last_became_active":"2024-03-21T14:12:00.537606+0000","last_became_peered":"2024-03-21T14:12:00.537606+0000","last_unstale":"2024-03-21T14:14:35.583913+0000","last_undegraded":"2024-03-21T14:14:35.583913+0000","last_fullsized":"2024-03-21T14:14:35.583913+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:13:05.267363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:05.267363+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316797+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"214'9","reported_seq":629,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581222+0000","last_change":"2024-03-21T14:14:07.574814+0000","last_active":"2024-03-21T14:14:35.581222+0000","last_peered":"2024-03-21T14:14:35.581222+0000","last_clean":"2024-03-21T14:14:35.581222+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:14:35.581222+0000","last_undegraded":"2024-03-21T14:14:35.581222+0000","last_fullsized":"2024-03-21T14:14:35.581222+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'9","last_scrub_stamp":"2024-03-21T14:14:07.574692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.574692+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:24.335788+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"274'7800","reported_seq":7870,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:37.222509+0000","last_change":"2024-03-21T14:13:09.392712+0000","last_active":"2024-03-21T14:14:37.222509+0000","last_peered":"2024-03-21T14:14:37.222509+0000","last_clean":"2024-03-21T14:14:37.222509+0000","last_became_active":"2024-03-21T14:13:09.392115+0000","last_became_peered":"2024-03-21T14:13:09.392115+0000","last_unstale":"2024-03-21T14:14:37.222509+0000","last_undegraded":"2024-03-21T14:14:37.222509+0000","last_fullsized":"2024-03-21T14:14:37.222509+0000","mapping_epoch":244,"log_start":"262'4375","ondisk_log_start":"262'4375","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511180800,"num_objects":7800,"num_object_clones":0,"num_object_copies":23400,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7800,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7800,"num_write_kb":499200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":204,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578504+0000","last_change":"2024-03-21T14:14:01.739227+0000","last_active":"2024-03-21T14:14:35.578504+0000","last_peered":"2024-03-21T14:14:35.578504+0000","last_clean":"2024-03-21T14:14:35.578504+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:14:35.578504+0000","last_undegraded":"2024-03-21T14:14:35.578504+0000","last_fullsized":"2024-03-21T14:14:35.578504+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:01.739104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:01.739104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:30.893064+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":482,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578449+0000","last_change":"2024-03-21T14:13:45.695342+0000","last_active":"2024-03-21T14:14:35.578449+0000","last_peered":"2024-03-21T14:14:35.578449+0000","last_clean":"2024-03-21T14:14:35.578449+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:14:35.578449+0000","last_undegraded":"2024-03-21T14:14:35.578449+0000","last_fullsized":"2024-03-21T14:14:35.578449+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:45.695201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:45.695201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:05.891374+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":529,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577625+0000","last_change":"2024-03-21T14:13:36.652349+0000","last_active":"2024-03-21T14:14:35.577625+0000","last_peered":"2024-03-21T14:14:35.577625+0000","last_clean":"2024-03-21T14:14:35.577625+0000","last_became_active":"2024-03-21T14:12:00.540768+0000","last_became_peered":"2024-03-21T14:12:00.540768+0000","last_unstale":"2024-03-21T14:14:35.577625+0000","last_undegraded":"2024-03-21T14:14:35.577625+0000","last_fullsized":"2024-03-21T14:14:35.577625+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:13:36.652162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.652162+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:00.176123+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":482,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580341+0000","last_change":"2024-03-21T14:12:54.200661+0000","last_active":"2024-03-21T14:14:35.580341+0000","last_peered":"2024-03-21T14:14:35.580341+0000","last_clean":"2024-03-21T14:14:35.580341+0000","last_became_active":"2024-03-21T14:02:37.457074+0000","last_became_peered":"2024-03-21T14:02:37.457074+0000","last_unstale":"2024-03-21T14:14:35.580341+0000","last_undegraded":"2024-03-21T14:14:35.580341+0000","last_fullsized":"2024-03-21T14:14:35.580341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:54.200554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.200554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.314721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"274'7850","reported_seq":7920,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.521879+0000","last_change":"2024-03-21T14:13:08.815145+0000","last_active":"2024-03-21T14:14:36.521879+0000","last_peered":"2024-03-21T14:14:36.521879+0000","last_clean":"2024-03-21T14:14:36.521879+0000","last_became_active":"2024-03-21T14:13:08.814832+0000","last_became_peered":"2024-03-21T14:13:08.814832+0000","last_unstale":"2024-03-21T14:14:36.521879+0000","last_undegraded":"2024-03-21T14:14:36.521879+0000","last_fullsized":"2024-03-21T14:14:36.521879+0000","mapping_epoch":244,"log_start":"255'3454","ondisk_log_start":"255'3454","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514457600,"num_objects":7850,"num_object_clones":0,"num_object_copies":23550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7850,"num_write_kb":502400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":462,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.576006+0000","last_change":"2024-03-21T14:12:54.779559+0000","last_active":"2024-03-21T14:14:35.576006+0000","last_peered":"2024-03-21T14:14:35.576006+0000","last_clean":"2024-03-21T14:14:35.576006+0000","last_became_active":"2024-03-21T14:02:39.469878+0000","last_became_peered":"2024-03-21T14:02:39.469878+0000","last_unstale":"2024-03-21T14:14:35.576006+0000","last_undegraded":"2024-03-21T14:14:35.576006+0000","last_fullsized":"2024-03-21T14:14:35.576006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:54.779479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.779479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":482,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579972+0000","last_change":"2024-03-21T14:13:43.154627+0000","last_active":"2024-03-21T14:14:35.579972+0000","last_peered":"2024-03-21T14:14:35.579972+0000","last_clean":"2024-03-21T14:14:35.579972+0000","last_became_active":"2024-03-21T14:02:37.459048+0000","last_became_peered":"2024-03-21T14:02:37.459048+0000","last_unstale":"2024-03-21T14:14:35.579972+0000","last_undegraded":"2024-03-21T14:14:35.579972+0000","last_fullsized":"2024-03-21T14:14:35.579972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:43.154506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.154506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:57.654214+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":474,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578249+0000","last_change":"2024-03-21T14:13:48.637245+0000","last_active":"2024-03-21T14:14:35.578249+0000","last_peered":"2024-03-21T14:14:35.578249+0000","last_clean":"2024-03-21T14:14:35.578249+0000","last_became_active":"2024-03-21T14:02:33.445271+0000","last_became_peered":"2024-03-21T14:02:33.445271+0000","last_unstale":"2024-03-21T14:14:35.578249+0000","last_undegraded":"2024-03-21T14:14:35.578249+0000","last_fullsized":"2024-03-21T14:14:35.578249+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:48.637120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:48.637120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:08.082382+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":422,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581805+0000","last_change":"2024-03-21T14:13:57.469144+0000","last_active":"2024-03-21T14:14:35.581805+0000","last_peered":"2024-03-21T14:14:35.581805+0000","last_clean":"2024-03-21T14:14:35.581805+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:14:35.581805+0000","last_undegraded":"2024-03-21T14:14:35.581805+0000","last_fullsized":"2024-03-21T14:14:35.581805+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:13:57.468072+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:11:10.962498+0000","last_clean_scrub_stamp":"2024-03-21T14:13:57.468072+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:05.076240+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"274'7774","reported_seq":7844,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.527552+0000","last_change":"2024-03-21T14:13:08.812607+0000","last_active":"2024-03-21T14:14:36.527552+0000","last_peered":"2024-03-21T14:14:36.527552+0000","last_clean":"2024-03-21T14:14:36.527552+0000","last_became_active":"2024-03-21T14:13:08.812224+0000","last_became_peered":"2024-03-21T14:13:08.812224+0000","last_unstale":"2024-03-21T14:14:36.527552+0000","last_undegraded":"2024-03-21T14:14:36.527552+0000","last_fullsized":"2024-03-21T14:14:36.527552+0000","mapping_epoch":244,"log_start":"255'3426","ondisk_log_start":"255'3426","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509476864,"num_objects":7774,"num_object_clones":0,"num_object_copies":23322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":497536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":319,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580753+0000","last_change":"2024-03-21T14:12:08.632071+0000","last_active":"2024-03-21T14:14:35.580753+0000","last_peered":"2024-03-21T14:14:35.580753+0000","last_clean":"2024-03-21T14:14:35.580753+0000","last_became_active":"2024-03-21T14:07:23.354443+0000","last_became_peered":"2024-03-21T14:07:23.354443+0000","last_unstale":"2024-03-21T14:14:35.580753+0000","last_undegraded":"2024-03-21T14:14:35.580753+0000","last_fullsized":"2024-03-21T14:14:35.580753+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:08.631976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:12:08.631976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.630534+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":482,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580175+0000","last_change":"2024-03-21T14:13:00.119409+0000","last_active":"2024-03-21T14:14:35.580175+0000","last_peered":"2024-03-21T14:14:35.580175+0000","last_clean":"2024-03-21T14:14:35.580175+0000","last_became_active":"2024-03-21T14:02:37.456559+0000","last_became_peered":"2024-03-21T14:02:37.456559+0000","last_unstale":"2024-03-21T14:14:35.580175+0000","last_undegraded":"2024-03-21T14:14:35.580175+0000","last_fullsized":"2024-03-21T14:14:35.580175+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:00.119325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:13:00.119325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.314796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":479,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586002+0000","last_change":"2024-03-21T14:13:03.296024+0000","last_active":"2024-03-21T14:14:35.586002+0000","last_peered":"2024-03-21T14:14:35.586002+0000","last_clean":"2024-03-21T14:14:35.586002+0000","last_became_active":"2024-03-21T14:12:00.530707+0000","last_became_peered":"2024-03-21T14:12:00.530707+0000","last_unstale":"2024-03-21T14:14:35.586002+0000","last_undegraded":"2024-03-21T14:14:35.586002+0000","last_fullsized":"2024-03-21T14:14:35.586002+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:03.295889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.295889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":176,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580316+0000","last_change":"2024-03-21T14:13:50.159521+0000","last_active":"2024-03-21T14:14:35.580316+0000","last_peered":"2024-03-21T14:14:35.580316+0000","last_clean":"2024-03-21T14:14:35.580316+0000","last_became_active":"2024-03-21T14:12:20.560843+0000","last_became_peered":"2024-03-21T14:12:20.560843+0000","last_unstale":"2024-03-21T14:14:35.580316+0000","last_undegraded":"2024-03-21T14:14:35.580316+0000","last_fullsized":"2024-03-21T14:14:35.580316+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.159386+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:54.974885+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":529,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.633328+0000","last_change":"2024-03-21T14:14:35.632982+0000","last_active":"2024-03-21T14:14:35.633328+0000","last_peered":"2024-03-21T14:14:35.633328+0000","last_clean":"2024-03-21T14:14:35.633328+0000","last_became_active":"2024-03-21T14:14:35.632592+0000","last_became_peered":"2024-03-21T14:14:35.632592+0000","last_unstale":"2024-03-21T14:14:35.633328+0000","last_undegraded":"2024-03-21T14:14:35.633328+0000","last_fullsized":"2024-03-21T14:14:35.633328+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:11:09.977051+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:11:09.977051+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"274'7826","reported_seq":7895,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.462160+0000","last_change":"2024-03-21T14:13:09.117304+0000","last_active":"2024-03-21T14:14:36.462160+0000","last_peered":"2024-03-21T14:14:36.462160+0000","last_clean":"2024-03-21T14:14:36.462160+0000","last_became_active":"2024-03-21T14:13:09.116976+0000","last_became_peered":"2024-03-21T14:13:09.116976+0000","last_unstale":"2024-03-21T14:14:36.462160+0000","last_undegraded":"2024-03-21T14:14:36.462160+0000","last_fullsized":"2024-03-21T14:14:36.462160+0000","mapping_epoch":244,"log_start":"258'3996","ondisk_log_start":"258'3996","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3830,"log_dups_size":0,"ondisk_log_size":3830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512884736,"num_objects":7826,"num_object_clones":0,"num_object_copies":23478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7826,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7826,"num_write_kb":500864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":454,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578697+0000","last_change":"2024-03-21T14:12:09.871728+0000","last_active":"2024-03-21T14:14:35.578697+0000","last_peered":"2024-03-21T14:14:35.578697+0000","last_clean":"2024-03-21T14:14:35.578697+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:14:35.578697+0000","last_undegraded":"2024-03-21T14:14:35.578697+0000","last_fullsized":"2024-03-21T14:14:35.578697+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:09.871639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:09.871639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700068+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":502,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583969+0000","last_change":"2024-03-21T14:13:48.229671+0000","last_active":"2024-03-21T14:14:35.583969+0000","last_peered":"2024-03-21T14:14:35.583969+0000","last_clean":"2024-03-21T14:14:35.583969+0000","last_became_active":"2024-03-21T14:12:00.538240+0000","last_became_peered":"2024-03-21T14:12:00.538240+0000","last_unstale":"2024-03-21T14:14:35.583969+0000","last_undegraded":"2024-03-21T14:14:35.583969+0000","last_fullsized":"2024-03-21T14:14:35.583969+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:13:48.229489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:13:48.229489+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:14.531882+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":471,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579622+0000","last_change":"2024-03-21T14:14:05.796577+0000","last_active":"2024-03-21T14:14:35.579622+0000","last_peered":"2024-03-21T14:14:35.579622+0000","last_clean":"2024-03-21T14:14:35.579622+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:14:35.579622+0000","last_undegraded":"2024-03-21T14:14:35.579622+0000","last_fullsized":"2024-03-21T14:14:35.579622+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:05.796461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:05.796461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:33.125930+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"236'51","reported_seq":644,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.585704+0000","last_change":"2024-03-21T14:12:51.291020+0000","last_active":"2024-03-21T14:14:35.585704+0000","last_peered":"2024-03-21T14:14:35.585704+0000","last_clean":"2024-03-21T14:14:35.585704+0000","last_became_active":"2024-03-21T14:02:58.509000+0000","last_became_peered":"2024-03-21T14:02:58.509000+0000","last_unstale":"2024-03-21T14:14:35.585704+0000","last_undegraded":"2024-03-21T14:14:35.585704+0000","last_fullsized":"2024-03-21T14:14:35.585704+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-03-21T14:12:51.290920+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:12:51.290920+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":333,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581254+0000","last_change":"2024-03-21T14:13:41.480464+0000","last_active":"2024-03-21T14:14:35.581254+0000","last_peered":"2024-03-21T14:14:35.581254+0000","last_clean":"2024-03-21T14:14:35.581254+0000","last_became_active":"2024-03-21T14:12:00.526762+0000","last_became_peered":"2024-03-21T14:12:00.526762+0000","last_unstale":"2024-03-21T14:14:35.581254+0000","last_undegraded":"2024-03-21T14:14:35.581254+0000","last_fullsized":"2024-03-21T14:14:35.581254+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:41.480136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.480136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:01.599929+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":778,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.622790+0000","last_change":"2024-03-21T14:14:35.622790+0000","last_active":"2024-03-21T14:14:35.622790+0000","last_peered":"2024-03-21T14:14:35.622790+0000","last_clean":"2024-03-21T14:14:35.622790+0000","last_became_active":"2024-03-21T14:14:35.621834+0000","last_became_peered":"2024-03-21T14:14:35.621834+0000","last_unstale":"2024-03-21T14:14:35.622790+0000","last_undegraded":"2024-03-21T14:14:35.622790+0000","last_fullsized":"2024-03-21T14:14:35.622790+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:11:09.977051+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:11:09.977051+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"274'7967","reported_seq":8037,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.547101+0000","last_change":"2024-03-21T14:13:08.814099+0000","last_active":"2024-03-21T14:14:36.547101+0000","last_peered":"2024-03-21T14:14:36.547101+0000","last_clean":"2024-03-21T14:14:36.547101+0000","last_became_active":"2024-03-21T14:13:08.813800+0000","last_became_peered":"2024-03-21T14:13:08.813800+0000","last_unstale":"2024-03-21T14:14:36.547101+0000","last_undegraded":"2024-03-21T14:14:36.547101+0000","last_fullsized":"2024-03-21T14:14:36.547101+0000","mapping_epoch":244,"log_start":"256'3588","ondisk_log_start":"256'3588","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4379,"log_dups_size":0,"ondisk_log_size":4379,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":522125312,"num_objects":7967,"num_object_clones":0,"num_object_copies":23901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7967,"num_write_kb":509888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":468,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578459+0000","last_change":"2024-03-21T14:13:37.711491+0000","last_active":"2024-03-21T14:14:35.578459+0000","last_peered":"2024-03-21T14:14:35.578459+0000","last_clean":"2024-03-21T14:14:35.578459+0000","last_became_active":"2024-03-21T14:12:00.544093+0000","last_became_peered":"2024-03-21T14:12:00.544093+0000","last_unstale":"2024-03-21T14:14:35.578459+0000","last_undegraded":"2024-03-21T14:14:35.578459+0000","last_fullsized":"2024-03-21T14:14:35.578459+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:37.711376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:37.711376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:57.687359+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":480,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578569+0000","last_change":"2024-03-21T14:12:58.667273+0000","last_active":"2024-03-21T14:14:35.578569+0000","last_peered":"2024-03-21T14:14:35.578569+0000","last_clean":"2024-03-21T14:14:35.578569+0000","last_became_active":"2024-03-21T14:12:00.535428+0000","last_became_peered":"2024-03-21T14:12:00.535428+0000","last_unstale":"2024-03-21T14:14:35.578569+0000","last_undegraded":"2024-03-21T14:14:35.578569+0000","last_fullsized":"2024-03-21T14:14:35.578569+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.667132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.667132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":507,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580619+0000","last_change":"2024-03-21T14:14:06.212889+0000","last_active":"2024-03-21T14:14:35.580619+0000","last_peered":"2024-03-21T14:14:35.580619+0000","last_clean":"2024-03-21T14:14:35.580619+0000","last_became_active":"2024-03-21T14:02:32.438415+0000","last_became_peered":"2024-03-21T14:02:32.438415+0000","last_unstale":"2024-03-21T14:14:35.580619+0000","last_undegraded":"2024-03-21T14:14:35.580619+0000","last_fullsized":"2024-03-21T14:14:35.580619+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.212763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.212763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:35.487106+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00046272300000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":541,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.582625+0000","last_change":"2024-03-21T14:13:59.418320+0000","last_active":"2024-03-21T14:14:35.582625+0000","last_peered":"2024-03-21T14:14:35.582625+0000","last_clean":"2024-03-21T14:14:35.582625+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:14:35.582625+0000","last_undegraded":"2024-03-21T14:14:35.582625+0000","last_fullsized":"2024-03-21T14:14:35.582625+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:13:59.418054+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.418054+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:09.213398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":490,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581060+0000","last_change":"2024-03-21T14:14:07.244767+0000","last_active":"2024-03-21T14:14:35.581060+0000","last_peered":"2024-03-21T14:14:35.581060+0000","last_clean":"2024-03-21T14:14:35.581060+0000","last_became_active":"2024-03-21T14:02:37.458955+0000","last_became_peered":"2024-03-21T14:02:37.458955+0000","last_unstale":"2024-03-21T14:14:35.581060+0000","last_undegraded":"2024-03-21T14:14:35.581060+0000","last_fullsized":"2024-03-21T14:14:35.581060+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:07.244640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.244640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:30.259905+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"274'7845","reported_seq":7915,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.559411+0000","last_change":"2024-03-21T14:13:09.117302+0000","last_active":"2024-03-21T14:14:36.559411+0000","last_peered":"2024-03-21T14:14:36.559411+0000","last_clean":"2024-03-21T14:14:36.559411+0000","last_became_active":"2024-03-21T14:13:09.116993+0000","last_became_peered":"2024-03-21T14:13:09.116993+0000","last_unstale":"2024-03-21T14:14:36.559411+0000","last_undegraded":"2024-03-21T14:14:36.559411+0000","last_fullsized":"2024-03-21T14:14:36.559411+0000","mapping_epoch":244,"log_start":"255'3472","ondisk_log_start":"255'3472","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514129920,"num_objects":7845,"num_object_clones":0,"num_object_copies":23535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7845,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7845,"num_write_kb":502080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":461,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.585950+0000","last_change":"2024-03-21T14:13:43.362540+0000","last_active":"2024-03-21T14:14:35.585950+0000","last_peered":"2024-03-21T14:14:35.585950+0000","last_clean":"2024-03-21T14:14:35.585950+0000","last_became_active":"2024-03-21T14:02:39.470546+0000","last_became_peered":"2024-03-21T14:02:39.470546+0000","last_unstale":"2024-03-21T14:14:35.585950+0000","last_undegraded":"2024-03-21T14:14:35.585950+0000","last_fullsized":"2024-03-21T14:14:35.585950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:43.362326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.362326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:09.710464+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":770,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.576115+0000","last_change":"2024-03-21T14:14:30.791274+0000","last_active":"2024-03-21T14:14:35.576115+0000","last_peered":"2024-03-21T14:14:35.576115+0000","last_clean":"2024-03-21T14:14:35.576115+0000","last_became_active":"2024-03-21T14:14:30.790875+0000","last_became_peered":"2024-03-21T14:14:30.790875+0000","last_unstale":"2024-03-21T14:14:35.576115+0000","last_undegraded":"2024-03-21T14:14:35.576115+0000","last_fullsized":"2024-03-21T14:14:35.576115+0000","mapping_epoch":213,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:11:09.977051+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:11:09.977051+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.653088+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":480,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578484+0000","last_change":"2024-03-21T14:12:53.182676+0000","last_active":"2024-03-21T14:14:35.578484+0000","last_peered":"2024-03-21T14:14:35.578484+0000","last_clean":"2024-03-21T14:14:35.578484+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:14:35.578484+0000","last_undegraded":"2024-03-21T14:14:35.578484+0000","last_fullsized":"2024-03-21T14:14:35.578484+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:53.182591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.182591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":250,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578544+0000","last_change":"2024-03-21T14:12:00.597903+0000","last_active":"2024-03-21T14:14:35.578544+0000","last_peered":"2024-03-21T14:14:35.578544+0000","last_clean":"2024-03-21T14:14:35.578544+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:14:35.578544+0000","last_undegraded":"2024-03-21T14:14:35.578544+0000","last_fullsized":"2024-03-21T14:14:35.578544+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'16","last_scrub_stamp":"2024-03-21T14:10:55.042070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.042070+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":466,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580532+0000","last_change":"2024-03-21T14:12:55.820482+0000","last_active":"2024-03-21T14:14:35.580532+0000","last_peered":"2024-03-21T14:14:35.580532+0000","last_clean":"2024-03-21T14:14:35.580532+0000","last_became_active":"2024-03-21T14:02:37.458719+0000","last_became_peered":"2024-03-21T14:02:37.458719+0000","last_unstale":"2024-03-21T14:14:35.580532+0000","last_undegraded":"2024-03-21T14:14:35.580532+0000","last_fullsized":"2024-03-21T14:14:35.580532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.820408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.820408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:35.630585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":543,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.606954+0000","last_change":"2024-03-21T14:14:35.607003+0000","last_active":"2024-03-21T14:14:35.606954+0000","last_peered":"2024-03-21T14:14:35.606954+0000","last_clean":"2024-03-21T14:14:35.606954+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:14:35.606954+0000","last_undegraded":"2024-03-21T14:14:35.606954+0000","last_fullsized":"2024-03-21T14:14:35.606954+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.605256+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.605256+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:16:02.551657+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"274'7803","reported_seq":7872,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.551963+0000","last_change":"2024-03-21T14:13:08.815820+0000","last_active":"2024-03-21T14:14:36.551963+0000","last_peered":"2024-03-21T14:14:36.551963+0000","last_clean":"2024-03-21T14:14:36.551963+0000","last_became_active":"2024-03-21T14:13:08.815075+0000","last_became_peered":"2024-03-21T14:13:08.815075+0000","last_unstale":"2024-03-21T14:14:36.551963+0000","last_undegraded":"2024-03-21T14:14:36.551963+0000","last_fullsized":"2024-03-21T14:14:36.551963+0000","mapping_epoch":244,"log_start":"255'3387","ondisk_log_start":"255'3387","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511377408,"num_objects":7803,"num_object_clones":0,"num_object_copies":23409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7803,"num_write_kb":499392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":473,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579672+0000","last_change":"2024-03-21T14:13:18.721382+0000","last_active":"2024-03-21T14:14:35.579672+0000","last_peered":"2024-03-21T14:14:35.579672+0000","last_clean":"2024-03-21T14:14:35.579672+0000","last_became_active":"2024-03-21T14:13:18.721013+0000","last_became_peered":"2024-03-21T14:13:18.721013+0000","last_unstale":"2024-03-21T14:14:35.579672+0000","last_undegraded":"2024-03-21T14:14:35.579672+0000","last_fullsized":"2024-03-21T14:14:35.579672+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:13:04.716819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599421+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":267,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577504+0000","last_change":"2024-03-21T14:12:20.560104+0000","last_active":"2024-03-21T14:14:35.577504+0000","last_peered":"2024-03-21T14:14:35.577504+0000","last_clean":"2024-03-21T14:14:35.577504+0000","last_became_active":"2024-03-21T14:12:20.559683+0000","last_became_peered":"2024-03-21T14:12:20.559683+0000","last_unstale":"2024-03-21T14:14:35.577504+0000","last_undegraded":"2024-03-21T14:14:35.577504+0000","last_fullsized":"2024-03-21T14:14:35.577504+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:11.081667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:11:11.081667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974974+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"73'5","reported_seq":1203,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.447347+0000","last_change":"2024-03-21T14:13:07.680778+0000","last_active":"2024-03-21T14:14:36.447347+0000","last_peered":"2024-03-21T14:14:36.447347+0000","last_clean":"2024-03-21T14:14:36.447347+0000","last_became_active":"2024-03-21T14:02:32.438337+0000","last_became_peered":"2024-03-21T14:02:32.438337+0000","last_unstale":"2024-03-21T14:14:36.447347+0000","last_undegraded":"2024-03-21T14:14:36.447347+0000","last_fullsized":"2024-03-21T14:14:36.447347+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-03-21T14:13:07.680672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.680672+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":716,"num_read_kb":711,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":395,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581084+0000","last_change":"2024-03-21T14:13:43.544913+0000","last_active":"2024-03-21T14:14:35.581084+0000","last_peered":"2024-03-21T14:14:35.581084+0000","last_clean":"2024-03-21T14:14:35.581084+0000","last_became_active":"2024-03-21T14:07:23.363490+0000","last_became_peered":"2024-03-21T14:07:23.363490+0000","last_unstale":"2024-03-21T14:14:35.581084+0000","last_undegraded":"2024-03-21T14:14:35.581084+0000","last_fullsized":"2024-03-21T14:14:35.581084+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:13:43.544671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.544671+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:51.992091+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":465,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586765+0000","last_change":"2024-03-21T14:13:39.314326+0000","last_active":"2024-03-21T14:14:35.586765+0000","last_peered":"2024-03-21T14:14:35.586765+0000","last_clean":"2024-03-21T14:14:35.586765+0000","last_became_active":"2024-03-21T14:02:37.455451+0000","last_became_peered":"2024-03-21T14:02:37.455451+0000","last_unstale":"2024-03-21T14:14:35.586765+0000","last_undegraded":"2024-03-21T14:14:35.586765+0000","last_fullsized":"2024-03-21T14:14:35.586765+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.313966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.313966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:51.859924+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":537,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.915010+0000","last_change":"2024-03-21T14:14:36.915033+0000","last_active":"2024-03-21T14:14:36.915010+0000","last_peered":"2024-03-21T14:14:36.915010+0000","last_clean":"2024-03-21T14:14:36.915010+0000","last_became_active":"2024-03-21T14:14:35.622383+0000","last_became_peered":"2024-03-21T14:14:35.622383+0000","last_unstale":"2024-03-21T14:14:36.915010+0000","last_undegraded":"2024-03-21T14:14:36.915010+0000","last_fullsized":"2024-03-21T14:14:36.915010+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:36.914941+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:36.914941+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:50.853315+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"274'7857","reported_seq":7927,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.496839+0000","last_change":"2024-03-21T14:13:09.392683+0000","last_active":"2024-03-21T14:14:36.496839+0000","last_peered":"2024-03-21T14:14:36.496839+0000","last_clean":"2024-03-21T14:14:36.496839+0000","last_became_active":"2024-03-21T14:13:09.392267+0000","last_became_peered":"2024-03-21T14:13:09.392267+0000","last_unstale":"2024-03-21T14:14:36.496839+0000","last_undegraded":"2024-03-21T14:14:36.496839+0000","last_fullsized":"2024-03-21T14:14:36.496839+0000","mapping_epoch":244,"log_start":"259'4025","ondisk_log_start":"259'4025","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3832,"log_dups_size":0,"ondisk_log_size":3832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514916352,"num_objects":7857,"num_object_clones":0,"num_object_copies":23571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7857,"num_write_kb":502848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":467,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578993+0000","last_change":"2024-03-21T14:12:59.668840+0000","last_active":"2024-03-21T14:14:35.578993+0000","last_peered":"2024-03-21T14:14:35.578993+0000","last_clean":"2024-03-21T14:14:35.578993+0000","last_became_active":"2024-03-21T14:12:00.533868+0000","last_became_peered":"2024-03-21T14:12:00.533868+0000","last_unstale":"2024-03-21T14:14:35.578993+0000","last_undegraded":"2024-03-21T14:14:35.578993+0000","last_fullsized":"2024-03-21T14:14:35.578993+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.663279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599322+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":481,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579788+0000","last_change":"2024-03-21T14:13:55.416345+0000","last_active":"2024-03-21T14:14:35.579788+0000","last_peered":"2024-03-21T14:14:35.579788+0000","last_clean":"2024-03-21T14:14:35.579788+0000","last_became_active":"2024-03-21T14:02:33.443973+0000","last_became_peered":"2024-03-21T14:02:33.443973+0000","last_unstale":"2024-03-21T14:14:35.579788+0000","last_undegraded":"2024-03-21T14:14:35.579788+0000","last_fullsized":"2024-03-21T14:14:35.579788+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:55.416068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.416068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:59.940103+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":189,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583332+0000","last_change":"2024-03-21T14:12:53.298944+0000","last_active":"2024-03-21T14:14:35.583332+0000","last_peered":"2024-03-21T14:14:35.583332+0000","last_clean":"2024-03-21T14:14:35.583332+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:14:35.583332+0000","last_undegraded":"2024-03-21T14:14:35.583332+0000","last_fullsized":"2024-03-21T14:14:35.583332+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:12:53.298845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.298845+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":121,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.588091+0000","last_change":"2024-03-21T14:12:55.222431+0000","last_active":"2024-03-21T14:14:35.588091+0000","last_peered":"2024-03-21T14:14:35.588091+0000","last_clean":"2024-03-21T14:14:35.588091+0000","last_became_active":"2024-03-21T14:12:20.561996+0000","last_became_peered":"2024-03-21T14:12:20.561996+0000","last_unstale":"2024-03-21T14:14:35.588091+0000","last_undegraded":"2024-03-21T14:14:35.588091+0000","last_fullsized":"2024-03-21T14:14:35.588091+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.222353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.222353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":778,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.599079+0000","last_change":"2024-03-21T14:14:35.599079+0000","last_active":"2024-03-21T14:14:35.599079+0000","last_peered":"2024-03-21T14:14:35.599079+0000","last_clean":"2024-03-21T14:14:35.599079+0000","last_became_active":"2024-03-21T14:14:35.597520+0000","last_became_peered":"2024-03-21T14:14:35.597520+0000","last_unstale":"2024-03-21T14:14:35.599079+0000","last_undegraded":"2024-03-21T14:14:35.599079+0000","last_fullsized":"2024-03-21T14:14:35.599079+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:11:09.977051+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:11:09.977051+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"274'7957","reported_seq":8027,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.551626+0000","last_change":"2024-03-21T14:13:08.815297+0000","last_active":"2024-03-21T14:14:36.551626+0000","last_peered":"2024-03-21T14:14:36.551626+0000","last_clean":"2024-03-21T14:14:36.551626+0000","last_became_active":"2024-03-21T14:13:08.815004+0000","last_became_peered":"2024-03-21T14:13:08.815004+0000","last_unstale":"2024-03-21T14:14:36.551626+0000","last_undegraded":"2024-03-21T14:14:36.551626+0000","last_fullsized":"2024-03-21T14:14:36.551626+0000","mapping_epoch":244,"log_start":"255'3585","ondisk_log_start":"255'3585","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":521469952,"num_objects":7957,"num_object_clones":0,"num_object_copies":23871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7957,"num_write_kb":509248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":303,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.571329+0000","last_change":"2024-03-21T14:13:42.065806+0000","last_active":"2024-03-21T14:14:35.571329+0000","last_peered":"2024-03-21T14:14:35.571329+0000","last_clean":"2024-03-21T14:14:35.571329+0000","last_became_active":"2024-03-21T14:12:20.560873+0000","last_became_peered":"2024-03-21T14:12:20.560873+0000","last_unstale":"2024-03-21T14:14:35.571329+0000","last_undegraded":"2024-03-21T14:14:35.571329+0000","last_fullsized":"2024-03-21T14:14:35.571329+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:42.065682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:42.065682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.243764+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":465,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586877+0000","last_change":"2024-03-21T14:12:45.295649+0000","last_active":"2024-03-21T14:14:35.586877+0000","last_peered":"2024-03-21T14:14:35.586877+0000","last_clean":"2024-03-21T14:14:35.586877+0000","last_became_active":"2024-03-21T14:02:33.442982+0000","last_became_peered":"2024-03-21T14:02:33.442982+0000","last_unstale":"2024-03-21T14:14:35.586877+0000","last_undegraded":"2024-03-21T14:14:35.586877+0000","last_fullsized":"2024-03-21T14:14:35.586877+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:45.295537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:45.295537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316538+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":512,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583677+0000","last_change":"2024-03-21T14:13:38.544949+0000","last_active":"2024-03-21T14:14:35.583677+0000","last_peered":"2024-03-21T14:14:35.583677+0000","last_clean":"2024-03-21T14:14:35.583677+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:14:35.583677+0000","last_undegraded":"2024-03-21T14:14:35.583677+0000","last_fullsized":"2024-03-21T14:14:35.583677+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:13:38.544738+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.544738+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:07.852333+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"226'10","reported_seq":171,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.583130+0000","last_change":"2024-03-21T14:13:47.212212+0000","last_active":"2024-03-21T14:14:35.583130+0000","last_peered":"2024-03-21T14:14:35.583130+0000","last_clean":"2024-03-21T14:14:35.583130+0000","last_became_active":"2024-03-21T14:12:20.561788+0000","last_became_peered":"2024-03-21T14:12:20.561788+0000","last_unstale":"2024-03-21T14:14:35.583130+0000","last_undegraded":"2024-03-21T14:14:35.583130+0000","last_fullsized":"2024-03-21T14:14:35.583130+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T14:13:47.212098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:47.212098+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.788156+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":536,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.751178+0000","last_change":"2024-03-21T14:14:35.751226+0000","last_active":"2024-03-21T14:14:35.751178+0000","last_peered":"2024-03-21T14:14:35.751178+0000","last_clean":"2024-03-21T14:14:35.751178+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:14:35.751178+0000","last_undegraded":"2024-03-21T14:14:35.751178+0000","last_fullsized":"2024-03-21T14:14:35.751178+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.751092+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.751092+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:58.009094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"274'7849","reported_seq":7919,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:37.074703+0000","last_change":"2024-03-21T14:13:09.392139+0000","last_active":"2024-03-21T14:14:37.074703+0000","last_peered":"2024-03-21T14:14:37.074703+0000","last_clean":"2024-03-21T14:14:37.074703+0000","last_became_active":"2024-03-21T14:13:09.391815+0000","last_became_peered":"2024-03-21T14:13:09.391815+0000","last_unstale":"2024-03-21T14:14:37.074703+0000","last_undegraded":"2024-03-21T14:14:37.074703+0000","last_fullsized":"2024-03-21T14:14:37.074703+0000","mapping_epoch":244,"log_start":"251'2188","ondisk_log_start":"251'2188","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514392064,"num_objects":7849,"num_object_clones":0,"num_object_copies":23547,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7849,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7849,"num_write_kb":502336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":468,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.581254+0000","last_change":"2024-03-21T14:13:51.612582+0000","last_active":"2024-03-21T14:14:35.581254+0000","last_peered":"2024-03-21T14:14:35.581254+0000","last_clean":"2024-03-21T14:14:35.581254+0000","last_became_active":"2024-03-21T14:12:00.530815+0000","last_became_peered":"2024-03-21T14:12:00.530815+0000","last_unstale":"2024-03-21T14:14:35.581254+0000","last_undegraded":"2024-03-21T14:14:35.581254+0000","last_fullsized":"2024-03-21T14:14:35.581254+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:51.612367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:02.083143+0000","last_clean_scrub_stamp":"2024-03-21T14:13:51.612367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:17.305704+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":331,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580857+0000","last_change":"2024-03-21T14:13:02.667615+0000","last_active":"2024-03-21T14:14:35.580857+0000","last_peered":"2024-03-21T14:14:35.580857+0000","last_clean":"2024-03-21T14:14:35.580857+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:14:35.580857+0000","last_undegraded":"2024-03-21T14:14:35.580857+0000","last_fullsized":"2024-03-21T14:14:35.580857+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:13:02.667511+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":503,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577512+0000","last_change":"2024-03-21T14:11:02.706392+0000","last_active":"2024-03-21T14:14:35.577512+0000","last_peered":"2024-03-21T14:14:35.577512+0000","last_clean":"2024-03-21T14:14:35.577512+0000","last_became_active":"2024-03-21T14:02:35.444171+0000","last_became_peered":"2024-03-21T14:02:35.444171+0000","last_unstale":"2024-03-21T14:14:35.577512+0000","last_undegraded":"2024-03-21T14:14:35.577512+0000","last_fullsized":"2024-03-21T14:14:35.577512+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'14","last_scrub_stamp":"2024-03-21T14:11:02.706261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:11:02.706261+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974945+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":619,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578048+0000","last_change":"2024-03-21T14:13:38.528704+0000","last_active":"2024-03-21T14:14:35.578048+0000","last_peered":"2024-03-21T14:14:35.578048+0000","last_clean":"2024-03-21T14:14:35.578048+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:14:35.578048+0000","last_undegraded":"2024-03-21T14:14:35.578048+0000","last_fullsized":"2024-03-21T14:14:35.578048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:13:38.528575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:38.528575+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:57.699075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":537,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.954925+0000","last_change":"2024-03-21T14:14:35.954925+0000","last_active":"2024-03-21T14:14:35.954925+0000","last_peered":"2024-03-21T14:14:35.954925+0000","last_clean":"2024-03-21T14:14:35.954925+0000","last_became_active":"2024-03-21T14:14:35.591277+0000","last_became_peered":"2024-03-21T14:14:35.591277+0000","last_unstale":"2024-03-21T14:14:35.954925+0000","last_undegraded":"2024-03-21T14:14:35.954925+0000","last_fullsized":"2024-03-21T14:14:35.954925+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.954782+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.954782+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:16:00.466247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"274'7744","reported_seq":7814,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.467229+0000","last_change":"2024-03-21T14:13:08.816071+0000","last_active":"2024-03-21T14:14:36.467229+0000","last_peered":"2024-03-21T14:14:36.467229+0000","last_clean":"2024-03-21T14:14:36.467229+0000","last_became_active":"2024-03-21T14:13:08.815381+0000","last_became_peered":"2024-03-21T14:13:08.815381+0000","last_unstale":"2024-03-21T14:14:36.467229+0000","last_undegraded":"2024-03-21T14:14:36.467229+0000","last_fullsized":"2024-03-21T14:14:36.467229+0000","mapping_epoch":244,"log_start":"258'3924","ondisk_log_start":"258'3924","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507510784,"num_objects":7744,"num_object_clones":0,"num_object_copies":23232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7744,"num_write_kb":495616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":471,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577761+0000","last_change":"2024-03-21T14:13:18.544218+0000","last_active":"2024-03-21T14:14:35.577761+0000","last_peered":"2024-03-21T14:14:35.577761+0000","last_clean":"2024-03-21T14:14:35.577761+0000","last_became_active":"2024-03-21T14:13:18.543885+0000","last_became_peered":"2024-03-21T14:13:18.543885+0000","last_unstale":"2024-03-21T14:14:35.577761+0000","last_undegraded":"2024-03-21T14:14:35.577761+0000","last_fullsized":"2024-03-21T14:14:35.577761+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:53.436026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.006246+0000","last_clean_scrub_stamp":"2024-03-21T14:11:53.436026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975000+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":474,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579376+0000","last_change":"2024-03-21T14:12:49.278950+0000","last_active":"2024-03-21T14:14:35.579376+0000","last_peered":"2024-03-21T14:14:35.579376+0000","last_clean":"2024-03-21T14:14:35.579376+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:14:35.579376+0000","last_undegraded":"2024-03-21T14:14:35.579376+0000","last_fullsized":"2024-03-21T14:14:35.579376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:49.278077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:12:49.278077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.700171+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":454,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.576283+0000","last_change":"2024-03-21T14:13:56.515249+0000","last_active":"2024-03-21T14:14:35.576283+0000","last_peered":"2024-03-21T14:14:35.576283+0000","last_clean":"2024-03-21T14:14:35.576283+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:14:35.576283+0000","last_undegraded":"2024-03-21T14:14:35.576283+0000","last_fullsized":"2024-03-21T14:14:35.576283+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:13:56.515093+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:21.854953+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":449,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577349+0000","last_change":"2024-03-21T14:11:01.663320+0000","last_active":"2024-03-21T14:14:35.577349+0000","last_peered":"2024-03-21T14:14:35.577349+0000","last_clean":"2024-03-21T14:14:35.577349+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:14:35.577349+0000","last_undegraded":"2024-03-21T14:14:35.577349+0000","last_fullsized":"2024-03-21T14:14:35.577349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:01.663200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:11:01.663200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974854+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":529,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.595631+0000","last_change":"2024-03-21T14:14:35.592092+0000","last_active":"2024-03-21T14:14:35.595631+0000","last_peered":"2024-03-21T14:14:35.595631+0000","last_clean":"2024-03-21T14:14:35.595631+0000","last_became_active":"2024-03-21T14:14:35.591595+0000","last_became_peered":"2024-03-21T14:14:35.591595+0000","last_unstale":"2024-03-21T14:14:35.595631+0000","last_undegraded":"2024-03-21T14:14:35.595631+0000","last_fullsized":"2024-03-21T14:14:35.595631+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:11:09.977051+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:11:09.977051+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"274'7948","reported_seq":8018,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:37.224734+0000","last_change":"2024-03-21T14:13:08.814595+0000","last_active":"2024-03-21T14:14:37.224734+0000","last_peered":"2024-03-21T14:14:37.224734+0000","last_clean":"2024-03-21T14:14:37.224734+0000","last_became_active":"2024-03-21T14:13:08.813908+0000","last_became_peered":"2024-03-21T14:13:08.813908+0000","last_unstale":"2024-03-21T14:14:37.224734+0000","last_undegraded":"2024-03-21T14:14:37.224734+0000","last_fullsized":"2024-03-21T14:14:37.224734+0000","mapping_epoch":244,"log_start":"262'4499","ondisk_log_start":"262'4499","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3449,"log_dups_size":0,"ondisk_log_size":3449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520880128,"num_objects":7948,"num_object_clones":0,"num_object_copies":23844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7948,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7948,"num_write_kb":508672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":473,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586580+0000","last_change":"2024-03-21T14:12:59.226585+0000","last_active":"2024-03-21T14:14:35.586580+0000","last_peered":"2024-03-21T14:14:35.586580+0000","last_clean":"2024-03-21T14:14:35.586580+0000","last_became_active":"2024-03-21T14:08:17.203307+0000","last_became_peered":"2024-03-21T14:08:17.203307+0000","last_unstale":"2024-03-21T14:14:35.586580+0000","last_undegraded":"2024-03-21T14:14:35.586580+0000","last_fullsized":"2024-03-21T14:14:35.586580+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.226457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.226457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.316323+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":611,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579548+0000","last_change":"2024-03-21T14:10:55.804376+0000","last_active":"2024-03-21T14:14:35.579548+0000","last_peered":"2024-03-21T14:14:35.579548+0000","last_clean":"2024-03-21T14:14:35.579548+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:14:35.579548+0000","last_undegraded":"2024-03-21T14:14:35.579548+0000","last_fullsized":"2024-03-21T14:14:35.579548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:10:55.804112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:10:55.804112+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.974839+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":319,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578827+0000","last_change":"2024-03-21T14:12:47.812244+0000","last_active":"2024-03-21T14:14:35.578827+0000","last_peered":"2024-03-21T14:14:35.578827+0000","last_clean":"2024-03-21T14:14:35.578827+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:14:35.578827+0000","last_undegraded":"2024-03-21T14:14:35.578827+0000","last_fullsized":"2024-03-21T14:14:35.578827+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:47.812131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:47.812131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599355+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":512,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.582706+0000","last_change":"2024-03-21T14:13:03.665475+0000","last_active":"2024-03-21T14:14:35.582706+0000","last_peered":"2024-03-21T14:14:35.582706+0000","last_clean":"2024-03-21T14:14:35.582706+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:14:35.582706+0000","last_undegraded":"2024-03-21T14:14:35.582706+0000","last_fullsized":"2024-03-21T14:14:35.582706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:13:03.665389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.665389+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:36.599295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"274'7680","reported_seq":7750,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.467842+0000","last_change":"2024-03-21T14:13:08.815949+0000","last_active":"2024-03-21T14:14:36.467842+0000","last_peered":"2024-03-21T14:14:36.467842+0000","last_clean":"2024-03-21T14:14:36.467842+0000","last_became_active":"2024-03-21T14:13:08.815576+0000","last_became_peered":"2024-03-21T14:13:08.815576+0000","last_unstale":"2024-03-21T14:14:36.467842+0000","last_undegraded":"2024-03-21T14:14:36.467842+0000","last_fullsized":"2024-03-21T14:14:36.467842+0000","mapping_epoch":244,"log_start":"258'3851","ondisk_log_start":"258'3851","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3829,"log_dups_size":0,"ondisk_log_size":3829,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":503316480,"num_objects":7680,"num_object_clones":0,"num_object_copies":23040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7680,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7680,"num_write_kb":491520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":468,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580447+0000","last_change":"2024-03-21T14:14:00.615422+0000","last_active":"2024-03-21T14:14:35.580447+0000","last_peered":"2024-03-21T14:14:35.580447+0000","last_clean":"2024-03-21T14:14:35.580447+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:14:35.580447+0000","last_undegraded":"2024-03-21T14:14:35.580447+0000","last_fullsized":"2024-03-21T14:14:35.580447+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.614983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.614983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:09.249329+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":465,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.586668+0000","last_change":"2024-03-21T14:13:50.211771+0000","last_active":"2024-03-21T14:14:35.586668+0000","last_peered":"2024-03-21T14:14:35.586668+0000","last_clean":"2024-03-21T14:14:35.586668+0000","last_became_active":"2024-03-21T14:02:37.453740+0000","last_became_peered":"2024-03-21T14:02:37.453740+0000","last_unstale":"2024-03-21T14:14:35.586668+0000","last_undegraded":"2024-03-21T14:14:35.586668+0000","last_fullsized":"2024-03-21T14:14:35.586668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:50.211665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.211665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:15.947203+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":473,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577386+0000","last_change":"2024-03-21T14:13:45.603203+0000","last_active":"2024-03-21T14:14:35.577386+0000","last_peered":"2024-03-21T14:14:35.577386+0000","last_clean":"2024-03-21T14:14:35.577386+0000","last_became_active":"2024-03-21T14:02:33.445372+0000","last_became_peered":"2024-03-21T14:02:33.445372+0000","last_unstale":"2024-03-21T14:14:35.577386+0000","last_undegraded":"2024-03-21T14:14:35.577386+0000","last_fullsized":"2024-03-21T14:14:35.577386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:45.602994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:45.602994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:00.370011+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":510,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578616+0000","last_change":"2024-03-21T14:12:11.483612+0000","last_active":"2024-03-21T14:14:35.578616+0000","last_peered":"2024-03-21T14:14:35.578616+0000","last_clean":"2024-03-21T14:14:35.578616+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:14:35.578616+0000","last_undegraded":"2024-03-21T14:14:35.578616+0000","last_fullsized":"2024-03-21T14:14:35.578616+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'11","last_scrub_stamp":"2024-03-21T14:12:11.483458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:11.483458+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":537,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:36.245001+0000","last_change":"2024-03-21T14:14:36.245001+0000","last_active":"2024-03-21T14:14:36.245001+0000","last_peered":"2024-03-21T14:14:36.245001+0000","last_clean":"2024-03-21T14:14:36.245001+0000","last_became_active":"2024-03-21T14:14:35.589477+0000","last_became_peered":"2024-03-21T14:14:35.589477+0000","last_unstale":"2024-03-21T14:14:36.245001+0000","last_undegraded":"2024-03-21T14:14:36.245001+0000","last_fullsized":"2024-03-21T14:14:36.245001+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:36.244697+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:36.244697+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:16:05.471959+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"274'7901","reported_seq":7970,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:37.224618+0000","last_change":"2024-03-21T14:13:09.117596+0000","last_active":"2024-03-21T14:14:37.224618+0000","last_peered":"2024-03-21T14:14:37.224618+0000","last_clean":"2024-03-21T14:14:37.224618+0000","last_became_active":"2024-03-21T14:13:09.117255+0000","last_became_peered":"2024-03-21T14:13:09.117255+0000","last_unstale":"2024-03-21T14:14:37.224618+0000","last_undegraded":"2024-03-21T14:14:37.224618+0000","last_fullsized":"2024-03-21T14:14:37.224618+0000","mapping_epoch":244,"log_start":"261'4461","ondisk_log_start":"261'4461","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":3440,"log_dups_size":0,"ondisk_log_size":3440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":517799936,"num_objects":7901,"num_object_clones":0,"num_object_copies":23703,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7901,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7901,"num_write_kb":505664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":479,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580689+0000","last_change":"2024-03-21T14:13:36.335019+0000","last_active":"2024-03-21T14:14:35.580689+0000","last_peered":"2024-03-21T14:14:35.580689+0000","last_clean":"2024-03-21T14:14:35.580689+0000","last_became_active":"2024-03-21T14:02:39.468252+0000","last_became_peered":"2024-03-21T14:02:39.468252+0000","last_unstale":"2024-03-21T14:14:35.580689+0000","last_undegraded":"2024-03-21T14:14:35.580689+0000","last_fullsized":"2024-03-21T14:14:35.580689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:13:36.334881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.334881+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:56.420200+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":473,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579875+0000","last_change":"2024-03-21T14:13:54.426318+0000","last_active":"2024-03-21T14:14:35.579875+0000","last_peered":"2024-03-21T14:14:35.579875+0000","last_clean":"2024-03-21T14:14:35.579875+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:14:35.579875+0000","last_undegraded":"2024-03-21T14:14:35.579875+0000","last_fullsized":"2024-03-21T14:14:35.579875+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_clean_scrub_stamp":"2024-03-21T14:13:54.426178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:01.191151+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":139,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577850+0000","last_change":"2024-03-21T14:12:00.531054+0000","last_active":"2024-03-21T14:14:35.577850+0000","last_peered":"2024-03-21T14:14:35.577850+0000","last_clean":"2024-03-21T14:14:35.577850+0000","last_became_active":"2024-03-21T14:12:00.530704+0000","last_became_peered":"2024-03-21T14:12:00.530704+0000","last_unstale":"2024-03-21T14:14:35.577850+0000","last_undegraded":"2024-03-21T14:14:35.577850+0000","last_fullsized":"2024-03-21T14:14:35.577850+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:11:52.959604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:11:52.959604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":510,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578651+0000","last_change":"2024-03-21T14:12:53.529498+0000","last_active":"2024-03-21T14:14:35.578651+0000","last_peered":"2024-03-21T14:14:35.578651+0000","last_clean":"2024-03-21T14:14:35.578651+0000","last_became_active":"2024-03-21T14:02:35.453474+0000","last_became_peered":"2024-03-21T14:02:35.453474+0000","last_unstale":"2024-03-21T14:14:35.578651+0000","last_undegraded":"2024-03-21T14:14:35.578651+0000","last_fullsized":"2024-03-21T14:14:35.578651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:12:53.529362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.529362+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975092+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":786,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.697088+0000","last_change":"2024-03-21T14:14:35.697127+0000","last_active":"2024-03-21T14:14:35.697088+0000","last_peered":"2024-03-21T14:14:35.697088+0000","last_clean":"2024-03-21T14:14:35.697088+0000","last_became_active":"2024-03-21T14:14:35.587482+0000","last_became_peered":"2024-03-21T14:14:35.587482+0000","last_unstale":"2024-03-21T14:14:35.697088+0000","last_undegraded":"2024-03-21T14:14:35.697088+0000","last_fullsized":"2024-03-21T14:14:35.697088+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.696891+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.696891+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:52.254131+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"274'7820","reported_seq":7890,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:37.052905+0000","last_change":"2024-03-21T14:13:09.391951+0000","last_active":"2024-03-21T14:14:37.052905+0000","last_peered":"2024-03-21T14:14:37.052905+0000","last_clean":"2024-03-21T14:14:37.052905+0000","last_became_active":"2024-03-21T14:13:09.391580+0000","last_became_peered":"2024-03-21T14:13:09.391580+0000","last_unstale":"2024-03-21T14:14:37.052905+0000","last_undegraded":"2024-03-21T14:14:37.052905+0000","last_fullsized":"2024-03-21T14:14:37.052905+0000","mapping_epoch":244,"log_start":"251'2071","ondisk_log_start":"251'2071","created":244,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:07.795714+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.795714+0000","objects_scrubbed":0,"log_size":5749,"log_dups_size":0,"ondisk_log_size":5749,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:37.701373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512491520,"num_objects":7820,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7820,"num_write_kb":500480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":459,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.579726+0000","last_change":"2024-03-21T14:13:18.721177+0000","last_active":"2024-03-21T14:14:35.579726+0000","last_peered":"2024-03-21T14:14:35.579726+0000","last_clean":"2024-03-21T14:14:35.579726+0000","last_became_active":"2024-03-21T14:13:18.720794+0000","last_became_peered":"2024-03-21T14:13:18.720794+0000","last_unstale":"2024-03-21T14:14:35.579726+0000","last_undegraded":"2024-03-21T14:14:35.579726+0000","last_fullsized":"2024-03-21T14:14:35.579726+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:50.567559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:50.567559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:38.975164+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"104'9","reported_seq":636,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.580992+0000","last_change":"2024-03-21T14:14:05.166407+0000","last_active":"2024-03-21T14:14:35.580992+0000","last_peered":"2024-03-21T14:14:35.580992+0000","last_clean":"2024-03-21T14:14:35.580992+0000","last_became_active":"2024-03-21T14:02:37.459805+0000","last_became_peered":"2024-03-21T14:02:37.459805+0000","last_unstale":"2024-03-21T14:14:35.580992+0000","last_undegraded":"2024-03-21T14:14:35.580992+0000","last_fullsized":"2024-03-21T14:14:35.580992+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:14:05.166173+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:14:05.166173+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:15:18.265678+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":480,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.578681+0000","last_change":"2024-03-21T14:13:50.440212+0000","last_active":"2024-03-21T14:14:35.578681+0000","last_peered":"2024-03-21T14:14:35.578681+0000","last_clean":"2024-03-21T14:14:35.578681+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:14:35.578681+0000","last_undegraded":"2024-03-21T14:14:35.578681+0000","last_fullsized":"2024-03-21T14:14:35.578681+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:50.439847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.439847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:53.131705+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":514,"reported_epoch":274,"state":"active+clean","last_fresh":"2024-03-21T14:14:35.577987+0000","last_change":"2024-03-21T14:13:35.642420+0000","last_active":"2024-03-21T14:14:35.577987+0000","last_peered":"2024-03-21T14:14:35.577987+0000","last_clean":"2024-03-21T14:14:35.577987+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:14:35.577987+0000","last_undegraded":"2024-03-21T14:14:35.577987+0000","last_fullsized":"2024-03-21T14:14:35.577987+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:13:35.642272+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:13:35.642272+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:14:47.161685+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8217100288,"num_objects":125383,"num_object_clones":0,"num_object_copies":376149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":125383,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125383,"num_write_kb":8024512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24788140032,"data_stored":24788140032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67687,"ondisk_log_size":67687,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":716,"num_read_kb":711,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":858716,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":97,"num_read_kb":138,"num_write":82,"num_write_kb":1098,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1234406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":2602,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103317,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5934080,"kb_used_data":5451696,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":87811072,"statfs":{"total":95995035648,"available":89918537728,"internally_reserved":0,"allocated":5582536704,"data_stored":5582141137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":493943242},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430363,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333172,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":332414,"kb_avail":93411980,"statfs":{"total":95995035648,"available":95653867520,"internally_reserved":0,"allocated":757760,"data_stored":436735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":340392390},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5524680,"kb_used_data":5010368,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":88220472,"statfs":{"total":95995035648,"available":90337763328,"internally_reserved":0,"allocated":5130616832,"data_stored":5130205188,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738554,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2401576,"kb_used_data":2009440,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91343576,"statfs":{"total":95995035648,"available":93535821824,"internally_reserved":0,"allocated":2057666560,"data_stored":2057285977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411503,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2936352,"kb_used_data":2503960,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":90808800,"statfs":{"total":95995035648,"available":92988211200,"internally_reserved":0,"allocated":2564055040,"data_stored":2563682712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182569,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2915260,"kb_used_data":2500844,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":90829892,"statfs":{"total":95995035648,"available":93009809408,"internally_reserved":0,"allocated":2560864256,"data_stored":2560472000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":424278471},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019046,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2488696,"kb_used_data":2096496,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91256456,"statfs":{"total":95995035648,"available":93446610944,"internally_reserved":0,"allocated":2146811904,"data_stored":2146415162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692000,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4967548,"kb_used_data":4499060,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":88777604,"statfs":{"total":95995035648,"available":90908266496,"internally_reserved":0,"allocated":4607037440,"data_stored":4606638017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2062811136,"data_stored":2062811136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5167316992,"data_stored":5167316992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2576809984,"data_stored":2576809984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2567897088,"data_stored":2567897088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2156396544,"data_stored":2156396544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4623106048,"data_stored":4623106048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":5633802240,"data_stored":5633802240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:14:43.683 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:43.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:43.824 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:43.925 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:43.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:43 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:14:43.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:43 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:14:43.963 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:43.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:43 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:14:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:43 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:14:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:43 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:14:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:43 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:14:44.049 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:44.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:44.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:44.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:44.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:44.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 135818 135802 98.6712 89.1875 0.0201228 0.0101303 2024-03-21T14:14:44.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:44.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:44.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:44.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:44.542 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:44.645 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:44.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:44.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:44.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:44.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:44.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: pgmap v544: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 97 MiB/s wr, 1.53k op/s 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:14:44.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:44 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:14:44.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:44.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:44.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: pgmap v544: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 97 MiB/s wr, 1.53k op/s 2024-03-21T14:14:44.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:14:44.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:14:44.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: pgmap v544: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 97 MiB/s wr, 1.53k op/s 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:14:44.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:14:44.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:44 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:14:45.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:45.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:45.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:45.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:45.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 137232 137216 98.5528 88.375 0.00855044 0.0101427 2024-03-21T14:14:45.316 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:45.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:45.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:45.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:45.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:45.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:45.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:14:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:14:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:14:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:14:45.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[29576]: 9.8 scrub starts 2024-03-21T14:14:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:14:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:14:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:14:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:14:45.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:45 smithi003 ceph-mon[40510]: 9.8 scrub starts 2024-03-21T14:14:45.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:45.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:45.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:45.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:45 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:14:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:45 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:14:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:45 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:14:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:45 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:14:45.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:45 smithi118 ceph-mon[33862]: 9.8 scrub starts 2024-03-21T14:14:45.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:45.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:45.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:46.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:46.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:46.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:46.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:46.242 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:46.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 15 138492 138477 98.3284 78.8125 0.00432488 0.0101656 2024-03-21T14:14:46.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:46.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:46.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:46.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:46.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:46.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:46.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:46.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:46.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[29576]: pgmap v545: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 86 MiB/s wr, 1.36k op/s 2024-03-21T14:14:46.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[29576]: 9.4 scrub starts 2024-03-21T14:14:46.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:14:46.705 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:14:46.706 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:14:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:14:46.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[40510]: pgmap v545: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 86 MiB/s wr, 1.36k op/s 2024-03-21T14:14:46.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[40510]: 9.4 scrub starts 2024-03-21T14:14:46.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:14:46.706 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:46 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:14:46.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:46.807 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:46.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:46 smithi118 ceph-mon[33862]: pgmap v545: 158 pgs: 158 active+clean; 8.1 GiB data, 27 GiB used, 688 GiB / 715 GiB avail; 86 MiB/s wr, 1.36k op/s 2024-03-21T14:14:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:46 smithi118 ceph-mon[33862]: 9.4 scrub starts 2024-03-21T14:14:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:46 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:14:46.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:46 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:14:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:46.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:46.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:47.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:47.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:47.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.233 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:47.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:47.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 139564 139548 97.9757 66.9375 0.010338 0.0102021 2024-03-21T14:14:47.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:47.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:47.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:14:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:14:47.471 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:47.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.507 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:47.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:47.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:47.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:47.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:47.678 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:14:47.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:47.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:47.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:47.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:47.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:47 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:47 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4013864694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:47 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:14:47.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:47 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:14:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4013864694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:14:48.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:14:48.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:14:48.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:48.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4013864694' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:14:48.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:14:48.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:47 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:14:48.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:14:48.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:48.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:14:48.169 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:48.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:48.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:48.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 90 7 140731 140724 97.7037 73.5 0.00862351 0.0102288 2024-03-21T14:14:48.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:48.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.333 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:48.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:48.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:48.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:48.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:48.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:48.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:48.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:48.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:14:48.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:14:48.671 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:14:48.672 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T14:14:48.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2 2024-03-21T14:14:48.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:48.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.715 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:48.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:48.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:48.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.842 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:48.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:14:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:14:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: pgmap v546: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-21T14:14:48.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 9.8 scrub ok 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: Cluster is now healthy 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4013864694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: osdmap e275: 8 total, 8 up, 8 in 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 4.10 deep-scrub starts 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 4.10 deep-scrub ok 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:14:48.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:48 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:14:48.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:48.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:48.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:49.004 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:48 smithi003 systemd[1]: Stopping Ceph osd.2 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:14:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:14:49.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: pgmap v546: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 9.8 scrub ok 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: Cluster is now healthy 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4013864694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: osdmap e275: 8 total, 8 up, 8 in 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 4.10 deep-scrub starts 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 4.10 deep-scrub ok 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:14:49.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: pgmap v546: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 107 MiB/s wr, 1.71k op/s 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 9.8 scrub ok 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: Cluster is now healthy 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4013864694' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: osdmap e275: 8 total, 8 up, 8 in 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 4.10 deep-scrub starts 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 4.10 deep-scrub ok 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:14:49.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:48 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:14:49.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:49.096 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:49.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:49.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:49.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:49.254 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.059 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 140731 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 97.6659 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 28.5903 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 202.375 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 62.3125 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1562 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 457.445 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3238 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 997 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0102315 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.00660789 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.152179 2024-03-21T14:14:49.255 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00113893 2024-03-21T14:14:49.258 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T14:14:49.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:49.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:49.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:49.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:49.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:49.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:49.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:49.629 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:49.731 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:49.767 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:49.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:49.869 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:49.904 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:49.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:49 smithi118 ceph-mon[33862]: 9.4 scrub ok 2024-03-21T14:14:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:49 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:14:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:49 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:14:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:49 smithi118 ceph-mon[33862]: 9.7 scrub starts 2024-03-21T14:14:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[29576]: 9.4 scrub ok 2024-03-21T14:14:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:14:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:14:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[29576]: 9.7 scrub starts 2024-03-21T14:14:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[40510]: 9.4 scrub ok 2024-03-21T14:14:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:14:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:14:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:49 smithi003 ceph-mon[40510]: 9.7 scrub starts 2024-03-21T14:14:50.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:50.042 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:50.144 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:50.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:50.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:50.283 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.307 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:50.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:50.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:50.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:50.547 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:50.583 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:50.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:50.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:50.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:50.861 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:50.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:50 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:14:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:50 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:14:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:50 smithi118 ceph-mon[33862]: pgmap v548: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-03-21T14:14:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:50 smithi118 ceph-mon[33862]: osdmap e276: 8 total, 8 up, 8 in 2024-03-21T14:14:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:50 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:14:50.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:50 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:14:50.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[29576]: pgmap v548: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[29576]: osdmap e276: 8 total, 8 up, 8 in 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:14:50.996 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:14:50.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[40510]: pgmap v548: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 85 MiB/s wr, 1.35k op/s 2024-03-21T14:14:50.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[40510]: osdmap e276: 8 total, 8 up, 8 in 2024-03-21T14:14:50.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:14:50.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:50 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:14:51.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:51.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:51.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:51.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:51.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:51.289 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:51.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:51.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:51.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:51.524 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:51.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:51.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:51.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:51.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:51.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:51.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.848 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:51.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:51.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:51.976 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:51.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:51.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:14:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:14:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:14:52.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:14:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:14:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:14:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:14:52.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:51 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:14:52.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:52.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:51 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:14:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:51 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:14:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:51 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:14:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:51 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:14:52.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:52.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:52.332 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:14:52.352 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:52.379 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:14:52.384 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:52.422 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:14:52.465 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:14:52.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:52.505 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:14:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.520 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:52.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:52.545 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:14:52.584 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:14:52.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:52.633 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:14:52.642 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:14:52.642 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:14:52.642 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:14:52.643 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:14:52.643 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:14:52.643 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:14:52.643 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:14:52.643 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:14:52.658 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:52.760 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:52.795 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:52.894 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:52 smithi003 ceph-mon[29576]: pgmap v550: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 838 op/s 2024-03-21T14:14:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:52 smithi003 ceph-mon[29576]: 9.7 scrub ok 2024-03-21T14:14:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:52 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/314680105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:52.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:52 smithi003 ceph-mon[40510]: pgmap v550: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 838 op/s 2024-03-21T14:14:52.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:52 smithi003 ceph-mon[40510]: 9.7 scrub ok 2024-03-21T14:14:52.895 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:52 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/314680105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:52.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:52.949 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:52.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:53.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:52 smithi118 ceph-mon[33862]: pgmap v550: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.5 GiB data, 29 GiB used, 686 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 838 op/s 2024-03-21T14:14:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:52 smithi118 ceph-mon[33862]: 9.7 scrub ok 2024-03-21T14:14:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:52 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/314680105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:14:53.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:53.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:53.339 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:53.372 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:53.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:53.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:53.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:53.592 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:14:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:14:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:14:53.593 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:14:53.593 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:14:53.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:53.639 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:53.741 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:14:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:14:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:14:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:14:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: osdmap e277: 8 total, 8 up, 8 in 2024-03-21T14:14:53.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: osd.2 marked itself down and dead 2024-03-21T14:14:53.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:14:53.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:14:53.755 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[56005]: 2024-03-21T14:14:53.587+0000 7f02c3525700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:14:53.755 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[56005]: 2024-03-21T14:14:53.587+0000 7f02c3525700 -1 osd.2 277 *** Got signal Terminated *** 2024-03-21T14:14:53.755 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[56005]: 2024-03-21T14:14:53.587+0000 7f02c3525700 -1 osd.2 277 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:14:53.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:14:53.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:14:53.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:14:53.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:14:53.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: osdmap e277: 8 total, 8 up, 8 in 2024-03-21T14:14:53.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: osd.2 marked itself down and dead 2024-03-21T14:14:53.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:14:53.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:53 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:14:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:53.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:53.867 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:53.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:53.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:53.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:54.017 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:54.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:54.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:54.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:14:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:14:54.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:14:54.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:14:54.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: osdmap e277: 8 total, 8 up, 8 in 2024-03-21T14:14:54.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: osd.2 marked itself down and dead 2024-03-21T14:14:54.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:14:54.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:53 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:14:54.244 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:54.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:54.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:54.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:54.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:54.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:54.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:54.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:54.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:54.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.785 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:54.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:54.853 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:54 smithi003 podman[145817]: 2024-03-21 14:14:54.521645744 +0000 UTC m=+1.547039644 container died c03bae57e535f3983cd80ac3c519cdbbc704a4148c3e8e9c436dcb214cca36cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-21T14:14:54.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: pgmap v552: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 807 op/s 2024-03-21T14:14:54.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T14:14:54.853 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: osdmap e278: 8 total, 7 up, 8 in 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: pgmap v552: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 807 op/s 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: osdmap e278: 8 total, 7 up, 8 in 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:14:54.854 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:54 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:14:54.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:54.919 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:54.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:54.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:54.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:14:55.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:55.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:55.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:55.117 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:54 smithi003 podman[145817]: 2024-03-21 14:14:54.950991697 +0000 UTC m=+1.976385596 container cleanup c03bae57e535f3983cd80ac3c519cdbbc704a4148c3e8e9c436dcb214cca36cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-21T14:14:55.117 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:54 smithi003 bash[145817]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2 2024-03-21T14:14:55.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: pgmap v552: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 807 op/s 2024-03-21T14:14:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-21T14:14:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: osdmap e278: 8 total, 7 up, 8 in 2024-03-21T14:14:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:14:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:14:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:14:55.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:54 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:14:55.199 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:55.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:55.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:55.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:55.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.480 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:55.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:55.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T14:14:55.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:55.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:55.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[29576]: osdmap e279: 8 total, 7 up, 8 in 2024-03-21T14:14:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:14:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:14:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:14:55.755 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:14:55 smithi003 podman[145920]: 2024-03-21 14:14:55.724031906 +0000 UTC m=+1.379902179 container remove c03bae57e535f3983cd80ac3c519cdbbc704a4148c3e8e9c436dcb214cca36cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T14:14:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[40510]: osdmap e279: 8 total, 7 up, 8 in 2024-03-21T14:14:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:14:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:14:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:55 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:55.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:55.925 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:55.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:55.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:55.956 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:56.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:56.063 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:56.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:55 smithi118 ceph-mon[33862]: osdmap e279: 8 total, 7 up, 8 in 2024-03-21T14:14:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:55 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:14:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:55 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:14:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:55 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:14:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:56.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:56.300 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:56.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:14:56.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:56.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:56.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:56.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:56.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:56.616 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:56.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:14:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:14:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:14:56.718 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:56.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:56.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:56.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:56.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:56.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:56 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:14:57.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:56 smithi003 ceph-mon[29576]: pgmap v555: 158 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 147 active+clean; 8.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 18 MiB/s wr, 834 op/s 2024-03-21T14:14:57.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:56 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:14:57.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:56 smithi003 ceph-mon[40510]: pgmap v555: 158 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 147 active+clean; 8.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 18 MiB/s wr, 834 op/s 2024-03-21T14:14:57.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:57.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:57.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.176 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:57.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:56 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:14:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:56 smithi118 ceph-mon[33862]: pgmap v555: 158 pgs: 10 stale+active+clean, 1 active+clean+scrubbing, 147 active+clean; 8.4 GiB data, 29 GiB used, 687 GiB / 715 GiB avail; 1.0 KiB/s rd, 18 MiB/s wr, 834 op/s 2024-03-21T14:14:57.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:14:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:14:57.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:57.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:57.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:57.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:57.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:57.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:57.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:57.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:57.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:57.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:57.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:57.867 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:57.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:57.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:58.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:58.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:58.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:58.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[29576]: 9.1 scrub starts 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[29576]: 9.3 scrub starts 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[40510]: 9.1 scrub starts 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[40510]: 9.3 scrub starts 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:14:58.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:57 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:14:58.139 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:57 smithi118 ceph-mon[33862]: 9.1 scrub starts 2024-03-21T14:14:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:57 smithi118 ceph-mon[33862]: 9.3 scrub starts 2024-03-21T14:14:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:14:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:57 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:14:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:57 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:14:58.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:58.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:58.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:58.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:58.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:58.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:14:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:58.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:14:58.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:58.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:58.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:14:58.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:14:58.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:14:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:58.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:58.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:14:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:14:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: pgmap v556: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 1.77k op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: Health check failed: Degraded data redundancy: 41098/396015 objects degraded (10.378%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:14:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: pgmap v556: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 1.77k op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: Health check failed: Degraded data redundancy: 41098/396015 objects degraded (10.378%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:14:59.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:58 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:14:59.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:59.084 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:14:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:59.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:14:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:14:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: pgmap v556: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 1.5 KiB/s rd, 18 MiB/s wr, 1.77k op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:14:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: Health check failed: Degraded data redundancy: 41098/396015 objects degraded (10.378%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:14:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:14:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:14:59.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:14:59.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:14:59.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:14:59.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:58 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:14:59.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:14:59.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:14:59.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.407 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:59.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:14:59.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:14:59.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:14:59.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:59.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:59.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:14:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.825 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:14:59.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:14:59.877 INFO:teuthology.orchestra.run.smithi003.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-21T14:14:59.877 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:14:59.877 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:14:59.877 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:14:59.877 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:14:59.877 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: osdmap e280: 8 total, 7 up, 8 in 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 9.3 scrub ok 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: osdmap e281: 8 total, 7 up, 8 in 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:14:59.878 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: osdmap e280: 8 total, 7 up, 8 in 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 9.3 scrub ok 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: osdmap e281: 8 total, 7 up, 8 in 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:14:59.879 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:14:59 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:14:59.927 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:14:59.959 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:14:59.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:14:59.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:00.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:00.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:15:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:15:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:15:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:15:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:15:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: osdmap e280: 8 total, 7 up, 8 in 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 9.3 scrub ok 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: osdmap e281: 8 total, 7 up, 8 in 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:15:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:14:59 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:15:00.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:00.235 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:00.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.374 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:00.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:00.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:00.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:00.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:00.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:00.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:00.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:00.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:00.754 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:15:00 smithi003 podman[146564]: 2024-03-21 14:15:00.256485208 +0000 UTC m=+1.179089027 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:15:00.754 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:15:00 smithi003 podman[146564]: 2024-03-21 14:15:00.391477367 +0000 UTC m=+1.314081169 container create 8dcf1c3e7bde1c978c29af550d5eb7308b56a23dbfec0b765d77ae754623d0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T14:15:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:00.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:00.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:00.919 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:00.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:00.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:01.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:01.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:01.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:01.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:00 smithi118 ceph-mon[33862]: 9.1 scrub ok 2024-03-21T14:15:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:00 smithi118 ceph-mon[33862]: pgmap v558: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 965 op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:00 smithi118 ceph-mon[33862]: 9.c scrub starts 2024-03-21T14:15:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:00 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:15:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:00 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:15:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:01.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[29576]: 9.1 scrub ok 2024-03-21T14:15:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[29576]: pgmap v558: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 965 op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[29576]: 9.c scrub starts 2024-03-21T14:15:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:15:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:15:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[40510]: 9.1 scrub ok 2024-03-21T14:15:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[40510]: pgmap v558: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 965 op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[40510]: 9.c scrub starts 2024-03-21T14:15:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:15:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:00 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:15:01.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:01.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:01.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:01.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:01.467 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:01.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:01.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:01.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:01.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:01.742 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:01.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:01.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:01.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:01.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:02.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:02.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:02.147 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:02.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:01 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:15:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:01 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:15:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:01 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:02.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.286 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:02.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:02.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:02.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:02.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:02.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:02.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:02.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:02.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:02.754 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:15:02 smithi003 podman[146564]: 2024-03-21 14:15:02.491563233 +0000 UTC m=+3.414167050 container init 8dcf1c3e7bde1c978c29af550d5eb7308b56a23dbfec0b765d77ae754623d0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-21T14:15:02.755 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:15:02 smithi003 podman[146564]: 2024-03-21 14:15:02.498046589 +0000 UTC m=+3.420650392 container start 8dcf1c3e7bde1c978c29af550d5eb7308b56a23dbfec0b765d77ae754623d0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:15:02.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.832 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:02.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:02.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:02.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:02.968 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:02.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:03.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-21T14:15:03.033 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:15:03.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:03.070 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:02 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:15:03.070 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:02 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:15:03.071 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:02 smithi118 ceph-mon[33862]: pgmap v560: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 454 B/s rd, 0 B/s wr, 857 op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:03.071 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:02 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:15:03.071 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:02 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:15:03.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:15:03.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:15:03.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[29576]: pgmap v560: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 454 B/s rd, 0 B/s wr, 857 op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[40510]: pgmap v560: 158 pgs: 21 active+undersized+degraded, 1 active+clean+scrubbing, 39 active+undersized, 97 active+clean; 8.0 GiB data, 28 GiB used, 688 GiB / 715 GiB avail; 454 B/s rd, 0 B/s wr, 857 op/s; 41098/396015 objects degraded (10.378%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:15:03.096 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:02 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:15:03.103 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:03.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.243 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:03.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:03.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.407 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:03.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:03.457 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:03.504 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:15:03 smithi003 podman[146564]: 2024-03-21 14:15:03.331284616 +0000 UTC m=+4.253888446 container attach 8dcf1c3e7bde1c978c29af550d5eb7308b56a23dbfec0b765d77ae754623d0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-21T14:15:03.504 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:15:03 smithi003 podman[146850]: 2024-03-21 14:15:03.332594078 +0000 UTC m=+0.369302496 container died 8dcf1c3e7bde1c978c29af550d5eb7308b56a23dbfec0b765d77ae754623d0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T14:15:03.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:03.553 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:03.655 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:03.694 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:03.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:03.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:03.834 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:03.863 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:03 smithi118 ceph-mon[33862]: 9.c scrub ok 2024-03-21T14:15:03.863 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:03 smithi118 ceph-mon[33862]: 9.a scrub starts 2024-03-21T14:15:03.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:03.952 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:03 smithi003 ceph-mon[29576]: 9.c scrub ok 2024-03-21T14:15:03.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:03 smithi003 ceph-mon[29576]: 9.a scrub starts 2024-03-21T14:15:03.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:03 smithi003 ceph-mon[40510]: 9.c scrub ok 2024-03-21T14:15:03.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:03 smithi003 ceph-mon[40510]: 9.a scrub starts 2024-03-21T14:15:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:03.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T14:15:03.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:04.049 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:15:04.049 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T14:15:04.049 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-21T14:15:04.050 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi003.front.sepia.ceph.com 2024-03-21T14:15:04.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-21T14:15:04.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:04.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:04.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:04.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:04.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:04.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:04.357 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:15:04.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:04.437 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:04.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:04.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:04.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:04.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:04.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:04.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:04.718 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:04.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:04.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:04.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:04.860 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:04.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: pgmap v561: 158 pgs: 1 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 97 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.66k op/s; 38789/373593 objects degraded (10.383%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 38789/373593 objects degraded (10.383%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:15:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:15:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:15:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:15:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: osdmap e282: 8 total, 7 up, 8 in 2024-03-21T14:15:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/497021635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:04.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:04 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:04.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.004 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:05.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: pgmap v561: 158 pgs: 1 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 97 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.66k op/s; 38789/373593 objects degraded (10.383%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 38789/373593 objects degraded (10.383%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: osdmap e282: 8 total, 7 up, 8 in 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/497021635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:05.072 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: pgmap v561: 158 pgs: 1 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 97 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.66k op/s; 38789/373593 objects degraded (10.383%); 0 B/s, 0 objects/s recovering 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 38789/373593 objects degraded (10.383%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: osdmap e282: 8 total, 7 up, 8 in 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/497021635' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:05.073 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:04 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:05.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:05.240 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:05.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:05.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:05.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:05.420 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:05.492 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:15:05.492 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:15:05.492 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-03-21T14:15:05.492 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:15:05.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:05.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:05.595 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:15:05.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:05.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:05.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.838 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:05.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: 6.2 deep-scrub starts 2024-03-21T14:15:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: 6.2 deep-scrub ok 2024-03-21T14:15:05.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:15:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:15:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:15:05.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:05 smithi118 ceph-mon[33862]: osdmap e283: 8 total, 7 up, 8 in 2024-03-21T14:15:05.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:05.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:05.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: 6.2 deep-scrub starts 2024-03-21T14:15:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: 6.2 deep-scrub ok 2024-03-21T14:15:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:15:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:15:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[29576]: osdmap e283: 8 total, 7 up, 8 in 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: 6.2 deep-scrub starts 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: 6.2 deep-scrub ok 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:15:06.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:05 smithi003 ceph-mon[40510]: osdmap e283: 8 total, 7 up, 8 in 2024-03-21T14:15:06.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:06.118 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:06.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:06.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:06.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:06.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:06.492 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:06.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.530 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:06.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:06.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:15:06.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:06.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:06.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:15:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:15:06.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:06.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:06.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:06.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:06.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:07.024 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:07.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:07.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:07.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:15:07.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:15:07.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-mon[29576]: pgmap v563: 158 pgs: 1 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 97 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 763 B/s rd, 0 B/s wr, 1.11k op/s; 38789/373593 objects degraded (10.383%) 2024-03-21T14:15:07.111 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:15:07.111 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:15:07.111 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:06 smithi003 ceph-mon[40510]: pgmap v563: 158 pgs: 1 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 97 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 763 B/s rd, 0 B/s wr, 1.11k op/s; 38789/373593 objects degraded (10.383%) 2024-03-21T14:15:07.172 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:15:07.179 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:15:06 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:15:07.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:06 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:15:07.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:06 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:15:07.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:06 smithi118 ceph-mon[33862]: pgmap v563: 158 pgs: 1 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 97 active+clean; 7.6 GiB data, 27 GiB used, 689 GiB / 715 GiB avail; 763 B/s rd, 0 B/s wr, 1.11k op/s; 38789/373593 objects degraded (10.383%) 2024-03-21T14:15:07.206 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:15:07.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:07.243 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:15:07.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:07.284 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:15:07.322 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:15:07.351 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:07.355 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:15:07.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:07.396 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:15:07.396 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:15:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:15:07.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:07.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:07.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:07.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.664 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:07.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:07.716 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:15:07.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.800 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:07.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:07.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:07.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:07.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:07 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/68924840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:07 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/68924840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:08.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:08.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:08.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:07 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/68924840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:08.220 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:08.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:08.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:08.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:08.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:08.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:15:08.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:15:08.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:08.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:08.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:08.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:08.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:15:08.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:08.930 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:08.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:09.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:09.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:09.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:09.141 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:08 smithi003 ceph-mon[29576]: pgmap v565: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.82k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:09.141 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:08 smithi003 ceph-mon[40510]: pgmap v565: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.82k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:09.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:08 smithi118 ceph-mon[33862]: pgmap v565: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.82k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:09.221 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:09.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:09.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:09.358 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:09.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:09.479 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:09.522 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:15:09.567 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:15:09.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:09.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:09.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:09.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:09.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:09.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:09.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:10.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:10.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:10.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:09 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 36542/352764 objects degraded (10.359%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:09 smithi118 ceph-mon[33862]: osdmap e284: 8 total, 7 up, 8 in 2024-03-21T14:15:10.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:10.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:09 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 36542/352764 objects degraded (10.359%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:09 smithi003 ceph-mon[29576]: osdmap e284: 8 total, 7 up, 8 in 2024-03-21T14:15:10.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:09 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 36542/352764 objects degraded (10.359%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:10.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:09 smithi003 ceph-mon[40510]: osdmap e284: 8 total, 7 up, 8 in 2024-03-21T14:15:10.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:15:10.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:10.364 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:10.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:10.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:10.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:10.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:15:10.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:10.653 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:10.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:10.665 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:10.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:10.808 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:10.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:10.910 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:10.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:10.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:10.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:10 smithi003 ceph-mon[29576]: pgmap v566: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:10.976 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:10 smithi003 ceph-mon[40510]: pgmap v566: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:11.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:11.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:10 smithi118 ceph-mon[33862]: pgmap v566: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:11.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:11.240 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:11.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:11.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:11.388 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:11.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:11.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:11.505 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:11.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:11.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:11.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:11.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:11.826 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:11.858 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:11.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:11.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:11.995 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:11.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:11 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:15:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:11 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:15:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:11 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:15:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:11 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:15:12.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:12.133 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:11 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:15:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:11 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:15:12.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:12.263 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:12.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:12.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:12.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:12.540 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:12.642 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:12.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.677 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:12.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:12.778 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:12.817 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:12.888 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:15:12.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:12.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:12.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:13.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:13.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:13.159 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:12 smithi118 ceph-mon[33862]: pgmap v568: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.00k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:13.159 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:12 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:15:13.159 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:12 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:15:13.159 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:12 smithi118 ceph-mon[33862]: osdmap e285: 8 total, 7 up, 8 in 2024-03-21T14:15:13.159 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2451447359' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:13.204 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.239 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:13.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[29576]: pgmap v568: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.00k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:15:13.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[29576]: osdmap e285: 8 total, 7 up, 8 in 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2451447359' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[40510]: pgmap v568: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 7.2 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 1.00k op/s; 36542/352764 objects degraded (10.359%) 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[40510]: osdmap e285: 8 total, 7 up, 8 in 2024-03-21T14:15:13.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2451447359' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:13.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:13.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:13.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:13.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:13.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:13.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:13.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:13.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:13.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:13.883 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:13.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:13.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:14.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:14.055 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:14.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:13 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:15:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:13 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:15:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:13 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2451447359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:13 smithi118 ceph-mon[33862]: osdmap e286: 8 total, 7 up, 8 in 2024-03-21T14:15:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.190 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:14.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:15:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:15:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2451447359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[29576]: osdmap e286: 8 total, 7 up, 8 in 2024-03-21T14:15:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:15:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:15:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2451447359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:13 smithi003 ceph-mon[40510]: osdmap e286: 8 total, 7 up, 8 in 2024-03-21T14:15:14.291 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:14.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:14.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.450 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:14.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:14.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:14.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.591 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:14.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:14.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:14.726 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:14.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:14.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:14.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:14.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:14.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:14.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:14 smithi003 ceph-mon[29576]: pgmap v571: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.25k op/s; 34198/330354 objects degraded (10.352%) 2024-03-21T14:15:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:14 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 34198/330354 objects degraded (10.352%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:14 smithi003 ceph-mon[40510]: pgmap v571: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.25k op/s; 34198/330354 objects degraded (10.352%) 2024-03-21T14:15:15.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:14 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 34198/330354 objects degraded (10.352%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:15.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:15.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:14 smithi118 ceph-mon[33862]: pgmap v571: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.25k op/s; 34198/330354 objects degraded (10.352%) 2024-03-21T14:15:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:14 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 34198/330354 objects degraded (10.352%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:9.b 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:9.7 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:9.2 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:9.0 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.9 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.5 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.18 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.17 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.12 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.11 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:6.0 2024-03-21T14:15:15.227 INFO:teuthology.orchestra.run.smithi003.stdout:5.a 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:3.4 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:5.b 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:5.14 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:3.1a 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:5.1c 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:3.13 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:1.1 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:5.8 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:4.19 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:3.1f 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:6.8 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:1.9 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:4.13 2024-03-21T14:15:15.228 INFO:teuthology.orchestra.run.smithi003.stdout:6.1 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:1.5 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:3.1e 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:2.1 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:1.0 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:3.18 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:6.1b 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:1.8 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:1.7 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:4.1e 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:4.d 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:5.6 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:3.0 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:1.a 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:3.5 2024-03-21T14:15:15.229 INFO:teuthology.orchestra.run.smithi003.stdout:4.12 2024-03-21T14:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout:6.f 2024-03-21T14:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout:5.2 2024-03-21T14:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout:5.d 2024-03-21T14:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout:3.14 2024-03-21T14:15:15.230 INFO:teuthology.orchestra.run.smithi003.stdout:4.17 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:6.19 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:6.14 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:5.4 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:4.18 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:4.1c 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:6.1c 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:4.4 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:4.6 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:3.9 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:4.b 2024-03-21T14:15:15.231 INFO:teuthology.orchestra.run.smithi003.stdout:9.6 2024-03-21T14:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout:6.3 2024-03-21T14:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout:5.18 2024-03-21T14:15:15.232 INFO:teuthology.orchestra.run.smithi003.stdout:5.1e 2024-03-21T14:15:15.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:15.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:15.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:15.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:15.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:15.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:15.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:15.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:15.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.663 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:15.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:15.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:15.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:15.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:15.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:15.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:15.946 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:15.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:16.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:16.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:16.217 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:16.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:16.319 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:16.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:16.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:16.488 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:16.574 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:15:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:15:16.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:16.613 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:16.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:16.741 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:15:16.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:16.779 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:15:16.815 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:15:16.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:16.856 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:15:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:16.891 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:16.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:16.896 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:15:16.931 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:15:16.972 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:15:16.973 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:15:16.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:17.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:17.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:17.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 1.8 --file /var/log/ceph/exp.1.8.2 2024-03-21T14:15:17.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.152 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:17.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:16 smithi118 ceph-mon[33862]: pgmap v572: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.25k op/s; 34198/330354 objects degraded (10.352%) 2024-03-21T14:15:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:16 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1207918867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:16 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:16 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:17.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:15:16 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:15:17.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[29576]: pgmap v572: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.25k op/s; 34198/330354 objects degraded (10.352%) 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1207918867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[40510]: pgmap v572: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.25k op/s; 34198/330354 objects degraded (10.352%) 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1207918867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:17.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:16 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:17.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.286 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:17.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:17.308 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:15:17.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:17.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:17.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:17.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:17.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:17.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:17.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:17.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:17.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:15:17.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:15:17.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:17.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:17.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:15:17.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:17.973 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:17.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:18.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:18.098 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:18.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:17 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:18.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:17 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:18.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:17 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:18.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:18.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:18.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:18.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:18.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:18.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:18.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:18.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:18.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:18.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:18.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:19.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:19.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:19.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:19.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:18 smithi118 ceph-mon[33862]: pgmap v573: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.73k op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:19.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:19.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:18 smithi003 ceph-mon[29576]: pgmap v573: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.73k op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:19.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:18 smithi003 ceph-mon[40510]: pgmap v573: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.73k op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:19.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:19.329 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:19.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:19.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:19.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:19.564 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:19.586 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:19.597 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:19.615 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:15:19.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:19.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:19.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:19.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:15:19.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:19.867 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:19.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:19.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:19.969 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:19.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:19 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 32235/311703 objects degraded (10.342%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:19.971 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:19 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 32235/311703 objects degraded (10.342%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.004 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:20.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:20.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.143 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:20.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:19 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 32235/311703 objects degraded (10.342%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:20.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:20.277 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:20.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:20.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:20.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:20.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:20.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.557 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:20.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:20.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:20.689 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:20.726 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193115 7.4 7:23e77ff3:::benchmark_data_smithi003_80362_object72098:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.718159+0000", 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 467.00290613499999, 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0401704999999999, 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.727 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193115 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718159+0000", 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718159+0000", 2024-03-21T14:15:20.728 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718160+0000", 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1909999999999999e-06 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718165+0000", 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0649999999999998e-06 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.729 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718177+0000", 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1471000000000001e-05 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718183+0000", 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2339999999999999e-06 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718232+0000", 2024-03-21T14:15:20.730 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8414000000000001e-05 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718324+0000", 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2880999999999994e-05 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,6", 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.718401+0000", 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6329999999999996e-05 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.731 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.720669+0000", 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00226822 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.720714+0000", 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4978000000000002e-05 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.726514+0000", 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057999740000000003 2024-03-21T14:15:20.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.726526+0000", 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2184999999999999e-05 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758285+0000", 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.031759222 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758330+0000", 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4335000000000002e-05 2024-03-21T14:15:20.733 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193119 7.4 7:2d9b1e7b:::benchmark_data_smithi003_80362_object72102:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.730256+0000", 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.99080892699999, 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0340747850000001, 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.734 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193119 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730256+0000", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730256+0000", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730258+0000", 2024-03-21T14:15:20.735 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.203e-06 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730262+0000", 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5430000000000004e-06 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730272+0000", 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0074e-05 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730278+0000", 2024-03-21T14:15:20.736 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0299999999999999e-06 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730319+0000", 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0463000000000002e-05 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.730325+0000", 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6710000000000002e-06 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.737 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758690+0000", 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.028364201 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758787+0000", 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7291000000000001e-05 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.758888+0000", 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000101508 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.738 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764227+0000", 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053386040000000003 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764265+0000", 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7851000000000001e-05 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764275+0000", 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0709e-05 2024-03-21T14:15:20.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764308+0000", 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2449000000000001e-05 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764331+0000", 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3187999999999999e-05 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.740 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193125 7.4 7:2ef2efbc:::benchmark_data_smithi003_80362_object72108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.749172+0000", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.97189282699998, 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.019614807, 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193125 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749172+0000", 2024-03-21T14:15:20.741 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.742 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.742 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.742 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749172+0000", 2024-03-21T14:15:20.742 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749174+0000", 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4559999999999999e-06 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749180+0000", 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4030000000000004e-06 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.743 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749194+0000", 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3648e-05 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749202+0000", 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4919999999999993e-06 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749263+0000", 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0850000000000002e-05 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.744 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.749272+0000", 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1689999999999998e-06 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.761037+0000", 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0117644429999999 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.761135+0000", 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7755000000000002e-05 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.745 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.761212+0000", 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6964000000000004e-05 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764339+0000", 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031274359999999999 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768583+0000", 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042440500000000001 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.746 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768677+0000", 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3974999999999994e-05 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768734+0000", 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6718000000000001e-05 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768787+0000", 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3448000000000001e-05 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193129 7.4 7:2937aaf4:::benchmark_data_smithi003_80362_object72112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.754621+0000", 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.96644463199999, 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.014328082, 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193129 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.748 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754621+0000", 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754621+0000", 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754622+0000", 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-03-21T14:15:20.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754626+0000", 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9940000000000002e-06 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754636+0000", 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0409e-05 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754644+0000", 2024-03-21T14:15:20.750 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5530000000000004e-06 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754691+0000", 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7607e-05 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.754700+0000", 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1990000000000006e-06 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.751 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.762687+0000", 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.007987382 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.762750+0000", 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3196999999999998e-05 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.752 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.762817+0000", 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7040999999999997e-05 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768592+0000", 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057748820000000003 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768629+0000", 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6761000000000003e-05 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.753 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768843+0000", 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021430599999999999 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768899+0000", 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5597999999999999e-05 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768949+0000", 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9920000000000003e-05 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.754 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.755 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.755 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.755 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193143 7.4 7:20ba6c65:::benchmark_data_smithi003_80362_object72126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.769319+0000", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.95174673299999, 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99980547399999997, 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193143 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769319+0000", 2024-03-21T14:15:20.756 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769319+0000", 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769320+0000", 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.274e-06 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769326+0000", 2024-03-21T14:15:20.757 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7520000000000002e-06 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769340+0000", 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4537e-05 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769349+0000", 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6819999999999992e-06 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.758 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769402+0000", 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3217999999999998e-05 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.769408+0000", 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7400000000000001e-06 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764364+0000", 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99495648800000003 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.759 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764429+0000", 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4486999999999994e-05 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.764494+0000", 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5752000000000003e-05 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768800+0000", 2024-03-21T14:15:20.760 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043050409999999999 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768984+0000", 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018443600000000001 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769021+0000", 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7500999999999999e-05 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.761 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769073+0000", 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1650000000000002e-05 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769124+0000", 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0915999999999998e-05 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193164 7.4 7:2c7218ff:::benchmark_data_smithi003_80362_object72147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.762 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.797383+0000", 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.92368240299999, 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97718391999999998, 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193164 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.763 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797383+0000", 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797383+0000", 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797384+0000", 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8700000000000004e-07 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.764 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797388+0000", 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8770000000000003e-06 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797395+0000", 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9859999999999997e-06 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797400+0000", 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9830000000000004e-06 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.765 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797445+0000", 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5787000000000001e-05 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797451+0000", 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9320000000000001e-06 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.766211+0000", 2024-03-21T14:15:20.766 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96876004699999996 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.766316+0000", 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105072 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.766435+0000", 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118355 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.767 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.768991+0000", 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0025563259999999998 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769158+0000", 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016738300000000001 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774462+0000", 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053030890000000004 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.768 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774518+0000", 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6388000000000003e-05 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774567+0000", 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8807999999999998e-05 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193165 7.4 7:27395a1d:::benchmark_data_smithi003_80362_object72148:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.769 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.797413+0000", 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.92365217899999, 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.981967598, 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193165 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797413+0000", 2024-03-21T14:15:20.770 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797413+0000", 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797414+0000", 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2599999999999998e-07 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.771 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797416+0000", 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2340000000000001e-06 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797421+0000", 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5910000000000003e-06 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797427+0000", 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.344e-06 2024-03-21T14:15:20.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797466+0000", 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9601000000000003e-05 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.797470+0000", 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3450000000000002e-06 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769193+0000", 2024-03-21T14:15:20.773 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97172361500000004 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769308+0000", 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011507500000000001 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.769435+0000", 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000126489 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774623+0000", 2024-03-21T14:15:20.774 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051881729999999999 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774637+0000", 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4122e-05 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779266+0000", 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046288290000000001 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.775 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779336+0000", 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9722999999999996e-05 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779381+0000", 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5031000000000003e-05 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193181 7.4 7:203760eb:::benchmark_data_smithi003_80362_object72164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.826429+0000", 2024-03-21T14:15:20.776 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.89463600099998, 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95319384900000004, 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193181 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826429+0000", 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.777 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826429+0000", 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826431+0000", 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.48e-06 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826437+0000", 2024-03-21T14:15:20.778 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8490000000000002e-06 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826449+0000", 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2614e-05 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826457+0000", 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6760000000000007e-06 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826506+0000", 2024-03-21T14:15:20.779 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9452e-05 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.826515+0000", 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1409999999999994e-06 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.772040+0000", 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94552436299999998 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.780 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.772148+0000", 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108558 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.772267+0000", 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118244 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779443+0000", 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071765889999999997 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.781 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779458+0000", 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4436e-05 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779564+0000", 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105952 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779596+0000", 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2041e-05 2024-03-21T14:15:20.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779623+0000", 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7453999999999998e-05 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:193188 7.4 7:2a95fdc0:::benchmark_data_smithi003_80362_object72171:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:07:33.833625+0000", 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 466.887439798, 2024-03-21T14:15:20.783 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95566202300000003, 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193188 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833625+0000", 2024-03-21T14:15:20.784 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833625+0000", 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833627+0000", 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3060000000000001e-06 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.785 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833632+0000", 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9060000000000001e-06 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833645+0000", 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3298999999999999e-05 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833653+0000", 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2109999999999998e-06 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.786 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833709+0000", 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5512e-05 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:33.833718+0000", 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3109999999999995e-06 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774673+0000", 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94095470599999997 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.787 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774792+0000", 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119088 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,6", 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.774925+0000", 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013321599999999999 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779462+0000", 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045366859999999998 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.788 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.779512+0000", 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9988000000000003e-05 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789180+0000", 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0096685169999999997 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789237+0000", 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6954999999999999e-05 2024-03-21T14:15:20.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:07:34.789287+0000", 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0321999999999997e-05 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191648 8.18 8:1e671e02:::benchmark_data_smithi003_110242_object60731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 219.97201816899999, 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99582953799999996, 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.790 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191648 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.791 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749048+0000", 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6000000000000003e-07 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749052+0000", 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.544e-06 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749074+0000", 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1254999999999999e-05 2024-03-21T14:15:20.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749082+0000", 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5469999999999997e-06 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749377+0000", 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029512199999999998 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749386+0000", 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8370000000000003e-06 2024-03-21T14:15:20.793 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739054+0000", 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98966760600000003 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739210+0000", 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015671000000000001 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739479+0000", 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026850799999999998 2024-03-21T14:15:20.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744724+0000", 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052452610000000002 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744795+0000", 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0326000000000004e-05 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744804+0000", 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3680000000000008e-06 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744841+0000", 2024-03-21T14:15:20.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7134000000000001e-05 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744877+0000", 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5559999999999998e-05 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191685 8.18 8:1d8c9114:::benchmark_data_smithi003_110242_object60768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.779057+0000", 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 219.94200819100001, 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96600729399999996, 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191685 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779057+0000", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779057+0000", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779058+0000", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779064+0000", 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0799999999999996e-06 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779074+0000", 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.004e-05 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779080+0000", 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8920000000000003e-06 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779153+0000", 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2942999999999996e-05 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779161+0000", 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5959999999999998e-06 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.741379+0000", 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96221795600000004 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.741493+0000", 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113732 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.741693+0000", 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020050499999999999 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744899+0000", 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003205449 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744973+0000", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3985000000000004e-05 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744995+0000", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1997999999999998e-05 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745027+0000", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1952999999999997e-05 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745064+0000", 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191764 8.18 8:1de6f7d8:::benchmark_data_smithi003_110242_object60847:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.846919+0000", 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 219.87414628400001, 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90162940300000005, 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.805 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191764 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.846919+0000", 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.846919+0000", 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.846920+0000", 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.358e-06 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.846926+0000", 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2410000000000003e-06 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.846939+0000", 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3502999999999999e-05 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.846947+0000", 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3890000000000005e-06 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847011+0000", 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3483000000000002e-05 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847020+0000", 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1949999999999992e-06 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.808 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.743057+0000", 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89603660299999999 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.743193+0000", 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135945 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.743424+0000", 2024-03-21T14:15:20.809 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000230886 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745110+0000", 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016864829999999999 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.746801+0000", 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016913849999999999 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.748443+0000", 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001641368 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.748499+0000", 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5699000000000003e-05 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.748548+0000", 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9864999999999998e-05 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191768 8.18 8:1ab39f41:::benchmark_data_smithi003_110242_object60851:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.847826+0000", 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 219.87323948400001, 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90271366799999997, 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191768 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.812 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847826+0000", 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847826+0000", 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847827+0000", 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.71e-07 2024-03-21T14:15:20.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847831+0000", 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1080000000000003e-06 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847841+0000", 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9299999999999998e-06 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847847+0000", 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0129999999999999e-06 2024-03-21T14:15:20.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847892+0000", 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5685000000000001e-05 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.847898+0000", 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1839999999999998e-06 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745184+0000", 2024-03-21T14:15:20.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89728690200000005 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745293+0000", 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010856 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745482+0000", 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000188887 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.748564+0000", 2024-03-21T14:15:20.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030820169999999998 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.750331+0000", 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017674120000000001 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.750385+0000", 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4088999999999997e-05 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.750471+0000", 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5987000000000002e-05 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.750539+0000", 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8022999999999996e-05 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219191 8.5 8:a034a08e:::benchmark_data_smithi003_110242_object88274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:15:20.819 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.19265135699999, 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0133928029999999, 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219191 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528415+0000", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3300000000000001e-07 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528419+0000", 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0219999999999998e-06 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528427+0000", 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5499999999999995e-06 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528435+0000", 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.199e-06 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528488+0000", 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3267999999999999e-05 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528497+0000", 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6379999999999999e-06 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536200+0000", 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.007702536 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536291+0000", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0702e-05 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536577+0000", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028596900000000001 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539703+0000", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003126396 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541643+0000", 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001940083 2024-03-21T14:15:20.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541733+0000", 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0246000000000003e-05 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541768+0000", 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4150000000000003e-05 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541807+0000", 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9211000000000001e-05 2024-03-21T14:15:20.825 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219208 8.5 8:a331d64d:::benchmark_data_smithi003_110242_object88291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.553819+0000", 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.16724666100001, 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98820481199999999, 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.826 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219208 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553819+0000", 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553819+0000", 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553820+0000", 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1069999999999999e-06 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553824+0000", 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1500000000000001e-06 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553834+0000", 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9469999999999998e-06 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553839+0000", 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3709999999999999e-06 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553884+0000", 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4468000000000002e-05 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553889+0000", 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5829999999999997e-06 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.538272+0000", 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98438244100000005 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.538358+0000", 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6626999999999996e-05 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.538604+0000", 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024539899999999999 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541850+0000", 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003246709 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541865+0000", 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4748e-05 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541933+0000", 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8108999999999995e-05 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541986+0000", 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3062999999999997e-05 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542023+0000", 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7089999999999999e-05 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.832 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219209 8.5 8:a65be5ed:::benchmark_data_smithi003_110242_object88292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.554777+0000", 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.166288702, 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98745134800000001, 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219209 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.833 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554777+0000", 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554777+0000", 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554778+0000", 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.187e-06 2024-03-21T14:15:20.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554783+0000", 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3619999999999997e-06 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554807+0000", 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4091999999999999e-05 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554816+0000", 2024-03-21T14:15:20.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9590000000000001e-06 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554874+0000", 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7794000000000003e-05 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554884+0000", 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0231000000000001e-05 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539749+0000", 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98486504200000002 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539862+0000", 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112456 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.540109+0000", 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024730100000000001 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541873+0000", 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001764034 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542094+0000", 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220984 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542143+0000", 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9013e-05 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542191+0000", 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7596000000000003e-05 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542228+0000", 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7296999999999999e-05 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219252 8.5 8:a50faeae:::benchmark_data_smithi003_110242_object88335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.839 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.13290131599999, 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96401971099999995, 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219252 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:15:20.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2399999999999998e-07 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588167+0000", 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.756e-06 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588177+0000", 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9049999999999992e-06 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588183+0000", 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2430000000000002e-06 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588301+0000", 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011804299999999999 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588309+0000", 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7999999999999999e-06 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542276+0000", 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95396667899999998 2024-03-21T14:15:20.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542404+0000", 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128181 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.543001+0000", 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059724599999999995 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547671+0000", 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046693409999999996 2024-03-21T14:15:20.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547685+0000", 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4728e-05 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552079+0000", 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043938019999999996 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552135+0000", 2024-03-21T14:15:20.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5318000000000001e-05 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552184+0000", 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9144999999999998e-05 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219264 8.5 8:a6e0eab4:::benchmark_data_smithi003_110242_object88347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.599939+0000", 2024-03-21T14:15:20.846 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.121126183, 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96534715100000001, 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219264 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599939+0000", 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599939+0000", 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599940+0000", 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.279e-06 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599946+0000", 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6489999999999996e-06 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599958+0000", 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2094e-05 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599969+0000", 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1185999999999999e-05 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.600013+0000", 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3992000000000002e-05 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.600019+0000", 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7969999999999997e-06 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.545425+0000", 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94540629399999998 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.545546+0000", 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120964 2024-03-21T14:15:20.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.546053+0000", 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050637399999999997 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.551970+0000", 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005916915 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552203+0000", 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000233699 2024-03-21T14:15:20.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565192+0000", 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012988474999999999 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565244+0000", 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2469000000000002e-05 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565286+0000", 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1964e-05 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219331 8.5 8:a2b12185:::benchmark_data_smithi003_110242_object88414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.659873+0000", 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.061192339, 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90553297399999999, 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.854 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219331 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659873+0000", 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659873+0000", 2024-03-21T14:15:20.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659874+0000", 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.291e-06 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659881+0000", 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3690000000000003e-06 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.856 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659892+0000", 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1008e-05 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659898+0000", 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.528e-06 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659971+0000", 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3299000000000004e-05 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659981+0000", 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547730+0000", 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88774864799999997 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547857+0000", 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012742799999999999 2024-03-21T14:15:20.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.548159+0000", 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000301685 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552037+0000", 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038776459999999998 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552211+0000", 2024-03-21T14:15:20.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017444100000000001 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565328+0000", 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013117070999999999 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565372+0000", 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3955000000000002e-05 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565406+0000", 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738000000000001e-05 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219333 8.5 8:a1fc9cb9:::benchmark_data_smithi003_110242_object88416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.660701+0000", 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 191.060364545, 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90488013300000003, 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:20.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219333 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660701+0000", 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660701+0000", 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:20.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660702+0000", 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4900000000000005e-07 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660706+0000", 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6280000000000004e-06 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660718+0000", 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2159e-05 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.863 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660727+0000", 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3410000000000003e-06 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660782+0000", 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4920999999999998e-05 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660790+0000", 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6889999999999993e-06 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.864 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.549933+0000", 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88914224600000002 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.550057+0000", 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012411099999999999 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.550306+0000", 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024982500000000001 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552257+0000", 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001950984 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.561349+0000", 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090918709999999996 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565442+0000", 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040930489999999996 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565531+0000", 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8644e-05 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565581+0000", 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9815999999999999e-05 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:20.867 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:15:20.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:20.869 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:20.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:20.970 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:21.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:20 smithi003 ceph-mon[29576]: pgmap v574: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.71k op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:21.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:20 smithi003 ceph-mon[40510]: pgmap v574: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.71k op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:21.006 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:Exporting 1.8 info 1.8( v 167'130 (0'0,167'130] local-lis/les=273/274 n=0 ec=268/19 lis/c=273/273 les/c/f=274/274/0 sis=273) 2024-03-21T14:15:21.054 INFO:teuthology.orchestra.run.smithi003.stderr:Read #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-03-21T14:15:21.054 INFO:teuthology.orchestra.run.smithi003.stderr:Export successful 2024-03-21T14:15:21.078 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:20 smithi118 ceph-mon[33862]: pgmap v574: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.71k op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:21.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:21.145 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:21.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:21.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:21.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:21.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:21.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:21.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:21.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:21.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:21.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:21.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:21.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:21.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:15:21.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:21.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:21.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:21.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:21.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:22.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:22.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:22.208 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:22.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:22.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:22.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:22.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:22.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:22.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:22.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:22.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 1.8 2024-03-21T14:15:22.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:22.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:22.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:22.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:22.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:22.820 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:15:22.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:22.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:22.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:22.971 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:22.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:23.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:23.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.108 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:23.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:23.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:22 smithi003 ceph-mon[29576]: pgmap v575: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 424 B/s rd, 0 B/s wr, 889 op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:23.174 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:22 smithi003 ceph-mon[40510]: pgmap v575: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 424 B/s rd, 0 B/s wr, 889 op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:23.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:23.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:23.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:23.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:23.390 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:22 smithi118 ceph-mon[33862]: pgmap v575: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 6.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 424 B/s rd, 0 B/s wr, 889 op/s; 32235/311703 objects degraded (10.342%) 2024-03-21T14:15:23.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:23.522 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:23.624 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:23.662 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:23.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:23.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:23.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:23.899 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:23.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:23.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:23.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:15:24.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:24.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:24.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:24.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:24.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/476356324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:15:24.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:23 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:15:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:23 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/476356324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:15:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:23 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:15:24.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:24.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:24.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:23 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/476356324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:15:24.428 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:23 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:15:24.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:24.490 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:24.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:24.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:24.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:24.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:24.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:24.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:24.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:24.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:24.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:24.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:25.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:25.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:25.131 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.168 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:25.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:15:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189850 9.a 9:578e36e8:::benchmark_data_smithi003_135814_object49116:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.158545+0000", 2024-03-21T14:15:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.01341385, 2024-03-21T14:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013448686, 2024-03-21T14:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189850 2024-03-21T14:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.158545+0000", 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.158545+0000", 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.158546+0000", 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.158552+0000", 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9020000000000004e-06 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.158561+0000", 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0860000000000008e-06 2024-03-21T14:15:25.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.158569+0000", 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7440000000000007e-06 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.161701+0000", 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031321869999999998 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.161804+0000", 2024-03-21T14:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010340600000000001 2024-03-21T14:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:15:25.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.161940+0000", 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013625700000000001 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167146+0000", 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052051620000000002 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167158+0000", 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2116000000000001e-05 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.181 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189859 9.6 9:6fc08b8b:::benchmark_data_smithi003_135814_object49125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.167302+0000", 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0046570450000000003, 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047852809999999997, 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.182 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189859 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167302+0000", 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167302+0000", 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167303+0000", 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3540000000000001e-06 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167308+0000", 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8659999999999997e-06 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167320+0000", 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2439000000000001e-05 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167328+0000", 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6739999999999997e-06 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167458+0000", 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013051499999999999 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167587+0000", 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128536 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1", 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167702+0000", 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114689 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170456+0000", 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027545640000000001 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.186 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189842 9.a 9:5ce48edb:::benchmark_data_smithi003_135814_object49108:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.153415+0000", 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.018543601, 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018750889, 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189842 2024-03-21T14:15:25.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153415+0000", 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153415+0000", 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153416+0000", 2024-03-21T14:15:25.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2440000000000001e-06 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153421+0000", 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.741e-06 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153430+0000", 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4390000000000006e-06 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153438+0000", 2024-03-21T14:15:25.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0530000000000007e-06 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153553+0000", 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114881 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153666+0000", 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112487 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153776+0000", 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109888 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.161634+0000", 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007857859 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.161674+0000", 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0933000000000003e-05 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.192 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189843 9.a 9:51f4c43d:::benchmark_data_smithi003_135814_object49109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.153457+0000", 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.018501191, 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018793302000000001, 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189843 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153457+0000", 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153457+0000", 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153458+0000", 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0399999999999996e-07 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153461+0000", 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3139999999999999e-06 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153468+0000", 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7290000000000003e-06 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153474+0000", 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9950000000000002e-06 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.156456+0000", 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029824700000000001 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.156568+0000", 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000111239 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,7", 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.157270+0000", 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070286299999999997 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.161638+0000", 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043678789999999999 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.167109+0000", 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054711289999999999 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189861 9.f 9:f2eb6ad4:::benchmark_data_smithi003_135814_object49127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.197 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.169369+0000", 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00258918, 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002968872, 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189861 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169369+0000", 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169369+0000", 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169371+0000", 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.513e-06 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169375+0000", 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8720000000000004e-06 2024-03-21T14:15:25.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169388+0000", 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3685e-05 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169395+0000", 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3799999999999999e-06 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169496+0000", 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010119 2024-03-21T14:15:25.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169635+0000", 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000138691 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,3", 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169755+0000", 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011983499999999999 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.201 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189844 9.7 9:ebbba101:::benchmark_data_smithi003_135814_object49110:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.153500+0000", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.018458456000000002, 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018919635000000001, 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189844 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153500+0000", 2024-03-21T14:15:25.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153500+0000", 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153501+0000", 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3000000000000001e-07 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153504+0000", 2024-03-21T14:15:25.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0410000000000002e-06 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153510+0000", 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3890000000000002e-06 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.153515+0000", 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2800000000000003e-06 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154720+0000", 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012050229999999999 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154825+0000", 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104943 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 7", 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154940+0000", 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114696 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.162681+0000", 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0077411279999999999 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189862 9.b 9:de0c6e87:::benchmark_data_smithi003_135814_object49128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.169412+0000", 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0025462979999999998, 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003073765, 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.206 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189862 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169412+0000", 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169412+0000", 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169413+0000", 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4099999999999999e-07 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169415+0000", 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3810000000000002e-06 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169421+0000", 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4689999999999996e-06 2024-03-21T14:15:25.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169425+0000", 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8779999999999997e-06 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169523+0000", 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7147999999999999e-05 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169816+0000", 2024-03-21T14:15:25.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000293622 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1", 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.169953+0000", 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137193 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189845 9.7 9:ee33bd68:::benchmark_data_smithi003_135814_object49111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.154655+0000", 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.017303820000000001, 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017874946999999999, 2024-03-21T14:15:25.210 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189845 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154655+0000", 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154655+0000", 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154655+0000", 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1200000000000002e-07 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154660+0000", 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.369e-06 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154667+0000", 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6699999999999994e-06 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.154674+0000", 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5420000000000002e-06 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.157556+0000", 2024-03-21T14:15:25.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028816129999999999 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.157666+0000", 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011035399999999999 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 7", 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.157775+0000", 2024-03-21T14:15:25.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000108603 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.162688+0000", 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049134310000000002 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189863 9.7 9:ec752a7d:::benchmark_data_smithi003_135814_object49129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.170963+0000", 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0009951719999999999, 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001626635, 2024-03-21T14:15:25.215 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189863 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170963+0000", 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170963+0000", 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170964+0000", 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7000000000000003e-07 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170970+0000", 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4399999999999996e-06 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170982+0000", 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2542e-05 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.170990+0000", 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4000000000000003e-06 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171058+0000", 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8668000000000001e-05 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171170+0000", 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000111329 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 7", 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171280+0000", 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000110297 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189846 9.7 9:e2ffdc11:::benchmark_data_smithi003_135814_object49112:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.155371+0000", 2024-03-21T14:15:25.219 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.016587896000000001, 2024-03-21T14:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.017261775, 2024-03-21T14:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:25.220 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189846 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.155371+0000", 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.155371+0000", 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.155371+0000", 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8999999999999996e-07 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.155375+0000", 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6349999999999999e-06 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.155383+0000", 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8390000000000007e-06 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.155388+0000", 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1569999999999999e-06 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.160249+0000", 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048611390000000004 2024-03-21T14:15:25.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.160360+0000", 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011068799999999999 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 7", 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.160473+0000", 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113365 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.162919+0000", 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00244604 2024-03-21T14:15:25.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:189865 9.6 9:6b301c36:::benchmark_data_smithi003_135814_object49131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:25.171005+0000", 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00095392299999999997, 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016731440000000001, 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:25.225 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 189865 2024-03-21T14:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.226 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171005+0000", 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171005+0000", 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:25.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171005+0000", 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5299999999999999e-07 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171008+0000", 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5179999999999999e-06 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171013+0000", 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.259e-06 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.171018+0000", 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8049999999999997e-06 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:25.172633+0000", 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0016151659999999999 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 11 2024-03-21T14:15:25.229 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: pgmap v576: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 867 B/s rd, 0 B/s wr, 1.26k op/s; 29949/290169 objects degraded (10.321%) 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: osdmap e287: 8 total, 7 up, 8 in 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 29949/290169 objects degraded (10.321%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: osdmap e288: 8 total, 7 up, 8 in 2024-03-21T14:15:25.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:15:25.231 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:24 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:15:25.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: pgmap v576: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 867 B/s rd, 0 B/s wr, 1.26k op/s; 29949/290169 objects degraded (10.321%) 2024-03-21T14:15:25.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:25.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:15:25.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: osdmap e287: 8 total, 7 up, 8 in 2024-03-21T14:15:25.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 29949/290169 objects degraded (10.321%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: osdmap e288: 8 total, 7 up, 8 in 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: pgmap v576: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 867 B/s rd, 0 B/s wr, 1.26k op/s; 29949/290169 objects degraded (10.321%) 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: osdmap e287: 8 total, 7 up, 8 in 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 29949/290169 objects degraded (10.321%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: osdmap e288: 8 total, 7 up, 8 in 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:15:25.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:24 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:15:25.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:25.302 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:25.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:25.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:25.478 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:15:25.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:25.580 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:25.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.718 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:25.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:25.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:25.859 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:25.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:25.961 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:26.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.001 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:26.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:26.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:26.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:25 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:15:26.130 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:25 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:15:26.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:25 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:15:26.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:15:26.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:26.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.338 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:26.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:26.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:26.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:26.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:26.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:26.670 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:26.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:26.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:15:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:15:26.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:26.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:26.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:26.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.028 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:27.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:27.075 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:26 smithi118 ceph-mon[33862]: pgmap v579: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s; 29949/290169 objects degraded (10.321%) 2024-03-21T14:15:27.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:26 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:15:27.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:26 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:15:27.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:26 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:15:27.076 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:26 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:15:27.076 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:15:27.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.165 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:27.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:27.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[29576]: pgmap v579: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s; 29949/290169 objects degraded (10.321%) 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[40510]: pgmap v579: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 897 op/s; 29949/290169 objects degraded (10.321%) 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:15:27.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:26 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:15:27.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:27.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:27.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:27.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:27.442 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:27.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:27.576 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:27.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:27.678 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:27.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:27.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:27.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:27.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:27.858 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:27.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:27.940 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:15:27.947 INFO:teuthology.orchestra.run.smithi003.stdout: marking collection for removal 2024-03-21T14:15:27.947 INFO:teuthology.orchestra.run.smithi003.stdout:setting '_remove' omap key 2024-03-21T14:15:27.948 INFO:teuthology.orchestra.run.smithi003.stdout:finish_remove_pgs 1.8_head removing 1.8 2024-03-21T14:15:27.951 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:15:27.953 INFO:teuthology.orchestra.run.smithi003.stdout:Remove successful 2024-03-21T14:15:27.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:27.988 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:15:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.007 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:28.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:28.020 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:15:28.059 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:15:28.093 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:15:28.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1581467253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:27 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1581467253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:28.095 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:15:28.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:15:28.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:15:28.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:15:28.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:15:28.096 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:27 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1581467253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:28.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:28.141 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:15:28.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:15:28.141 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:15:28.141 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:15:28.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:15:28.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:15:28.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:15:28.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:15:28.142 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:15:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:28.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:28.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:28.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:28.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:28.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:28.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:28.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:28.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:28.580 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:15:28.580 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:15:28.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:28.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:28.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:28.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:28.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:28.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:15:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:15:28.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:15:28.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:28.979 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:28.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:29.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.117 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:29.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:29.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:29.230 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:28 smithi118 ceph-mon[33862]: pgmap v580: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:29.231 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:28 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:15:29.231 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:28 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:15:29.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:28 smithi003 ceph-mon[40510]: pgmap v580: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:29.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:28 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:15:29.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:28 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:15:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:28 smithi003 ceph-mon[29576]: pgmap v580: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:29.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:28 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:15:29.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:28 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:15:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.260 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:29.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:29.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:15:29.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:29.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:29.499 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:29.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:29.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:29.753 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:29.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:29.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.1.8.2 2024-03-21T14:15:29.834 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:29.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:29.852 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:29.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:29.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:29.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:29.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 27694/268506 objects degraded (10.314%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:15:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:15:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:29 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 27694/268506 objects degraded (10.314%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:30.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:30.116 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:30.167 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:15:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:29 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:15:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:29 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:15:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:29 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:15:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:29 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:15:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:29 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 27694/268506 objects degraded (10.314%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:30.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:30.249 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:30.352 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:30.389 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:30.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:30.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:30.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:30.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:30.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:30.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:30.800 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:30.902 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:30.944 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:30.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:30.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:15:31.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:31.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:31.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:31.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:31.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:30 smithi118 ceph-mon[33862]: pgmap v581: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:31.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:30 smithi118 ceph-mon[33862]: 9.9 scrub starts 2024-03-21T14:15:31.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:30 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:15:31.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:30 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:15:31.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:30 smithi118 ceph-mon[33862]: 9.a scrub starts 2024-03-21T14:15:31.221 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[29576]: pgmap v581: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[29576]: 9.9 scrub starts 2024-03-21T14:15:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:15:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:15:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[29576]: 9.a scrub starts 2024-03-21T14:15:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[40510]: pgmap v581: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[40510]: 9.9 scrub starts 2024-03-21T14:15:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:15:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:15:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:30 smithi003 ceph-mon[40510]: 9.a scrub starts 2024-03-21T14:15:31.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:31.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:31.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:31.487 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.488 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:31.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:31.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:31.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:31.716 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:15:31.716 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:15:31.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:31.770 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:31.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:31.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:31.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:32.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:32.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:32.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:32.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:15:32.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:15:32.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:15:32.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:15:32.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:32.224 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:15:32.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:15:32.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:15:32.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:15:32.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:31 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:32.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:32.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:32.352 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:32.364 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:31 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:15:32.364 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:31 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:15:32.364 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:31 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:15:32.364 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:31 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:15:32.365 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:31 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:32.390 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:32.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:15:32.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:32.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:32.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:32.664 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:32.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:32.797 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:32.878 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:32.899 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:32.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:32.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:32.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:33.009 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:15:33.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:33.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:33.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:33.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:33.220 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: pgmap v582: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 902 op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: 9.9 scrub ok 2024-03-21T14:15:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:15:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:15:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: 9.a scrub ok 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: 9.5 scrub starts 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1158059811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: pgmap v582: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 902 op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: 9.9 scrub ok 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: 9.a scrub ok 2024-03-21T14:15:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: 9.5 scrub starts 2024-03-21T14:15:33.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1158059811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:33.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:32 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:33.318 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: pgmap v582: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 902 op/s; 27694/268506 objects degraded (10.314%) 2024-03-21T14:15:33.318 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: 9.9 scrub ok 2024-03-21T14:15:33.318 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:15:33.318 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:15:33.318 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: 9.a scrub ok 2024-03-21T14:15:33.319 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: 9.5 scrub starts 2024-03-21T14:15:33.319 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1158059811' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:33.319 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:32 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:33.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:33.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:33.428 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:15:33.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:33.497 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:33.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:33.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:33.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:33.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:33.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:33.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:33.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:33.909 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:33.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:34.010 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:34.049 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:34.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:15:34.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:34.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: osdmap e289: 8 total, 7 up, 8 in 2024-03-21T14:15:34.101 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: osdmap e289: 8 total, 7 up, 8 in 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:15:34.102 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:33 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:15:34.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:34.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:34.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:34.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:34.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: osdmap e289: 8 total, 7 up, 8 in 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:15:34.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:33 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:15:34.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:34.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:34.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:34.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:34.739 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:34.841 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:34.854 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:15:34.854 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:15:34.879 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:34.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:34.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:35.015 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:35.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:35.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:35.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: pgmap v584: 158 pgs: 2 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 96 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 25495/246888 objects degraded (10.327%) 2024-03-21T14:15:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: 9.5 scrub ok 2024-03-21T14:15:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:15:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:15:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:15:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:15:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 25495/246888 objects degraded (10.327%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:34 smithi118 ceph-mon[33862]: osdmap e290: 8 total, 7 up, 8 in 2024-03-21T14:15:35.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: pgmap v584: 158 pgs: 2 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 96 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 25495/246888 objects degraded (10.327%) 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: 9.5 scrub ok 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 25495/246888 objects degraded (10.327%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[29576]: osdmap e290: 8 total, 7 up, 8 in 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: pgmap v584: 158 pgs: 2 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 96 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 25495/246888 objects degraded (10.327%) 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: 9.5 scrub ok 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 25495/246888 objects degraded (10.327%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:34 smithi003 ceph-mon[40510]: osdmap e290: 8 total, 7 up, 8 in 2024-03-21T14:15:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:35.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:35.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:35.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:35.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.559 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:35.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:35.576 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T14:15:35.660 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:35.717 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:35.819 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:35.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:35.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:35.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:35.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:36.040 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:36.086 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.133 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:36.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:36.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:15:36.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:36 smithi003 ceph-mon[29576]: 9.d scrub starts 2024-03-21T14:15:36.178 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:36 smithi003 ceph-mon[40510]: 9.d scrub starts 2024-03-21T14:15:36.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:36.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:36.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:36.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:36 smithi118 ceph-mon[33862]: 9.d scrub starts 2024-03-21T14:15:36.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:36.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:36.436 INFO:teuthology.orchestra.run.smithi003.stdout:get_pg_num_history pg_num_history pg_num_history(e277 pg_nums {1={19=1,268=11},2={63=3},3={64=32},4={66=32},5={68=32},6={70=32},8={195=26},9={244=16}} deleted_pools 243,8) 2024-03-21T14:15:36.436 INFO:teuthology.orchestra.run.smithi003.stdout:Importing pgid 1.8 2024-03-21T14:15:36.436 INFO:teuthology.orchestra.run.smithi003.stdout:Write #1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head# 2024-03-21T14:15:36.436 INFO:teuthology.orchestra.run.smithi003.stdout:snapset 0=[]:{} 2024-03-21T14:15:36.436 INFO:teuthology.orchestra.run.smithi003.stdout:write_pg epoch 274 info 1.8( v 167'130 (0'0,167'130] local-lis/les=273/274 n=0 ec=268/19 lis/c=273/273 les/c/f=274/274/0 sis=273) 2024-03-21T14:15:36.436 INFO:teuthology.orchestra.run.smithi003.stdout:Import successful 2024-03-21T14:15:36.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:36.548 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:36.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.690 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:36.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:15:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:15:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:15:36.791 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:36.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:36.929 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:36.963 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:36.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:37.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:37.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:37.244 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:37.284 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:37 smithi118 ceph-mon[33862]: pgmap v586: 158 pgs: 2 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 96 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 901 op/s; 25495/246888 objects degraded (10.327%) 2024-03-21T14:15:37.284 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:37 smithi118 ceph-mon[33862]: 9.e scrub starts 2024-03-21T14:15:37.284 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:37 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:15:37.284 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:37 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:15:37.284 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:37 smithi118 ceph-mon[33862]: 9.f scrub starts 2024-03-21T14:15:37.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:15:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:15:37.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:37.413 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:37.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[29576]: pgmap v586: 158 pgs: 2 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 96 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 901 op/s; 25495/246888 objects degraded (10.327%) 2024-03-21T14:15:37.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[29576]: 9.e scrub starts 2024-03-21T14:15:37.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:15:37.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:15:37.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[29576]: 9.f scrub starts 2024-03-21T14:15:37.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[40510]: pgmap v586: 158 pgs: 2 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 96 active+clean; 5.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 767 B/s rd, 0 B/s wr, 901 op/s; 25495/246888 objects degraded (10.327%) 2024-03-21T14:15:37.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[40510]: 9.e scrub starts 2024-03-21T14:15:37.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:15:37.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:15:37.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:37 smithi003 ceph-mon[40510]: 9.f scrub starts 2024-03-21T14:15:37.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.549 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:37.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.699 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:37.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:37.825 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:37.926 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:37.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:37.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:38.004 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:15:38.004 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:15:38.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:38.084 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 9.d scrub ok 2024-03-21T14:15:38.084 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 2.0 deep-scrub starts 2024-03-21T14:15:38.084 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 2.0 deep-scrub ok 2024-03-21T14:15:38.084 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:38.084 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 3.1b deep-scrub starts 2024-03-21T14:15:38.085 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 3.1b deep-scrub ok 2024-03-21T14:15:38.085 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 9.e scrub ok 2024-03-21T14:15:38.085 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:15:38.085 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:38 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:15:38.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:38.199 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:38.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:38.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:38.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 9.d scrub ok 2024-03-21T14:15:38.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 2.0 deep-scrub starts 2024-03-21T14:15:38.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 2.0 deep-scrub ok 2024-03-21T14:15:38.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:38.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 3.1b deep-scrub starts 2024-03-21T14:15:38.337 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 3.1b deep-scrub ok 2024-03-21T14:15:38.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 9.e scrub ok 2024-03-21T14:15:38.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:15:38.338 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:15:38.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 9.d scrub ok 2024-03-21T14:15:38.338 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 2.0 deep-scrub starts 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 2.0 deep-scrub ok 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 3.1b deep-scrub starts 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 3.1b deep-scrub ok 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 9.e scrub ok 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:15:38.339 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:38 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:15:38.372 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:38.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:38.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:38.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.647 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:38.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:38.744 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T14:15:38.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:38.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:38.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/exp.1.8.2 2024-03-21T14:15:38.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:38.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:38.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:39.033 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:39.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:39.165 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:39.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:39.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:39.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: pgmap v587: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.76k op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:39.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 5.15 deep-scrub starts 2024-03-21T14:15:39.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 5.15 deep-scrub ok 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 9.f scrub ok 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:15:39.305 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:39 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:15:39.322 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:39.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.367 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:39.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:39.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:39.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: pgmap v587: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.76k op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 5.15 deep-scrub starts 2024-03-21T14:15:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 5.15 deep-scrub ok 2024-03-21T14:15:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:15:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:15:39.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:15:39.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:15:39.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 9.f scrub ok 2024-03-21T14:15:39.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:15:39.506 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:15:39.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: pgmap v587: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.76k op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:39.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 5.15 deep-scrub starts 2024-03-21T14:15:39.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 5.15 deep-scrub ok 2024-03-21T14:15:39.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:15:39.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:15:39.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:15:39.507 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:15:39.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 9.f scrub ok 2024-03-21T14:15:39.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:15:39.508 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:39 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:15:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.520 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:39.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:39.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:39.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:39.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:39.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:39.794 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.807 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:39.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:39.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:39.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:39.938 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:39.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:40.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:40.078 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:40.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.217 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:40.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:40.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:15:40.305 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 23336/226176 objects degraded (10.318%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:15:40.306 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:40 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 23336/226176 objects degraded (10.318%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:40.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:40.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:40 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:15:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:40 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:15:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:40 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:15:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:40 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:15:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:40 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 23336/226176 objects degraded (10.318%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:40.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:40.488 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:40.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:40.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:40.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:40.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:40.763 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:40.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:40.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:40.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:40.951 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:15:40.951 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:15:40.951 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:15:40.952 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:15:40.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:41.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:41.131 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:41.181 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:41.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[29576]: pgmap v588: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.76k op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:15:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:15:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4079383072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:15:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:15:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[40510]: pgmap v588: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.76k op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:15:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:15:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4079383072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:15:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:41 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:15:41.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:41.327 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:41.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:41.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:41.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:41 smithi118 ceph-mon[33862]: pgmap v588: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.76k op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:41.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:41 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:15:41.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:41 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:15:41.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4079383072' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:41.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:41 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:15:41.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:41 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:15:41.464 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:41.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:41.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:41.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:41.743 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:41.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:15:41.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:15:41.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:41.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:41.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:41.986 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:42.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:42.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:42.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:42.157 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:42.250 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154023 7.7 7:ec8384af:::benchmark_data_smithi003_80362_object33006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 531.62642888899995, 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.14729518, 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.251 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154023 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.252 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621103+0000", 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621105+0000", 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.155e-06 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621110+0000", 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5790000000000003e-06 2024-03-21T14:15:42.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621119+0000", 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1670000000000005e-06 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.621125+0000", 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9370000000000001e-06 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665294+0000", 2024-03-21T14:15:42.254 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044168488999999998 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665370+0000", 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6055000000000003e-05 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.665441+0000", 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1031000000000005e-05 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.668869+0000", 2024-03-21T14:15:42.255 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034284250000000001 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.767406+0000", 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.098536532999999996 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768291+0000", 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00088539800000000002 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768347+0000", 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5769000000000001e-05 2024-03-21T14:15:42.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:50.768399+0000", 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1641999999999998e-05 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154716 7.7 7:eb3d6bd8:::benchmark_data_smithi003_80362_object33699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 530.47369885299997, 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.135279182, 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.257 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154716 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773833+0000", 2024-03-21T14:15:42.258 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773834+0000", 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0399999999999995e-07 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773838+0000", 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9910000000000004e-06 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773846+0000", 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2209999999999997e-06 2024-03-21T14:15:42.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.773853+0000", 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3890000000000002e-06 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814039+0000", 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040186573000000003 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814159+0000", 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012002600000000001 2024-03-21T14:15:42.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.814282+0000", 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000123091 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817930+0000", 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036476630000000002 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831508+0000", 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013578365 2024-03-21T14:15:42.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.908985+0000", 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077476415000000007 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909037+0000", 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1929999999999999e-05 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909113+0000", 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5814000000000003e-05 2024-03-21T14:15:42.262 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154717 7.7 7:e048d7ab:::benchmark_data_smithi003_80362_object33700:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 530.47343815800002, 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13536177999999999, 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.263 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154717 2024-03-21T14:15:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.265 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774094+0000", 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.267 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774095+0000", 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3600000000000003e-07 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774098+0000", 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1640000000000001e-06 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774106+0000", 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3059999999999998e-06 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.268 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774112+0000", 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5740000000000003e-06 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816429+0000", 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.042316843 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816497+0000", 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8421000000000001e-05 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.269 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.816574+0000", 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6460999999999999e-05 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820900+0000", 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043267039999999998 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.831554+0000", 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010653673000000001 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.270 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909389+0000", 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.077835369000000001 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909424+0000", 2024-03-21T14:15:42.271 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4275e-05 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909456+0000", 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2254e-05 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:154718 7.7 7:eaddd76b:::benchmark_data_smithi003_80362_object33701:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 530.47269321900001, 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13475282299999999, 2024-03-21T14:15:42.272 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 154718 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.273 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774839+0000", 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774840+0000", 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7000000000000004e-07 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774844+0000", 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.783e-06 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.274 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774853+0000", 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5459999999999997e-06 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.774859+0000", 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2639999999999997e-06 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.817968+0000", 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.043108377000000003 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.275 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818046+0000", 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8118e-05 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.818119+0000", 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2712000000000004e-05 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.820930+0000", 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028118449999999999 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.276 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909502+0000", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.088571516000000003 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909536+0000", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3813999999999998e-05 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909563+0000", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7297e-05 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:06:51.909592+0000", 2024-03-21T14:15:42.277 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8881e-05 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221486 7.7 7:ee81363c:::benchmark_data_smithi003_80362_object100469:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.15311839999998, 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1032354099999999, 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.278 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221486 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094414+0000", 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.279 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094415+0000", 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.164e-06 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094423+0000", 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3129999999999999e-06 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094436+0000", 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2646999999999999e-05 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.280 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094445+0000", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7069999999999998e-06 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094500+0000", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4882000000000001e-05 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.094509+0000", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9369999999999993e-06 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.281 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192346+0000", 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0978373850000001 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192463+0000", 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011746799999999999 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.192545+0000", 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2050000000000002e-05 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.282 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197520+0000", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0049742279999999998 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197578+0000", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8791e-05 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197589+0000", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0293e-05 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197620+0000", 2024-03-21T14:15:42.283 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1065000000000002e-05 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197649+0000", 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9479999999999999e-05 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221488 7.7 7:e6035441:::benchmark_data_smithi003_80362_object100471:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.151840037, 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.107609375, 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.284 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221488 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095692+0000", 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.285 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095693+0000", 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5999999999999991e-07 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095698+0000", 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0239999999999997e-06 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095711+0000", 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2445e-05 2024-03-21T14:15:42.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095719+0000", 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0730000000000005e-06 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095770+0000", 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1650000000000002e-05 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.095777+0000", 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2169999999999999e-06 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.287 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194772+0000", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.09899547 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194854+0000", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2081000000000002e-05 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.194929+0000", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5067999999999993e-05 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199113+0000", 2024-03-21T14:15:42.288 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041835680000000004 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203213+0000", 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040997869999999997 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203248+0000", 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5432999999999998e-05 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203277+0000", 2024-03-21T14:15:42.289 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8952e-05 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203302+0000", 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4647e-05 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221491 7.7 7:eaaedede:::benchmark_data_smithi003_80362_object100474:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.14903018500002, 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1086137819999999, 2024-03-21T14:15:42.290 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221491 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098502+0000", 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098503+0000", 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3090000000000001e-06 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098508+0000", 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1899999999999997e-06 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098518+0000", 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0258000000000001e-05 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.292 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098525+0000", 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7229999999999998e-06 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098580+0000", 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5015000000000001e-05 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.098590+0000", 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.061e-05 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.293 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196184+0000", 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0975940799999999 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196262+0000", 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7281999999999996e-05 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.196328+0000", 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6224999999999997e-05 2024-03-21T14:15:42.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199116+0000", 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027880639999999998 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203355+0000", 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042392979999999999 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207064+0000", 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0037090000000000001 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.295 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207093+0000", 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8351999999999999e-05 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207116+0000", 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3376000000000001e-05 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221516 7.7 7:ee74cf53:::benchmark_data_smithi003_80362_object100499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.296 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.12956223100002, 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.095088184, 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221516 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117970+0000", 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117972+0000", 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6029999999999999e-06 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117979+0000", 2024-03-21T14:15:42.298 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6650000000000003e-06 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.117994+0000", 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5021999999999999e-05 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118002+0000", 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6129999999999998e-06 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118062+0000", 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9768000000000001e-05 2024-03-21T14:15:42.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.118072+0000", 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0028e-05 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197714+0000", 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0796421430000001 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197781+0000", 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6649000000000005e-05 2024-03-21T14:15:42.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.197849+0000", 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8150000000000003e-05 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203310+0000", 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054610889999999997 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203389+0000", 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8960000000000003e-05 2024-03-21T14:15:42.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.212981+0000", 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095917339999999993 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213031+0000", 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0633999999999997e-05 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213058+0000", 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7126000000000001e-05 2024-03-21T14:15:42.302 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221558 7.7 7:e23945db:::benchmark_data_smithi003_80362_object100541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.09226242800003, 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0579410659999999, 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221558 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.303 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155270+0000", 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155271+0000", 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-03-21T14:15:42.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155279+0000", 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9899999999999997e-06 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155295+0000", 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5366000000000002e-05 2024-03-21T14:15:42.305 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155303+0000", 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9030000000000004e-06 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155469+0000", 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016613000000000001 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.155488+0000", 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9151e-05 2024-03-21T14:15:42.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199159+0000", 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0436712960000001 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199258+0000", 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9086999999999995e-05 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.199325+0000", 2024-03-21T14:15:42.307 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6824999999999998e-05 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203317+0000", 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003992125 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213108+0000", 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.009791213 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213160+0000", 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1712000000000002e-05 2024-03-21T14:15:42.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213189+0000", 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8915999999999999e-05 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213211+0000", 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1837000000000001e-05 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.309 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221566 7.7 7:e351aeff:::benchmark_data_smithi003_80362_object100549:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.08242025800001, 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0550122820000001, 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221566 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.310 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165112+0000", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165113+0000", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1999999999999999e-06 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.311 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165118+0000", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3710000000000002e-06 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165126+0000", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1980000000000001e-06 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165131+0000", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1229999999999999e-06 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165172+0000", 2024-03-21T14:15:42.312 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1094999999999999e-05 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.165178+0000", 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6659999999999996e-06 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200612+0000", 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.035434781 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200723+0000", 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011056 2024-03-21T14:15:42.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.200803+0000", 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9759000000000007e-05 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203324+0000", 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002521129 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213257+0000", 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0099332840000000006 2024-03-21T14:15:42.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220034+0000", 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0067764390000000004 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220082+0000", 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8179000000000001e-05 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220124+0000", 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2497999999999999e-05 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.315 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221598 7.7 7:e857fd29:::benchmark_data_smithi003_80362_object100581:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 454.01308904699999, 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99124171000000005, 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221598 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.316 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234443+0000", 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.317 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.317 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:42 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:15:42.318 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:42 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:15:42.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234445+0000", 2024-03-21T14:15:42.318 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5460000000000001e-06 2024-03-21T14:15:42.318 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.318 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.318 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.318 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234452+0000", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6590000000000007e-06 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234470+0000", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7876999999999998e-05 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234480+0000", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8770000000000005e-06 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234548+0000", 2024-03-21T14:15:42.319 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7343999999999998e-05 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.234560+0000", 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2707999999999999e-05 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.201983+0000", 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96742289599999998 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.202047+0000", 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3736000000000001e-05 2024-03-21T14:15:42.320 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.202116+0000", 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.902e-05 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203398+0000", 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001282327 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.220150+0000", 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.016752229 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.321 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225577+0000", 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0054261459999999997 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225637+0000", 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0489999999999999e-05 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225685+0000", 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7855000000000002e-05 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.322 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221605 7.7 7:e86450e5:::benchmark_data_smithi003_80362_object100588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.97756428899999, 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96634132500000003, 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221605 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.323 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269968+0000", 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269969+0000", 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2470000000000001e-06 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.324 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269975+0000", 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7320000000000003e-06 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269990+0000", 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4745e-05 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.269999+0000", 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0990000000000005e-06 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.325 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.270076+0000", 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7044000000000003e-05 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.270085+0000", 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.55e-06 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203422+0000", 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.93333645799999998 2024-03-21T14:15:42.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203489+0000", 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7290000000000001e-05 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203555+0000", 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6123000000000004e-05 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207125+0000", 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035693439999999999 2024-03-21T14:15:42.327 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225732+0000", 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018607716999999999 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236215+0000", 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010482131 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236265+0000", 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0158000000000003e-05 2024-03-21T14:15:42.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236309+0000", 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4687000000000001e-05 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221618 7.7 7:e934ecf7:::benchmark_data_smithi003_80362_object100601:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.964356409, 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95328595999999999, 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.329 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221618 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283176+0000", 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283177+0000", 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.423e-06 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283183+0000", 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5829999999999997e-06 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283199+0000", 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5786000000000001e-05 2024-03-21T14:15:42.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283207+0000", 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6370000000000005e-06 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283266+0000", 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8603000000000001e-05 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.283274+0000", 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4249999999999998e-06 2024-03-21T14:15:42.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204661+0000", 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.92138625600000001 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204729+0000", 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7949999999999998e-05 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.204796+0000", 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7284000000000002e-05 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.333 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209602+0000", 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00480606 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227784+0000", 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018181953000000001 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236361+0000", 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0085767780000000002 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.334 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236418+0000", 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7538999999999999e-05 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236462+0000", 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3683000000000002e-05 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221625 7.7 7:e490a489:::benchmark_data_smithi003_80362_object100608:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.95949621, 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94858037900000003, 2024-03-21T14:15:42.335 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221625 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288036+0000", 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288038+0000", 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.511e-06 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288044+0000", 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0360000000000003e-06 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288060+0000", 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6498e-05 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.337 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288068+0000", 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6079999999999998e-06 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288124+0000", 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6141999999999999e-05 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.288133+0000", 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4350000000000003e-06 2024-03-21T14:15:42.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.205877+0000", 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91774332000000003 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.205944+0000", 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7387999999999999e-05 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.206008+0000", 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4078000000000006e-05 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.339 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209608+0000", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035997080000000001 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227901+0000", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018292940000000001 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236522+0000", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0086215389999999992 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.340 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236574+0000", 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1910000000000003e-05 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236616+0000", 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2265999999999998e-05 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221680 7.7 7:ec92397d:::benchmark_data_smithi003_80362_object100663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.89540478999999, 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.884639709, 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.341 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221680 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.342 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.343 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352128+0000", 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352130+0000", 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0250000000000001e-06 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352138+0000", 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5930000000000006e-06 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.344 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352157+0000", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9318e-05 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352168+0000", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0098000000000001e-05 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352233+0000", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5362000000000002e-05 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.352244+0000", 2024-03-21T14:15:42.345 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0985000000000001e-05 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207147+0000", 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85490348999999999 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207222+0000", 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4796999999999996e-05 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.207287+0000", 2024-03-21T14:15:42.346 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4978e-05 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.212923+0000", 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056357789999999996 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227945+0000", 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.015022064 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.347 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236673+0000", 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087278890000000008 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236725+0000", 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2469999999999997e-05 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236767+0000", 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1860999999999998e-05 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.348 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221689 7.7 7:e59419c3:::benchmark_data_smithi003_80362_object100672:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.88192268, 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87131467699999998, 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221689 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.349 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365610+0000", 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365611+0000", 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2020000000000001e-06 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.350 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365616+0000", 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3789999999999997e-06 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365628+0000", 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1807000000000001e-05 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365635+0000", 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0899999999999999e-06 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.351 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365698+0000", 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2691e-05 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.365709+0000", 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0706999999999999e-05 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208374+0000", 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84266594399999994 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.352 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208442+0000", 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7799999999999995e-05 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.208504+0000", 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1441999999999999e-05 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213123+0000", 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046188710000000001 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.353 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.227986+0000", 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.01486326 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236829+0000", 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0088433829999999998 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236880+0000", 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0977999999999999e-05 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.354 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236924+0000", 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4122999999999998e-05 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221705 7.7 7:e134962a:::benchmark_data_smithi003_80362_object100688:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.86122683999997, 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85076709500000003, 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.355 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221705 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386305+0000", 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.356 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386307+0000", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.336e-06 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386313+0000", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8209999999999998e-06 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386324+0000", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0895e-05 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.357 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386332+0000", 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2740000000000007e-06 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386407+0000", 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5479e-05 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.386417+0000", 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3039999999999994e-06 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.358 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209643+0000", 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82322670899999995 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209715+0000", 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1707999999999999e-05 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.209777+0000", 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2005e-05 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.359 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213129+0000", 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033520189999999999 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.228015+0000", 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014885631 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236988+0000", 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0089736309999999993 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.360 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237035+0000", 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7126999999999998e-05 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237073+0000", 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7156000000000001e-05 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221707 7.7 7:ee1b885c:::benchmark_data_smithi003_80362_object100690:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.85890427200002, 2024-03-21T14:15:42.361 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84853804399999999, 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221707 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.362 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388628+0000", 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388630+0000", 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4929999999999999e-06 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388636+0000", 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.207e-06 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.363 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388650+0000", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4283e-05 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388659+0000", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5290000000000008e-06 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388717+0000", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8374e-05 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.364 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.388729+0000", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1666e-05 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.210963+0000", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.82223472099999995 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.211082+0000", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011889500000000001 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.211202+0000", 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119864 2024-03-21T14:15:42.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213270+0000", 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020684119999999999 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.228042+0000", 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.014771993000000001 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.366 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237112+0000", 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090695329999999994 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237143+0000", 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0642999999999998e-05 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237166+0000", 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3431e-05 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221710 7.7 7:e7a06586:::benchmark_data_smithi003_80362_object100693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.85355458200002, 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.84331796999999997, 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221710 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.368 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393978+0000", 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393979+0000", 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.17e-06 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.369 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393985+0000", 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8660000000000002e-06 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.393997+0000", 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2218999999999999e-05 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394006+0000", 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0899999999999994e-06 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.370 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394056+0000", 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0389000000000002e-05 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.394063+0000", 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7619999999999998e-06 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213298+0000", 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81923456500000003 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.371 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213370+0000", 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1940999999999994e-05 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.213432+0000", 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2206999999999996e-05 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.218798+0000", 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005365791 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.372 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.231526+0000", 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012728385 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237210+0000", 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056836960000000002 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237258+0000", 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8300000000000002e-05 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237296+0000", 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7589000000000002e-05 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221712 7.7 7:efee874f:::benchmark_data_smithi003_80362_object100695:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 453.84873251599998, 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.83858521399999997, 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:15:42.374 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221712 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398800+0000", 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:42.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398801+0000", 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3e-06 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398810+0000", 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7399999999999993e-06 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398823+0000", 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3444e-05 2024-03-21T14:15:42.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398832+0000", 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0459999999999994e-06 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398880+0000", 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7713000000000002e-05 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:08.398886+0000", 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2949999999999999e-06 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.377 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215610+0000", 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.81672409700000004 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215726+0000", 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115151 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.215844+0000", 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011856100000000001 2024-03-21T14:15:42.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.219013+0000", 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003168424 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.231671+0000", 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012658584000000001 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237340+0000", 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0056683879999999999 2024-03-21T14:15:42.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237366+0000", 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6338000000000001e-05 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.237385+0000", 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9133000000000001e-05 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:42.380 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:42.381 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:15:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:42.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:42 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:15:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:42 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:15:42.382 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:42 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:15:42.383 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:42 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:15:42.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:42.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:42.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:42.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:42.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.661 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:42.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:42.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:42.795 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:42.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:42.897 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:42.934 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:42.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:43.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:43.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:43.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:43.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:43.202 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:43.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:43.237 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:15:43.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:43 smithi003 ceph-mon[29576]: pgmap v589: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 602 B/s rd, 0 B/s wr, 812 op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:43.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:43 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:15:43.285 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:43 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:15:43.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:43 smithi003 ceph-mon[40510]: pgmap v589: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 602 B/s rd, 0 B/s wr, 812 op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:43.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:43 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:15:43.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:43 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:15:43.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:43.314 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:43 smithi118 ceph-mon[33862]: pgmap v589: 158 pgs: 3 active+clean+scrubbing, 21 active+undersized+degraded, 39 active+undersized, 95 active+clean; 4.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 602 B/s rd, 0 B/s wr, 812 op/s; 23336/226176 objects degraded (10.318%) 2024-03-21T14:15:43.314 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:43 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:15:43.314 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:43 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:15:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:43.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:43.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:43.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:43.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:43.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:43.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:43.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:43.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:43.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:15:43.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:43.875 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:15:43.875 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:15:43.876 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-21T14:15:43.876 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T14:15:43.876 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T14:15:43.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-21T14:15:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:43.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:43.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:44.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:44.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:44.062 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:15:44.078 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:44 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:15:44.079 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:44 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:15:44.079 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:44 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3075820826' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:44.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:44.194 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:44.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:44 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:15:44.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:44 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:15:44.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:44 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3075820826' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:44 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:15:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:44 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:15:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:44 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3075820826' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:15:44.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:44.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.331 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:44.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:44.467 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:44.569 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:44.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:44.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:44.746 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:44.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:44.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:44.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:44.948 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:44.986 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.033 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:45.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:45.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:45 smithi118 ceph-mon[33862]: pgmap v590: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 21293/206496 objects degraded (10.312%) 2024-03-21T14:15:45.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:45 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:45.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3075820826' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:15:45.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:45 smithi118 ceph-mon[33862]: osdmap e291: 8 total, 7 up, 8 in 2024-03-21T14:15:45.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:45 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 21293/206496 objects degraded (10.312%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:45.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:45 smithi118 ceph-mon[33862]: osdmap e292: 8 total, 7 up, 8 in 2024-03-21T14:15:45.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[29576]: pgmap v590: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 21293/206496 objects degraded (10.312%) 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3075820826' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[29576]: osdmap e291: 8 total, 7 up, 8 in 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 21293/206496 objects degraded (10.312%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[29576]: osdmap e292: 8 total, 7 up, 8 in 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[40510]: pgmap v590: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 21293/206496 objects degraded (10.312%) 2024-03-21T14:15:45.135 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:45.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3075820826' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:15:45.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[40510]: osdmap e291: 8 total, 7 up, 8 in 2024-03-21T14:15:45.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 21293/206496 objects degraded (10.312%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:45.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:45 smithi003 ceph-mon[40510]: osdmap e292: 8 total, 7 up, 8 in 2024-03-21T14:15:45.173 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:45.275 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:45.309 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:45.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:45.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:45.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:45.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:45.575 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:45.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:45.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:45.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:45.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:45.856 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:45.958 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:45.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:45.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:46.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:46.130 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:46.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:46.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:46.274 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:46.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:46.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:46.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:46.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.539 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:46.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:46.562 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:15:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:15:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:15:46.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:46.681 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:46.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:46.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:46.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:15:46.909 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:46.943 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:46.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:47.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:47.081 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:47.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:47.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:47 smithi003 ceph-mon[29576]: pgmap v593: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 820 op/s; 21293/206496 objects degraded (10.312%) 2024-03-21T14:15:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:47.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:47 smithi003 ceph-mon[40510]: pgmap v593: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 820 op/s; 21293/206496 objects degraded (10.312%) 2024-03-21T14:15:47.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:47.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:47.345 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:47.390 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:47 smithi118 ceph-mon[33862]: pgmap v593: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 4.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 820 op/s; 21293/206496 objects degraded (10.312%) 2024-03-21T14:15:47.390 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:15:47.390 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:15:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:15:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219273 9.9 9:95d2d9a1:::benchmark_data_smithi003_135814_object78539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.390 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.214301+0000", 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.010794332, 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010826199, 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219273 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.391 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214301+0000", 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214301+0000", 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214302+0000", 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5600000000000005e-07 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.392 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214306+0000", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3730000000000003e-06 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214314+0000", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9689999999999994e-06 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214322+0000", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1790000000000002e-06 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.216593+0000", 2024-03-21T14:15:47.393 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022715309999999998 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.216695+0000", 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010204099999999999 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.216865+0000", 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016962600000000001 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.25232.0:219272 9.1 e292/244 9:84d947c0:::benchmark_data_smithi003_135814_object78538:head v 292'13650, mlcod=292'13648)", 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.214504+0000", 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.01059216, 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.010706189, 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214504+0000", 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.395 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214504+0000", 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214504+0000", 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1200000000000003e-07 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214509+0000", 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5059999999999997e-06 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.396 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214515+0000", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9260000000000003e-06 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214522+0000", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0950000000000001e-06 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.216543+0000", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020211970000000002 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.397 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.219264+0000", 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027214449999999999 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219276 9.3 9:c774966a:::benchmark_data_smithi003_135814_object78542:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.220005+0000", 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0050904110000000004, 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052892160000000002, 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.398 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219276 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220005+0000", 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220005+0000", 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220007+0000", 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3149999999999999e-06 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220011+0000", 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9489999999999998e-06 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220020+0000", 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9630000000000004e-06 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.400 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220026+0000", 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8240000000000002e-06 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220076+0000", 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9672999999999997e-05 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220199+0000", 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012284000000000001 2024-03-21T14:15:47.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,3", 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220315+0000", 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116019 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.224931+0000", 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046162349999999998 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.402 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219277 9.9 9:9215c57c:::benchmark_data_smithi003_135814_object78543:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.220197+0000", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.004898375, 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051927099999999997, 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219277 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220197+0000", 2024-03-21T14:15:47.403 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220197+0000", 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220198+0000", 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0500000000000003e-07 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220202+0000", 2024-03-21T14:15:47.404 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6559999999999998e-06 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220209+0000", 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5950000000000001e-06 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220216+0000", 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3820000000000001e-06 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220915+0000", 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069884500000000002 2024-03-21T14:15:47.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221013+0000", 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8301000000000006e-05 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221121+0000", 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010822799999999999 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219278 9.9 9.6165beb9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.221058+0000", 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0040379989999999996, 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044126189999999996, 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.407 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219278 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221058+0000", 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.409 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221058+0000", 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221059+0000", 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0899999999999999e-06 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221064+0000", 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3449999999999996e-06 2024-03-21T14:15:47.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221076+0000", 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1438e-05 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221084+0000", 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6780000000000006e-06 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.411 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.25232.0:219273 9.9 e292/244)", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.221275+0000", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0038209120000000001, 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042532409999999996, 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221275+0000", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221275+0000", 2024-03-21T14:15:47.412 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221275+0000", 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.82e-07 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221279+0000", 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2590000000000001e-06 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221288+0000", 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4779999999999997e-06 2024-03-21T14:15:47.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.221295+0000", 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5749999999999997e-06 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.25232.0:219279 9.1 e292/244 9:88fe64e1:::benchmark_data_smithi003_135814_object78545:head v 292'13651, mlcod=292'13648)", 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222476+0000", 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.002619845, 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031119530000000002, 2024-03-21T14:15:47.414 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222476+0000", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222476+0000", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222476+0000", 2024-03-21T14:15:47.415 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3300000000000002e-07 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222481+0000", 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3209999999999998e-06 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222489+0000", 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5150000000000006e-06 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222496+0000", 2024-03-21T14:15:47.416 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7229999999999998e-06 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223280+0000", 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078432100000000004 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223297+0000", 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7189000000000001e-05 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.417 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.25232.0:219269 9.9 e292/244)", 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222524+0000", 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0025717819999999999, 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031371010000000002, 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.418 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.418 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:15:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222524+0000", 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222524+0000", 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222524+0000", 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6300000000000001e-07 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222527+0000", 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5550000000000001e-06 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222533+0000", 2024-03-21T14:15:47.420 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8459999999999996e-06 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222537+0000", 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5109999999999997e-06 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.25232.0:219273 9.9 e292/244)", 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222549+0000", 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0025468869999999998, 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031657859999999999, 2024-03-21T14:15:47.421 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222549+0000", 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222549+0000", 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.422 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222549+0000", 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6199999999999999e-07 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222551+0000", 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7409999999999999e-06 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222555+0000", 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0940000000000001e-06 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.423 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222558+0000", 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6109999999999998e-06 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.25232.0:219281 9.a e292/244 9:5e19477b:::benchmark_data_smithi003_135814_object78547:head v 292'13957, mlcod=292'13955)", 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222563+0000", 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.002532435, 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032073230000000002, 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.424 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222563+0000", 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222563+0000", 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222563+0000", 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7800000000000001e-07 2024-03-21T14:15:47.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222566+0000", 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.954e-06 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222571+0000", 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1790000000000001e-06 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222574+0000", 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7790000000000002e-06 2024-03-21T14:15:47.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225689+0000", 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003114623 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225718+0000", 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8821000000000001e-05 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.427 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.25232.0:219283 9.8 e292/244 9:1e960916:::benchmark_data_smithi003_135814_object78549:head v 292'13723, mlcod=292'13721)", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222587+0000", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0025085239999999998, 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032173000000000002, 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222587+0000", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222587+0000", 2024-03-21T14:15:47.428 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222587+0000", 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0599999999999999e-07 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222591+0000", 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1920000000000001e-06 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222595+0000", 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1649999999999999e-06 2024-03-21T14:15:47.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222598+0000", 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5860000000000001e-06 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223963+0000", 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013642389999999999 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223982+0000", 2024-03-21T14:15:47.430 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9496000000000001e-05 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219280 9.9 9.61200949 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222618+0000", 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0024781669999999999, 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032201310000000002, 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.431 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219280 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222618+0000", 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222618+0000", 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.432 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222618+0000", 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1200000000000001e-07 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222622+0000", 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.117e-06 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222631+0000", 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0739999999999999e-06 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.433 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222636+0000", 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3229999999999997e-06 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219282 9.e 9:7d0899b9:::benchmark_data_smithi003_135814_object78548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.222644+0000", 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0024521019999999998, 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032278490000000001, 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.434 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219282 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222644+0000", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222644+0000", 2024-03-21T14:15:47.435 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222644+0000", 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6199999999999999e-07 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222646+0000", 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3599999999999999e-06 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222651+0000", 2024-03-21T14:15:47.436 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5129999999999998e-06 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222655+0000", 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3189999999999996e-06 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222731+0000", 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5957999999999993e-05 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222844+0000", 2024-03-21T14:15:47.437 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112589 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,4", 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.222953+0000", 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109145 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225565+0000", 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026123840000000001 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.438 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.25232.0:219277 9.9 e292/244)", 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.223819+0000", 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0012762229999999999, 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021067059999999999, 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223819+0000", 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.439 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223819+0000", 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223820+0000", 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6800000000000001e-07 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223823+0000", 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.433e-06 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.440 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223834+0000", 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.048e-05 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.223840+0000", 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0220000000000001e-06 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.25232.0:219285 9.1 e292/244)", 2024-03-21T14:15:47.441 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.225220+0000", 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4294967295.999876, 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00073801400000000005, 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225220+0000", 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225220+0000", 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.442 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225220+0000", 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6899999999999998e-07 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225225+0000", 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5040000000000004e-06 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225232+0000", 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2400000000000001e-06 2024-03-21T14:15:47.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225238+0000", 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0240000000000003e-06 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.25232.0:219277 9.9 e292/244)", 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.225288+0000", 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4294967295.9998074, 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000698451, 2024-03-21T14:15:47.444 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225288+0000", 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225288+0000", 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.445 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225288+0000", 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5800000000000001e-07 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225290+0000", 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.821e-06 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225295+0000", 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2189999999999998e-06 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.446 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225298+0000", 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7969999999999998e-06 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.25232.0:219286 9.7 e292/278 9:ebcf579b:::benchmark_data_smithi003_135814_object78552:head v 292'13770, mlcod=292'13768)", 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.225578+0000", 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4294967295.9995179, 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00045540899999999999, 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.447 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225578+0000", 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225578+0000", 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225578+0000", 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9800000000000004e-07 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.448 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225583+0000", 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1769999999999997e-06 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225590+0000", 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4559999999999997e-06 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225595+0000", 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5450000000000003e-06 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225638+0000", 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2518000000000002e-05 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225659+0000", 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0805000000000002e-05 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219269 9.9 9:940b9054:::benchmark_data_smithi003_135814_object78535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.450 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.213869+0000", 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.011227097, 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012199008000000001, 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219269 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.213869+0000", 2024-03-21T14:15:47.451 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.213869+0000", 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.213869+0000", 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9800000000000003e-07 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.213872+0000", 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9349999999999999e-06 2024-03-21T14:15:47.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.213878+0000", 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1680000000000001e-06 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.213884+0000", 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2170000000000002e-06 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214008+0000", 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012420899999999999 2024-03-21T14:15:47.453 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214115+0000", 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010658 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.214226+0000", 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000111141 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.220877+0000", 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0066512990000000003 2024-03-21T14:15:47.454 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:219290 9.d 9.93e0012d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:46.225850+0000", 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 4294967295.9992456, 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026390499999999999, 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219290 2024-03-21T14:15:47.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225850+0000", 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225850+0000", 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225851+0000", 2024-03-21T14:15:47.456 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1200000000000001e-06 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225855+0000", 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2459999999999997e-06 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225863+0000", 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8609999999999998e-06 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:46.225870+0000", 2024-03-21T14:15:47.457 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5080000000000002e-06 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 19 2024-03-21T14:15:47.458 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:15:47.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:47.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:47.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:47.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:47.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.614 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:47.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:47.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:47.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:47.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:47.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:47.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:48.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:48.060 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:48.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:48.090 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:15:48.090 INFO:teuthology.orchestra.run.smithi003.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-21T14:15:48.090 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:15:48.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:48.185 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:48.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:48 smithi003 ceph-mon[29576]: pgmap v594: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:48 smithi003 ceph-mon[40510]: pgmap v594: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:48.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:48.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:48.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:48 smithi118 ceph-mon[33862]: pgmap v594: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:48.462 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:48.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:48.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:48.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:48.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:48.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:48.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:48.886 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:48.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:48.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:49.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:49.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:49.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:49.168 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:49.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.304 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:49.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:49.306 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:49.362 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:15:49.382 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:15:49.402 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:15:49.405 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:49.423 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:15:49.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:49.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:49.460 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:15:49.497 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:15:49.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:49.538 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:15:49.538 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1181853323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:49.539 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1181853323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:15:49.539 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:15:49.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:49.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:49.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:49.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1181853323' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:49.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:49.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:49.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:49.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:49.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:49.971 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:49.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:50.073 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:50.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:50.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:50.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:50.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:50.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:15:50.241 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:15:50.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:50.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:50.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:50.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:50 smithi003 ceph-mon[29576]: pgmap v595: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:50 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 19037/184938 objects degraded (10.294%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:50 smithi003 ceph-mon[40510]: pgmap v595: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:50 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 19037/184938 objects degraded (10.294%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.512 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:50.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:50.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:50.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:50 smithi118 ceph-mon[33862]: pgmap v595: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:50 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 19037/184938 objects degraded (10.294%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:50.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:50.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:50.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:50.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:50.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:50.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:51.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:51.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:51.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:51.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:51.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:51.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:51.304 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:51.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:51.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:51.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:51.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:51.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:51.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:51.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:51.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:51.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:51.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:51.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:52.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:52.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:52.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:52.127 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:52.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:52.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:15:52.267 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:52.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:52.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.404 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:52.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:52.505 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:52.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:52.554 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:15:52.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:52.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:52.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:52.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:52.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:52.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:52.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:52 smithi003 ceph-mon[29576]: pgmap v596: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:52.916 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:52 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/958983124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:52.916 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:52 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:52.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:52 smithi003 ceph-mon[40510]: pgmap v596: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:52.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:52 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/958983124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:52.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:52 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:52 smithi118 ceph-mon[33862]: pgmap v596: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 898 op/s; 19037/184938 objects degraded (10.294%) 2024-03-21T14:15:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:52 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/958983124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:52 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:15:52.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:52.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:53.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:53.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:53.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:53.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:53.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:53.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.344 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:53.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:53.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:53.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:53.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:53.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:53.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.610 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:53.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:53.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:53 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:53.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:53 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:53.674 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:53 smithi003 ceph-mon[29576]: osdmap e293: 8 total, 7 up, 8 in 2024-03-21T14:15:53.674 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:53 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:53.674 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:53 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:53.674 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:53 smithi003 ceph-mon[40510]: osdmap e293: 8 total, 7 up, 8 in 2024-03-21T14:15:53.711 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:53.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:53.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:53.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:53.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:53.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:53 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:15:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:53 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:15:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:53 smithi118 ceph-mon[33862]: osdmap e293: 8 total, 7 up, 8 in 2024-03-21T14:15:53.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:54.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:54.036 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:15:54.036 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:15:54.036 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-21T14:15:54.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd out 6 2024-03-21T14:15:54.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:54.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:54.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:54.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:54.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:54.420 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:54.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:54.559 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:54.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:54.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:54.674 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:54.684 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:54.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:54.820 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:54.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:54 smithi003 ceph-mon[29576]: pgmap v598: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s; 17242/167397 objects degraded (10.300%) 2024-03-21T14:15:54.839 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:54 smithi003 ceph-mon[40510]: pgmap v598: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s; 17242/167397 objects degraded (10.300%) 2024-03-21T14:15:54.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:54 smithi118 ceph-mon[33862]: pgmap v598: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.46k op/s; 17242/167397 objects degraded (10.300%) 2024-03-21T14:15:54.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:54.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:55.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:55.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:55.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:55.215 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:55.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:55.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:15:55.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:55.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:55.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:55 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 17242/167397 objects degraded (10.300%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:55.714 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:55 smithi003 ceph-mon[29576]: osdmap e294: 8 total, 7 up, 8 in 2024-03-21T14:15:55.715 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:55 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 17242/167397 objects degraded (10.300%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:55.715 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:55 smithi003 ceph-mon[40510]: osdmap e294: 8 total, 7 up, 8 in 2024-03-21T14:15:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:55.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:55.817 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:55.853 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:55 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 17242/167397 objects degraded (10.300%), 21 pgs degraded (PG_DEGRADED) 2024-03-21T14:15:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:55 smithi118 ceph-mon[33862]: osdmap e294: 8 total, 7 up, 8 in 2024-03-21T14:15:55.954 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:55.989 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:55.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:56.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:15:56.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:56.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:56.217 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:56.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:56.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:56.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:56.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:56.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:56.605 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:15:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:15:56.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:56.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:56.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:56.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.813 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:56.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:56.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:56 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:15:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:56 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:15:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:56 smithi118 ceph-mon[33862]: pgmap v600: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 731 op/s; 17242/167397 objects degraded (10.300%) 2024-03-21T14:15:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:56 smithi118 ceph-mon[33862]: osdmap e295: 8 total, 7 up, 8 in 2024-03-21T14:15:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:15:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:15:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[29576]: pgmap v600: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 731 op/s; 17242/167397 objects degraded (10.300%) 2024-03-21T14:15:56.933 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[29576]: osdmap e295: 8 total, 7 up, 8 in 2024-03-21T14:15:56.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:15:56.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:15:56.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[40510]: pgmap v600: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.4 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 731 op/s; 17242/167397 objects degraded (10.300%) 2024-03-21T14:15:56.934 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:56 smithi003 ceph-mon[40510]: osdmap e295: 8 total, 7 up, 8 in 2024-03-21T14:15:56.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:56.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:57.030 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:15:57.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:57.075 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:15:57.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:57.109 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:15:57.149 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:15:57.192 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:15:57.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:57.225 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:15:57.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:57.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:57.261 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:15:57.261 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:15:57.261 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:15:57.261 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:15:57.261 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:15:57.262 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:15:57.262 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:15:57.262 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:15:57.262 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:15:57.282 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:15:57.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:57.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:57.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:57.424 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:15:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:15:57.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.512 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:57.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:57.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:57.647 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:57.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2560341892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:57 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:57.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:15:57.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:57.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.813 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:57.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:57.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:57.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:57.948 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:57.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2560341892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:57 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2560341892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:15:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:57 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:15:58.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:58.195 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:58.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:58.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:15:58.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:15:58.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:15:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:58.335 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:58.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:15:58.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:58.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:58.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:58.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:58.473 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a[29572]: 2024-03-21T14:15:58.142+0000 7f8851090700 -1 mon.a@0(leader).osd e295 definitely_dead 0 2024-03-21T14:15:58.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:15:58.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:58.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:58.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:15:58.643 INFO:teuthology.orchestra.run.smithi003.stderr:marked out osd.6. 2024-03-21T14:15:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:58 smithi118 ceph-mon[33862]: pgmap v602: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.02k op/s; 15250/148560 objects degraded (10.265%) 2024-03-21T14:15:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4270515453' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-21T14:15:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:58 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-21T14:15:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:58 smithi118 ceph-mon[33862]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-21T14:15:58.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:58.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[29576]: pgmap v602: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.02k op/s; 15250/148560 objects degraded (10.265%) 2024-03-21T14:15:58.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4270515453' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-21T14:15:58.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-21T14:15:58.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[29576]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-21T14:15:58.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[40510]: pgmap v602: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.02k op/s; 15250/148560 objects degraded (10.265%) 2024-03-21T14:15:58.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4270515453' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-21T14:15:58.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-21T14:15:58.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:58 smithi003 ceph-mon[40510]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-21T14:15:58.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:58.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:58.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:15:58.885 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:58.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:58.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:15:58.987 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:59.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:59.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:59.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:59.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:59.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.301 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:59.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:59.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:15:59.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:15:59.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:15:59.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:15:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:15:59.675 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:15:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.715 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:59.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:15:59.772 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:59 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-21T14:15:59.772 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:59 smithi118 ceph-mon[33862]: osdmap e296: 8 total, 7 up, 7 in 2024-03-21T14:15:59.772 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:59 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 15250/148560 objects degraded (10.265%), 21 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-21T14:15:59.772 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:15:59 smithi118 ceph-mon[33862]: osdmap e297: 8 total, 7 up, 7 in 2024-03-21T14:15:59.817 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:59.861 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:15:59.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:15:59.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:15:59.963 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:15:59.987 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:15:59.987 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:15:59.987 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:15:59.987 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:15:59.987 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:15:59.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-21T14:15:59.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[29576]: osdmap e296: 8 total, 7 up, 7 in 2024-03-21T14:15:59.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 15250/148560 objects degraded (10.265%), 21 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-21T14:15:59.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[29576]: osdmap e297: 8 total, 7 up, 7 in 2024-03-21T14:15:59.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-21T14:15:59.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[40510]: osdmap e296: 8 total, 7 up, 7 in 2024-03-21T14:15:59.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 15250/148560 objects degraded (10.265%), 21 pgs degraded, 60 pgs undersized (PG_DEGRADED) 2024-03-21T14:15:59.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:15:59 smithi003 ceph-mon[40510]: osdmap e297: 8 total, 7 up, 7 in 2024-03-21T14:16:00.003 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:00.105 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:00.139 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:00.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:00.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:00.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:16:00.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:00.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.412 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:00.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:00.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:00.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:00.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:00.655 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:00.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:16:00.692 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:00.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:00.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:00.843 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:00 smithi118 ceph-mon[33862]: pgmap v604: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 15250/148560 objects degraded (10.265%) 2024-03-21T14:16:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:00 smithi118 ceph-mon[33862]: osdmap e298: 8 total, 7 up, 7 in 2024-03-21T14:16:00.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:00.967 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:00.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:00 smithi003 ceph-mon[29576]: pgmap v604: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 15250/148560 objects degraded (10.265%) 2024-03-21T14:16:00.987 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:00 smithi003 ceph-mon[29576]: osdmap e298: 8 total, 7 up, 7 in 2024-03-21T14:16:00.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:00 smithi003 ceph-mon[40510]: pgmap v604: 158 pgs: 21 active+undersized+degraded, 39 active+undersized, 98 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 15250/148560 objects degraded (10.265%) 2024-03-21T14:16:00.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:00 smithi003 ceph-mon[40510]: osdmap e298: 8 total, 7 up, 7 in 2024-03-21T14:16:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:00.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:01.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:01.109 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:16:01.136 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:01.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:01.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:01.276 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:01.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:01.378 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:01.424 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:01.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:01.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:01.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:01.567 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:01.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:01.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:01.708 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:01.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:01.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:01.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:01.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:01.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:01.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:02.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:02.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:02.143 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:02.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:02.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:02.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:02.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:02.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:02.558 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:02.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:02.658 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:16:02.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:02.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:02.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:02.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:02.866 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:02 smithi118 ceph-mon[33862]: pgmap v607: 158 pgs: 3 active+clean+remapped, 20 active+undersized+degraded, 39 active+undersized, 96 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 869 B/s rd, 0 B/s wr, 1.07k op/s; 15245/148560 objects degraded (10.262%); 6/148560 objects misplaced (0.004%); 18 B/s, 2 objects/s recovering 2024-03-21T14:16:02.866 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2937569058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:02.866 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:02.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:02.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:02.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:02 smithi003 ceph-mon[29576]: pgmap v607: 158 pgs: 3 active+clean+remapped, 20 active+undersized+degraded, 39 active+undersized, 96 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 869 B/s rd, 0 B/s wr, 1.07k op/s; 15245/148560 objects degraded (10.262%); 6/148560 objects misplaced (0.004%); 18 B/s, 2 objects/s recovering 2024-03-21T14:16:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2937569058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:03.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:02 smithi003 ceph-mon[40510]: pgmap v607: 158 pgs: 3 active+clean+remapped, 20 active+undersized+degraded, 39 active+undersized, 96 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 869 B/s rd, 0 B/s wr, 1.07k op/s; 15245/148560 objects degraded (10.262%); 6/148560 objects misplaced (0.004%); 18 B/s, 2 objects/s recovering 2024-03-21T14:16:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2937569058' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:03.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221714 7.5 7:a085cce0:::benchmark_data_smithi003_80362_object100697:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:09.203594+0000", 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 473.86600691299998, 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91755642800000004, 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.077 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221714 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203594+0000", 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203594+0000", 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203596+0000", 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5069999999999999e-06 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203602+0000", 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5080000000000002e-06 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203617+0000", 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5393999999999999e-05 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203627+0000", 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8009999999999998e-06 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203691+0000", 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3232999999999996e-05 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.203698+0000", 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6059999999999997e-06 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.108460+0000", 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.904761708 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.108554+0000", 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4648999999999995e-05 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.108661+0000", 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010609 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.113447+0000", 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047865759999999999 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.114675+0000", 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012281149999999999 2024-03-21T14:16:03.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121049+0000", 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0063740799999999999 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121105+0000", 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5921999999999998e-05 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121150+0000", 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5238999999999998e-05 2024-03-21T14:16:03.083 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221727 7.5 7:abe64f4d:::benchmark_data_smithi003_80362_object100710:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:09.225874+0000", 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 473.84372710999997, 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89618225600000001, 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:16:03.084 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221727 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225874+0000", 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225874+0000", 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225876+0000", 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.652e-06 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225886+0000", 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0139999999999999e-05 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225900+0000", 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3848e-05 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225918+0000", 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8848999999999999e-05 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225979+0000", 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1045999999999998e-05 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.225986+0000", 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8549999999999996e-06 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.110662+0000", 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88467538999999995 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.110731+0000", 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9122999999999995e-05 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.110796+0000", 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4794000000000003e-05 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.114631+0000", 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038355260000000001 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121196+0000", 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006564917 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121953+0000", 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00075671699999999996 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122008+0000", 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5500000000000001e-05 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122056+0000", 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7899000000000003e-05 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221734 7.5 7:a009b07c:::benchmark_data_smithi003_80362_object100717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:09.236693+0000", 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 473.832907584, 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88550805300000002, 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221734 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236693+0000", 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236693+0000", 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236695+0000", 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.567e-06 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236702+0000", 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1169999999999998e-06 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236718+0000", 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5786000000000001e-05 2024-03-21T14:16:03.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236727+0000", 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5140000000000007e-06 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236915+0000", 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018751600000000001 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.236926+0000", 2024-03-21T14:16:03.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0886999999999999e-05 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.111979+0000", 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87505370800000004 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.112061+0000", 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1099000000000004e-05 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.112207+0000", 2024-03-21T14:16:03.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014620599999999999 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.115715+0000", 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0035083549999999999 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121247+0000", 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055320940000000004 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122111+0000", 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00086350999999999995 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122167+0000", 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5794000000000001e-05 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122201+0000", 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4900000000000001e-05 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221740 7.5 7:a622baf6:::benchmark_data_smithi003_80362_object100723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:09.238383+0000", 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 473.831217874, 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88393924999999995, 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.098 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221740 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238383+0000", 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238383+0000", 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238384+0000", 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7600000000000006e-07 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238393+0000", 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6079999999999996e-06 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238406+0000", 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3011e-05 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238414+0000", 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1480000000000006e-06 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238477+0000", 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3563000000000001e-05 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.238485+0000", 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0490000000000004e-06 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.113480+0000", 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87499434399999998 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.113586+0000", 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106368 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.113659+0000", 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3039000000000003e-05 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.115723+0000", 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002063446 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121285+0000", 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055627410000000004 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122235+0000", 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00094923499999999999 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122274+0000", 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9558999999999999e-05 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122322+0000", 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8162999999999999e-05 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221748 7.5 7:a28ddefb:::benchmark_data_smithi003_80362_object100731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:09.246413+0000", 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 473.82318776800003, 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87604279600000001, 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221748 2024-03-21T14:16:03.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246413+0000", 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246413+0000", 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246414+0000", 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2720000000000001e-06 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246420+0000", 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6319999999999996e-06 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246432+0000", 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2363e-05 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246440+0000", 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5209999999999997e-06 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246505+0000", 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5481999999999994e-05 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.246516+0000", 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.049e-05 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.114708+0000", 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86819165899999995 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.114772+0000", 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3986999999999996e-05 2024-03-21T14:16:03.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.114836+0000", 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4712e-05 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.116837+0000", 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020009120000000001 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121319+0000", 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004481825 2024-03-21T14:16:03.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122365+0000", 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0010455709999999999 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122415+0000", 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0621999999999997e-05 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122456+0000", 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0747999999999997e-05 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24692.0:221751 7.5 7:ad990cea:::benchmark_data_smithi003_80362_object100734:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:08:09.251925+0000", 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 473.81767595299999, 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.87066079299999999, 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24692", 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 221751 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251925+0000", 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251925+0000", 2024-03-21T14:16:03.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251926+0000", 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.065e-06 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251932+0000", 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5430000000000001e-06 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251944+0000", 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1946e-05 2024-03-21T14:16:03.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251950+0000", 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0070000000000003e-06 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.251995+0000", 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5161999999999999e-05 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:09.252002+0000", 2024-03-21T14:16:03.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.097e-06 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.115757+0000", 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86375479399999999 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.115828+0000", 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1113999999999997e-05 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.115895+0000", 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7230999999999994e-05 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.116920+0000", 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001025518 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.121354+0000", 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0044333159999999996 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122498+0000", 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001144574 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122549+0000", 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0772000000000001e-05 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:08:10.122586+0000", 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6653999999999998e-05 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218154 8.17 8:e9ce89ea:::benchmark_data_smithi003_110242_object87237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:07.964667+0000", 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 235.10493399999999, 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0141021400000001, 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.120 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218154 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964667+0000", 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964667+0000", 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964668+0000", 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8999999999999996e-07 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964671+0000", 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6059999999999999e-06 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964679+0000", 2024-03-21T14:16:03.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4919999999999996e-06 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964686+0000", 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8689999999999998e-06 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964753+0000", 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7691e-05 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964761+0000", 2024-03-21T14:16:03.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9570000000000002e-06 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.974133+0000", 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0093721529999999 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.974256+0000", 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012236699999999999 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.974600+0000", 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034449200000000001 2024-03-21T14:16:03.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978523+0000", 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039230080000000004 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978568+0000", 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5030000000000001e-05 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978691+0000", 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000122984 2024-03-21T14:16:03.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978732+0000", 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1128999999999999e-05 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978769+0000", 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6671999999999997e-05 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218177 8.17 8:e895dc9a:::benchmark_data_smithi003_110242_object87260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:07.984637+0000", 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 235.08496424000001, 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99749402899999995, 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218177 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984637+0000", 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984637+0000", 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984638+0000", 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.547e-06 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984643+0000", 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6609999999999999e-06 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984652+0000", 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2539999999999998e-06 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984659+0000", 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0500000000000003e-06 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984707+0000", 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7985999999999998e-05 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984716+0000", 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4570000000000005e-06 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.976740+0000", 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99202411899999998 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.976877+0000", 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013677700000000001 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.977128+0000", 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025135400000000002 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978779+0000", 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001651146 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982000+0000", 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003220503 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982052+0000", 2024-03-21T14:16:03.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2018000000000002e-05 2024-03-21T14:16:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:03.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982086+0000", 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4187999999999999e-05 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982131+0000", 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4969000000000003e-05 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218191 8.17 8:e950946d:::benchmark_data_smithi003_110242_object87274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:07.992480+0000", 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 235.07712139200001, 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99410021699999995, 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.135 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218191 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992480+0000", 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992480+0000", 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992481+0000", 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.285e-06 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992486+0000", 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519999999999997e-06 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992500+0000", 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3328e-05 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992508+0000", 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3340000000000002e-06 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992565+0000", 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7463999999999997e-05 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992571+0000", 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.071e-06 2024-03-21T14:16:03.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978807+0000", 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98623560600000004 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978924+0000", 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001171 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.979134+0000", 2024-03-21T14:16:03.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000209724 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982183+0000", 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030487520000000001 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982198+0000", 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5091e-05 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986498+0000", 2024-03-21T14:16:03.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043003549999999996 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986536+0000", 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8401e-05 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986580+0000", 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3353999999999999e-05 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218274 8.17 8:e974f6a2:::benchmark_data_smithi003_110242_object87357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.089562+0000", 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 234.98003909400001, 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89721891499999995, 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218274 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089562+0000", 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089562+0000", 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089563+0000", 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089568+0000", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8099999999999997e-06 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089578+0000", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.031e-05 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089585+0000", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6900000000000003e-06 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089658+0000", 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3218000000000003e-05 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089669+0000", 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-05 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.980361+0000", 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89069226700000004 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.980477+0000", 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115708 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.980702+0000", 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224692 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982267+0000", 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001565227 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.983750+0000", 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001482997 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986695+0000", 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029450180000000002 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986746+0000", 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1450999999999999e-05 2024-03-21T14:16:03.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986781+0000", 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4390000000000001e-05 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218280 8.17 8:edc70fe4:::benchmark_data_smithi003_110242_object87363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.094569+0000", 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 234.97503186700001, 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89239709199999995, 2024-03-21T14:16:03.148 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218280 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094569+0000", 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094569+0000", 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094571+0000", 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5519999999999999e-06 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094579+0000", 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0739999999999993e-06 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094591+0000", 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-05 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094599+0000", 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1000000000000004e-06 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094659+0000", 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9682000000000002e-05 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094668+0000", 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0950000000000002e-06 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982289+0000", 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88762121900000002 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982357+0000", 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8149000000000001e-05 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982496+0000", 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000138947 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986635+0000", 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041387170000000001 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986834+0000", 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019904999999999999 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986847+0000", 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2480000000000001e-05 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986922+0000", 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5099999999999996e-05 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986966+0000", 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4597e-05 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.154 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218398 8.17 8:eb064d09:::benchmark_data_smithi003_110242_object87481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.188249+0000", 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 234.88135199600001, 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.79902141999999998, 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218398 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.155 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188249+0000", 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188249+0000", 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188250+0000", 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.364e-06 2024-03-21T14:16:03.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188256+0000", 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7139999999999998e-06 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188271+0000", 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.456e-05 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188277+0000", 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6359999999999997e-06 2024-03-21T14:16:03.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188338+0000", 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0778e-05 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.188347+0000", 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4569999999999994e-06 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.983821+0000", 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.79547370399999995 2024-03-21T14:16:03.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.983948+0000", 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012709400000000001 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.984132+0000", 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018405799999999999 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986977+0000", 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028446790000000001 2024-03-21T14:16:03.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987036+0000", 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9385999999999997e-05 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987172+0000", 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013549800000000001 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987232+0000", 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0409999999999999e-05 2024-03-21T14:16:03.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987270+0000", 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8081999999999999e-05 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218410 8.17 8:ecd6c9fc:::benchmark_data_smithi003_110242_object87493:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.202346+0000", 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 234.86725512800001, 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.79447481900000005, 2024-03-21T14:16:03.161 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218410 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202346+0000", 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202346+0000", 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202347+0000", 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1170000000000001e-06 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202352+0000", 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4060000000000004e-06 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202363+0000", 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0981e-05 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202369+0000", 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8900000000000004e-06 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202416+0000", 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7049000000000003e-05 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.202422+0000", 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1489999999999999e-06 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987305+0000", 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.78488220500000005 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987426+0000", 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121783 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.987716+0000", 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028973699999999998 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.993036+0000", 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053196090000000003 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.996704+0000", 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036680660000000002 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.996718+0000", 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4559000000000001e-05 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.996772+0000", 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3291000000000002e-05 2024-03-21T14:16:03.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.996821+0000", 2024-03-21T14:16:03.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8977000000000002e-05 2024-03-21T14:16:03.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.169 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218417 8.17 8:ea8bd165:::benchmark_data_smithi003_110242_object87500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.209329+0000", 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 234.86027222999999, 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.79653006100000001, 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218417 2024-03-21T14:16:03.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209329+0000", 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209329+0000", 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209330+0000", 2024-03-21T14:16:03.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4160000000000001e-06 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209335+0000", 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3329999999999996e-06 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209349+0000", 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3967e-05 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209358+0000", 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7450000000000001e-06 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209429+0000", 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0469000000000006e-05 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.209438+0000", 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7210000000000006e-06 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.989212+0000", 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.77977376200000004 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.989337+0000", 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012500799999999999 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.989603+0000", 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000266309 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.996893+0000", 2024-03-21T14:16:03.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.007289589 2024-03-21T14:16:03.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.996961+0000", 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7770999999999999e-05 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.005722+0000", 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0087613080000000006 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.005810+0000", 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7880000000000006e-05 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.005859+0000", 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8782999999999997e-05 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228274 8.6 8:60f051e7:::benchmark_data_smithi003_110242_object97357:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.562509+0000", 2024-03-21T14:16:03.177 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.50709163100001, 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99705723000000002, 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228274 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562509+0000", 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562509+0000", 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562511+0000", 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9269999999999999e-06 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562519+0000", 2024-03-21T14:16:03.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9959999999999993e-06 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562533+0000", 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3838e-05 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562545+0000", 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2347000000000001e-05 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.563036+0000", 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049081699999999995 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.563049+0000", 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2536e-05 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.544628+0000", 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98157942899999995 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.544752+0000", 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012386 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.545144+0000", 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039143199999999999 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556353+0000", 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011209566000000001 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559313+0000", 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002959602 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559349+0000", 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6090999999999997e-05 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559429+0000", 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9766999999999997e-05 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559467+0000", 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8642e-05 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559567+0000", 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9380000000000001e-05 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228413 8.6 8:67cc4f9b:::benchmark_data_smithi003_110242_object97496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.646732+0000", 2024-03-21T14:16:03.184 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.422868655, 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91498634899999998, 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228413 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646732+0000", 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646732+0000", 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646734+0000", 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.593e-06 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646740+0000", 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8479999999999997e-06 2024-03-21T14:16:03.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646752+0000", 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2476e-05 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646761+0000", 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5229999999999995e-06 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646825+0000", 2024-03-21T14:16:03.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3948999999999994e-05 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646834+0000", 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7729999999999994e-06 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556388+0000", 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90955385300000002 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556477+0000", 2024-03-21T14:16:03.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8911000000000003e-05 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556691+0000", 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021380800000000001 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559386+0000", 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002694723 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559394+0000", 2024-03-21T14:16:03.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3219999999999993e-06 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561494+0000", 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020995200000000001 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561599+0000", 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105534 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561635+0000", 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6341999999999999e-05 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561719+0000", 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3174e-05 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228445 8.6 8:60e0e6c0:::benchmark_data_smithi003_110242_object97528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.667121+0000", 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.40247949900001, 2024-03-21T14:16:03.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89474894699999996, 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228445 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667121+0000", 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667121+0000", 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667123+0000", 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.53e-06 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667129+0000", 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.964e-06 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667143+0000", 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3947999999999999e-05 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667151+0000", 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2770000000000005e-06 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667212+0000", 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0946999999999998e-05 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667221+0000", 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6729999999999998e-06 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.557860+0000", 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89063941499999999 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.558000+0000", 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013936800000000001 2024-03-21T14:16:03.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.558236+0000", 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023654500000000001 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559399+0000", 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011625610000000001 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561529+0000", 2024-03-21T14:16:03.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021306739999999999 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561560+0000", 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0176e-05 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561755+0000", 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000195596 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561788+0000", 2024-03-21T14:16:03.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2755000000000001e-05 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561870+0000", 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2517999999999998e-05 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228480 8.6 8:66f7579a:::benchmark_data_smithi003_110242_object97563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.693153+0000", 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.376448377, 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86899963800000002, 2024-03-21T14:16:03.198 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228480 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693153+0000", 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693153+0000", 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693154+0000", 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5269999999999999e-06 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693159+0000", 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8459999999999999e-06 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693172+0000", 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3341e-05 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693180+0000", 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7749999999999993e-06 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693237+0000", 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6993999999999997e-05 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.201 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.693246+0000", 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3379999999999994e-06 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559594+0000", 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86634780099999997 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559667+0000", 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2972000000000005e-05 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:16:03.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559896+0000", 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022843599999999999 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561570+0000", 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001674182 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561917+0000", 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034678399999999999 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561950+0000", 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2938000000000002e-05 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561981+0000", 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0976000000000003e-05 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.562033+0000", 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2006000000000003e-05 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.562152+0000", 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119722 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228529 8.6 8:64e85a5a:::benchmark_data_smithi003_110242_object97612:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.723971+0000", 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.345630333, 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.85467870400000001, 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.205 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228529 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.723971+0000", 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.723971+0000", 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.723972+0000", 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5880000000000001e-06 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.723978+0000", 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0140000000000004e-06 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.723991+0000", 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2621e-05 2024-03-21T14:16:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.723999+0000", 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8140000000000001e-06 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.724056+0000", 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7229999999999999e-05 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.724067+0000", 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0854e-05 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.562205+0000", 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.838138406 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.562334+0000", 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128975 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.562632+0000", 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029800699999999998 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.569088+0000", 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064561510000000003 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.569106+0000", 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7918e-05 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.575133+0000", 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006026549 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.578432+0000", 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032992569999999999 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.578501+0000", 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8732000000000005e-05 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.578649+0000", 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014858800000000001 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.211 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228531 8.6 8:648266ac:::benchmark_data_smithi003_110242_object97614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.725219+0000", 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 222.344381771, 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86224852699999999, 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228531 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.212 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725219+0000", 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725219+0000", 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725220+0000", 2024-03-21T14:16:03.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3200000000000003e-07 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725224+0000", 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5619999999999999e-06 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725231+0000", 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0099999999999998e-06 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725236+0000", 2024-03-21T14:16:03.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4870000000000002e-06 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725290+0000", 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3841000000000002e-05 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.725299+0000", 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8619999999999992e-06 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:16:03.215 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.564394+0000", 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.83909514299999999 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.564556+0000", 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016185700000000001 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.564898+0000", 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034257500000000002 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.569154+0000", 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0042552930000000003 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.570475+0000", 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001321425 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.583335+0000", 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012860284 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:03.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.587297+0000", 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039618559999999997 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.587355+0000", 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7574999999999997e-05 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.587468+0000", 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112825 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.218 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:16:03.219 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:16:03.219 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:16:03.234 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:03.268 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:03.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:03.405 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:03.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:03.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:03.643 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.678 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:03.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:03.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:03.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:03 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:03 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2937569058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:16:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:03 smithi118 ceph-mon[33862]: osdmap e299: 8 total, 7 up, 7 in 2024-03-21T14:16:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:03 smithi118 ceph-mon[33862]: 1.3 scrub starts 2024-03-21T14:16:03.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:03 smithi118 ceph-mon[33862]: 1.3 scrub ok 2024-03-21T14:16:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:03.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2937569058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:16:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[29576]: osdmap e299: 8 total, 7 up, 7 in 2024-03-21T14:16:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[29576]: 1.3 scrub starts 2024-03-21T14:16:04.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[29576]: 1.3 scrub ok 2024-03-21T14:16:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2937569058' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:16:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[40510]: osdmap e299: 8 total, 7 up, 7 in 2024-03-21T14:16:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[40510]: 1.3 scrub starts 2024-03-21T14:16:04.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:03 smithi003 ceph-mon[40510]: 1.3 scrub ok 2024-03-21T14:16:04.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:04.087 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:04.189 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:04.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:16:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.244 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:04.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:04.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:04.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:04.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:04.496 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:04.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:04.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:04.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:04.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:04.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:04.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:04.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:04.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:04 smithi118 ceph-mon[33862]: pgmap v609: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.50k op/s; 12919/126588 objects degraded (10.206%); 6/126588 objects misplaced (0.005%); 22 B/s, 2 objects/s recovering 2024-03-21T14:16:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:04 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 12919/126588 objects degraded (10.206%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:04.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:04 smithi118 ceph-mon[33862]: osdmap e300: 8 total, 7 up, 7 in 2024-03-21T14:16:04.939 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:04.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:04.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:04.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:04 smithi003 ceph-mon[29576]: pgmap v609: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.50k op/s; 12919/126588 objects degraded (10.206%); 6/126588 objects misplaced (0.005%); 22 B/s, 2 objects/s recovering 2024-03-21T14:16:04.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:04 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 12919/126588 objects degraded (10.206%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:04.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:04 smithi003 ceph-mon[29576]: osdmap e300: 8 total, 7 up, 7 in 2024-03-21T14:16:04.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:04 smithi003 ceph-mon[40510]: pgmap v609: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.50k op/s; 12919/126588 objects degraded (10.206%); 6/126588 objects misplaced (0.005%); 22 B/s, 2 objects/s recovering 2024-03-21T14:16:04.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:04 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 12919/126588 objects degraded (10.206%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:04.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:04 smithi003 ceph-mon[40510]: osdmap e300: 8 total, 7 up, 7 in 2024-03-21T14:16:05.030 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:16:05.030 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:16:05.030 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-03-21T14:16:05.030 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T14:16:05.031 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T14:16:05.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-03-21T14:16:05.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:05.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:05.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:05.227 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:05.329 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:05.461 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:05.472 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:16:05.492 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:05.594 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.694 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:05.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:05.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:05.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.838 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:05.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:05.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:05.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:05.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:06.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:06.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.114 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:06.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:06.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:06.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.243 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:06.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:06.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.382 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:06.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:06.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:06.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:06.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:06.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:06.652 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:06.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:06 smithi003 ceph-mon[29576]: pgmap v611: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 866 B/s rd, 0 B/s wr, 1.24k op/s; 12919/126588 objects degraded (10.206%); 6/126588 objects misplaced (0.005%); 18 B/s, 2 objects/s recovering 2024-03-21T14:16:06.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:16:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:16:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:16:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:06 smithi003 ceph-mon[40510]: pgmap v611: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 866 B/s rd, 0 B/s wr, 1.24k op/s; 12919/126588 objects degraded (10.206%); 6/126588 objects misplaced (0.005%); 18 B/s, 2 objects/s recovering 2024-03-21T14:16:06.796 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:06.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:06 smithi118 ceph-mon[33862]: pgmap v611: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 866 B/s rd, 0 B/s wr, 1.24k op/s; 12919/126588 objects degraded (10.206%); 6/126588 objects misplaced (0.005%); 18 B/s, 2 objects/s recovering 2024-03-21T14:16:06.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:06.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:07.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:16:07.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:07.072 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:07.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:07.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:07.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:07.212 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:07.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:07.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:07.230 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:16:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:16:07.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:07.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:07.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:07.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:07.433 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:16:07.433 INFO:teuthology.orchestra.run.smithi118.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-21T14:16:07.433 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:16:07.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:07.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:07.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:07.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:07.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:07.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:07.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:07.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:07.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:07.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:07.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:07.850 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:07.891 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:07.893 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:07 smithi118 ceph-mon[33862]: 9.c scrub starts 2024-03-21T14:16:07.893 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:07 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:16:07.893 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:07 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:16:07.893 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:07 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:16:07.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[29576]: 9.c scrub starts 2024-03-21T14:16:07.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:16:07.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:16:07.991 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:16:07.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[40510]: 9.c scrub starts 2024-03-21T14:16:07.991 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:16:07.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:16:07.992 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:07 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:16:07.992 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:16:08.024 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:16:08.062 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:16:08.110 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:16:08.145 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:16:08.182 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:16:08.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:16:08.223 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:16:08.223 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:16:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:16:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:16:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:16:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:16:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:16:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:16:08.228 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:08.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:08.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:08.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:08.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:08.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:08.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:08.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:08.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:08.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:08.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:08.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:08.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:08.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:08.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:08.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:08.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:08.872 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:08.908 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:08.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: 9.c scrub ok 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: pgmap v612: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.13k op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%); 15 B/s, 1 objects/s recovering 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1618833846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:16:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:08 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: 9.c scrub ok 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: pgmap v612: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.13k op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%); 15 B/s, 1 objects/s recovering 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1618833846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:16:09.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: 9.c scrub ok 2024-03-21T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: pgmap v612: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2.13k op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%); 15 B/s, 1 objects/s recovering 2024-03-21T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1618833846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:16:09.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:08 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:16:09.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:09.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:09.142 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:09.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:09.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:09.284 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.324 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:09.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:09.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:09.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:09.492 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:16:09.493 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:16:09.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:09.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:09.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:09.752 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:09 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 10685/104436 objects degraded (10.231%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:09.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:09 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 10685/104436 objects degraded (10.231%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:09.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:09.892 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:09.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:09.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:09.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:09 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 10685/104436 objects degraded (10.231%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:09.971 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:09.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:10.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:10.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:10.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:10.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.316 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:10.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:10.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:10.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:10.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.600 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:10.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:10.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.736 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:10.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:10.833 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:10 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:16:10.833 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:10 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:16:10.833 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:10 smithi118 ceph-mon[33862]: pgmap v613: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%) 2024-03-21T14:16:10.833 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:10 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:16:10.833 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:10 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:16:10.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:10.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:10.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:10.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:16:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:16:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[29576]: pgmap v613: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%) 2024-03-21T14:16:11.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:16:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:16:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:16:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:16:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[40510]: pgmap v613: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.84k op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%) 2024-03-21T14:16:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:16:11.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:10 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:16:11.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:11.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:11.147 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:11.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.290 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:11.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:11.390 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:11.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:11.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:11.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:16:11.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:11.563 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:11.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:11.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:11.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:11.840 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:11.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:11.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:11.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:16:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:16:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:16:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:16:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:16:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:16:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:16:12.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:11 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:16:12.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:12.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:12.143 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:16:12.143 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:16:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:11 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:16:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:11 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:16:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:11 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:16:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:11 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:16:12.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:12.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:12.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:12.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:12.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:12.532 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:12.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:12.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:12.668 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:12.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:12.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:12.807 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:12.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:12.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:12.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:12.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:13.002 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:12 smithi003 ceph-mon[29576]: pgmap v614: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 578 B/s rd, 0 B/s wr, 834 op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%) 2024-03-21T14:16:13.002 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:12 smithi003 ceph-mon[40510]: pgmap v614: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 578 B/s rd, 0 B/s wr, 834 op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%) 2024-03-21T14:16:13.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:13.088 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:12 smithi118 ceph-mon[33862]: pgmap v614: 158 pgs: 1 active+clean+scrubbing, 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 92 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 578 B/s rd, 0 B/s wr, 834 op/s; 10685/104436 objects degraded (10.231%); 6/104436 objects misplaced (0.006%) 2024-03-21T14:16:13.189 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:13.227 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:13.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:13.329 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:13.359 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:16:13.359 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:16:13.359 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T14:16:13.360 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.026660 2024-03-21T14:16:13.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 5 0.02666041412128639 2024-03-21T14:16:13.364 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:13.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:13.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:13.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:13.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:13.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:13.704 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:16:13.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.799 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:13.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:13.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:13.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:13 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:16:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:13 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:16:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1316031334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:16:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:13 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:16:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:13 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:16:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1316031334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:16:14.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:14.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:14.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:13 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:16:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:13 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:16:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1316031334' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:16:14.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:16:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:14.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:14.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:14.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:14.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:14.504 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:14.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.508 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:14.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:14.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:14.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:14.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:14.747 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:14.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:14.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:14.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: pgmap v615: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 8341/81621 objects degraded (10.219%); 6/81621 objects misplaced (0.007%) 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 1.4 scrub starts 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 1.4 scrub ok 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1316031334' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:16:14.923 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: osdmap e301: 8 total, 7 up, 7 in 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 8341/81621 objects degraded (10.219%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[29576]: osdmap e302: 8 total, 7 up, 7 in 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: pgmap v615: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 8341/81621 objects degraded (10.219%); 6/81621 objects misplaced (0.007%) 2024-03-21T14:16:14.924 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 1.4 scrub starts 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 1.4 scrub ok 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1316031334' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: osdmap e301: 8 total, 7 up, 7 in 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 8341/81621 objects degraded (10.219%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:16:14.925 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:14 smithi003 ceph-mon[40510]: osdmap e302: 8 total, 7 up, 7 in 2024-03-21T14:16:14.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:14.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:14.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:15.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:15.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:15.162 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:16:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:16:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: pgmap v615: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.50k op/s; 8341/81621 objects degraded (10.219%); 6/81621 objects misplaced (0.007%) 2024-03-21T14:16:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 1.4 scrub starts 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 1.4 scrub ok 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1316031334' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: osdmap e301: 8 total, 7 up, 7 in 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 8341/81621 objects degraded (10.219%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:16:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:14 smithi118 ceph-mon[33862]: osdmap e302: 8 total, 7 up, 7 in 2024-03-21T14:16:15.212 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:15.281 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:15.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.350 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:15.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:15.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:15.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:15.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:15.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:15.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:15.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:15.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:15.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:15.901 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:15.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:16.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:16.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.039 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:16.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:16.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:16.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:16.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:16.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:16.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:16.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:16.453 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:16.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:16.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.592 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:16.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:16.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:16.706 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:16:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:16:16.735 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:16.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:16.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:16.875 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:16.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:16.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:16:16.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:16:16.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[29576]: pgmap v618: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 8341/81621 objects degraded (10.219%); 6/81621 objects misplaced (0.007%) 2024-03-21T14:16:16.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:16:16.972 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:16:16.973 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:16:16.973 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:16:16.973 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[40510]: pgmap v618: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 8341/81621 objects degraded (10.219%); 6/81621 objects misplaced (0.007%) 2024-03-21T14:16:16.973 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:16:16.973 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:16 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:16:16.975 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:17.005 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:17.066 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:16 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:16:17.067 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:16 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:16:17.067 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:16 smithi118 ceph-mon[33862]: pgmap v618: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 950 op/s; 8341/81621 objects degraded (10.219%); 6/81621 objects misplaced (0.007%) 2024-03-21T14:16:17.067 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:16 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:16:17.067 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:16 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:16:17.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:17.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:17.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:17.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:17.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:17.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:17.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:17.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:17.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:16:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:16:17.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:17.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:17.663 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:17.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:17.705 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:17.745 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:16:17.783 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:16:17.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:17.821 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:16:17.841 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:17.855 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:16:17.893 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:16:17.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:16:17.928 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:16:17.943 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:16:17.976 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:16:17.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:17.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:18.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:18.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.116 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:18.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:17 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2340599747' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:18.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:18.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:18.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:18.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:17 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2340599747' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:18.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:18.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:18.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:17 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2340599747' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:18.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:18.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:18.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:18.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:18.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:18.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:18.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:18.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:18.666 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:18.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:18.805 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:18.907 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:18.945 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:18.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:19.047 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:19.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:19.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:18 smithi118 ceph-mon[33862]: pgmap v619: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.96k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%) 2024-03-21T14:16:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:18 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:16:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:18 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:16:19.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[29576]: pgmap v619: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.96k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%) 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[40510]: pgmap v619: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.96k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%) 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:16:19.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:18 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:16:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:19.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:16:19.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:16:19.262 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:19.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:19.346 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:19.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:19.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:19.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:19.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:19.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:19.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:19.814 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:19.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:19.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:19.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:19.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:20.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:20.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:20.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:16:20.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[29576]: osdmap e303: 8 total, 7 up, 7 in 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 5789/57420 objects degraded (10.082%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[29576]: osdmap e304: 8 total, 7 up, 7 in 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[40510]: osdmap e303: 8 total, 7 up, 7 in 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 5789/57420 objects degraded (10.082%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:20.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:19 smithi003 ceph-mon[40510]: osdmap e304: 8 total, 7 up, 7 in 2024-03-21T14:16:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:19 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:16:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:19 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:16:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:19 smithi118 ceph-mon[33862]: osdmap e303: 8 total, 7 up, 7 in 2024-03-21T14:16:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:19 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 5789/57420 objects degraded (10.082%), 20 pgs degraded, 58 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:19 smithi118 ceph-mon[33862]: osdmap e304: 8 total, 7 up, 7 in 2024-03-21T14:16:20.194 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:20.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:20.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:20.371 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:20.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:20.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:20.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.506 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:20.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:20.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.644 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:20.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:20.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:20.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:20.869 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 0.0266604 (817472) 2024-03-21T14:16:20.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:20.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:21.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:21.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:21.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:21.159 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:21.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:21.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:16:21.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:16:21.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: pgmap v621: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%) 2024-03-21T14:16:21.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:16:21.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:16:21.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2451414823' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]: dispatch 2024-03-21T14:16:21.162 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:20 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]: dispatch 2024-03-21T14:16:21.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:16:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: pgmap v621: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%) 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2451414823' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]: dispatch 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]: dispatch 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: pgmap v621: 158 pgs: 7 active+clean+remapped, 20 active+undersized+degraded, 38 active+undersized, 93 active+clean; 1.2 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%) 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:16:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2451414823' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]: dispatch 2024-03-21T14:16:21.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:20 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]: dispatch 2024-03-21T14:16:21.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:21.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:21.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:21.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:21.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:21.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:21.534 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:21.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:21.663 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:21.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:21.765 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:21.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:21.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:21.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:21.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:22.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:22.065 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:22.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:22.167 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:22.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:21 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:16:22.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:21 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:16:22.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:21 smithi118 ceph-mon[33862]: 3.2 deep-scrub starts 2024-03-21T14:16:22.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:21 smithi118 ceph-mon[33862]: 3.2 deep-scrub ok 2024-03-21T14:16:22.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:21 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]': finished 2024-03-21T14:16:22.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:21 smithi118 ceph-mon[33862]: osdmap e305: 8 total, 7 up, 7 in 2024-03-21T14:16:22.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.204 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:22.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:16:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:16:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[29576]: 3.2 deep-scrub starts 2024-03-21T14:16:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[29576]: 3.2 deep-scrub ok 2024-03-21T14:16:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]': finished 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[29576]: osdmap e305: 8 total, 7 up, 7 in 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[40510]: 3.2 deep-scrub starts 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[40510]: 3.2 deep-scrub ok 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.02666041412128639}]': finished 2024-03-21T14:16:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:21 smithi003 ceph-mon[40510]: osdmap e305: 8 total, 7 up, 7 in 2024-03-21T14:16:22.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:22.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:22.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:22.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:22.480 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:22.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:22.581 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:22.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:22.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:22.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:22.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:22.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:22.898 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:22.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:23.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: pgmap v624: 158 pgs: 7 active+clean+remapped, 4 peering, 20 active+undersized+degraded, 36 active+undersized, 91 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 5.3 KiB/s rd, 36 KiB/s wr, 1.35k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%); 35 KiB/s, 0 objects/s recovering 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: 9.1 scrub starts 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: 9.1 scrub ok 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: osdmap e306: 8 total, 7 up, 7 in 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:16:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: pgmap v624: 158 pgs: 7 active+clean+remapped, 4 peering, 20 active+undersized+degraded, 36 active+undersized, 91 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 5.3 KiB/s rd, 36 KiB/s wr, 1.35k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%); 35 KiB/s, 0 objects/s recovering 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: 9.1 scrub starts 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: 9.1 scrub ok 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: osdmap e306: 8 total, 7 up, 7 in 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:16:23.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:22 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:16:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:23.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:23.141 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:23.176 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: pgmap v624: 158 pgs: 7 active+clean+remapped, 4 peering, 20 active+undersized+degraded, 36 active+undersized, 91 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 5.3 KiB/s rd, 36 KiB/s wr, 1.35k op/s; 5789/57420 objects degraded (10.082%); 6/57420 objects misplaced (0.010%); 35 KiB/s, 0 objects/s recovering 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: 9.1 scrub starts 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: 9.1 scrub ok 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: Health check failed: Reduced data availability: 4 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: osdmap e306: 8 total, 7 up, 7 in 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:16:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:22 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:16:23.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:23.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.313 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:23.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:23.414 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:23.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:23.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:23.492 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:16:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:16:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:16:23.492 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:16:23.492 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:16:23.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:23.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:23.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:23.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:23.741 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:23.843 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:23.881 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:23.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:23.900 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:23.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:24.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:24.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:24.123 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:24.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:24.164 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:23 smithi003 ceph-mon[29576]: osdmap e307: 8 total, 7 up, 7 in 2024-03-21T14:16:24.165 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:23 smithi003 ceph-mon[40510]: osdmap e307: 8 total, 7 up, 7 in 2024-03-21T14:16:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:23 smithi118 ceph-mon[33862]: osdmap e307: 8 total, 7 up, 7 in 2024-03-21T14:16:24.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:24.294 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:24.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:24.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:24.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:24.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:24.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.547 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:24.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:24.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:24.678 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:24.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:24.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:24.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:24.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:24.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:24.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:24.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:25.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:25.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:25.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: pgmap v627: 158 pgs: 2 remapped+peering, 5 active+clean+remapped, 11 peering, 18 active+undersized+degraded, 36 active+undersized, 86 active+clean; 777 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 12 KiB/s rd, 109 KiB/s wr, 1.65k op/s; 2970/37854 objects degraded (7.846%); 6/37854 objects misplaced (0.016%); 105 KiB/s, 0 objects/s recovering 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: osdmap e308: 8 total, 7 up, 7 in 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 2970/37854 objects degraded (7.846%), 18 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1740555733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:16:25.178 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: pgmap v627: 158 pgs: 2 remapped+peering, 5 active+clean+remapped, 11 peering, 18 active+undersized+degraded, 36 active+undersized, 86 active+clean; 777 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 12 KiB/s rd, 109 KiB/s wr, 1.65k op/s; 2970/37854 objects degraded (7.846%); 6/37854 objects misplaced (0.016%); 105 KiB/s, 0 objects/s recovering 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: osdmap e308: 8 total, 7 up, 7 in 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:16:25.179 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 2970/37854 objects degraded (7.846%), 18 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:25.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1740555733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:16:25.180 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:24 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:16:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: pgmap v627: 158 pgs: 2 remapped+peering, 5 active+clean+remapped, 11 peering, 18 active+undersized+degraded, 36 active+undersized, 86 active+clean; 777 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 12 KiB/s rd, 109 KiB/s wr, 1.65k op/s; 2970/37854 objects degraded (7.846%); 6/37854 objects misplaced (0.016%); 105 KiB/s, 0 objects/s recovering 2024-03-21T14:16:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:16:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:16:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: osdmap e308: 8 total, 7 up, 7 in 2024-03-21T14:16:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:16:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:16:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 2970/37854 objects degraded (7.846%), 18 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1740555733' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:16:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:24 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:16:25.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:25.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:25.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:25.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.404 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:25.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:25.505 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:25.541 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:25.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:16:25.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:25.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:25.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:25.817 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:25.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:25.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:25.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:25.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:26.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:26.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:25 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:16:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:25 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:16:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:25 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:25 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:16:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:25 smithi118 ceph-mon[33862]: osdmap e309: 8 total, 7 up, 7 in 2024-03-21T14:16:26.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:26.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:16:26.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:16:26.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:26.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:16:26.215 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[29576]: osdmap e309: 8 total, 7 up, 7 in 2024-03-21T14:16:26.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:16:26.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:16:26.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:26.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:16:26.216 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:25 smithi003 ceph-mon[40510]: osdmap e309: 8 total, 7 up, 7 in 2024-03-21T14:16:26.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:26.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:26.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:26.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:26.524 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:26.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:26.626 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:26.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:16:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:16:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:16:26.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:26.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:26.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:26.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:26.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:26.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:27.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:27.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:27.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:27.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:26 smithi003 ceph-mon[29576]: pgmap v629: 158 pgs: 2 remapped+peering, 5 active+clean+remapped, 11 peering, 18 active+undersized+degraded, 36 active+undersized, 86 active+clean; 777 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 11 KiB/s rd, 94 KiB/s wr, 1.41k op/s; 2970/37854 objects degraded (7.846%); 6/37854 objects misplaced (0.016%); 90 KiB/s, 0 objects/s recovering 2024-03-21T14:16:27.167 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:26 smithi003 ceph-mon[40510]: pgmap v629: 158 pgs: 2 remapped+peering, 5 active+clean+remapped, 11 peering, 18 active+undersized+degraded, 36 active+undersized, 86 active+clean; 777 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 11 KiB/s rd, 94 KiB/s wr, 1.41k op/s; 2970/37854 objects degraded (7.846%); 6/37854 objects misplaced (0.016%); 90 KiB/s, 0 objects/s recovering 2024-03-21T14:16:27.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:27.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:16:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:16:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:26 smithi118 ceph-mon[33862]: pgmap v629: 158 pgs: 2 remapped+peering, 5 active+clean+remapped, 11 peering, 18 active+undersized+degraded, 36 active+undersized, 86 active+clean; 777 MiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 11 KiB/s rd, 94 KiB/s wr, 1.41k op/s; 2970/37854 objects degraded (7.846%); 6/37854 objects misplaced (0.016%); 90 KiB/s, 0 objects/s recovering 2024-03-21T14:16:27.209 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:27.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.344 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:27.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:27.386 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:16:27.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:16:27.387 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T14:16:27.387 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.616514 2024-03-21T14:16:27.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 0.6165136924402305 2024-03-21T14:16:27.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:27.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:27.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:27.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:27.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:27.722 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.759 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:27.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:27.860 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:27.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:27.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:27.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:28.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:28.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:28.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:27 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:16:28.184 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:28.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:28.213 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:27 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:16:28.214 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:27 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:16:28.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:28.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:28.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:28.525 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:28.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:28.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.597 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:28.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:28.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:28.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:28.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:28.869 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:28.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:28.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:28.971 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:29.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:29.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:29.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:29.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[29576]: pgmap v631: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 5.5 KiB/s rd, 39 KiB/s wr, 2.08k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 37 KiB/s, 0 objects/s recovering 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[29576]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[40510]: pgmap v631: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 5.5 KiB/s rd, 39 KiB/s wr, 2.08k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 37 KiB/s, 0 objects/s recovering 2024-03-21T14:16:29.170 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[40510]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-03-21T14:16:29.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:16:29.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:16:29.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:28 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:28 smithi118 ceph-mon[33862]: pgmap v631: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 5.5 KiB/s rd, 39 KiB/s wr, 2.08k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 37 KiB/s, 0 objects/s recovering 2024-03-21T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:28 smithi118 ceph-mon[33862]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-03-21T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:28 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:28 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:16:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:28 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:29.241 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:29.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:29.283 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:29.287 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:16:29.324 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:16:29.363 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:16:29.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:29.408 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:16:29.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:29.448 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:16:29.482 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:16:29.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:29.525 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:16:29.525 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:16:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:16:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:16:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:16:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:16:29.526 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:16:29.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:29.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:29.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:29.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:29.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:29.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:29.864 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 140731 objects 2024-03-21T14:16:29.865 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :100.606 2024-03-21T14:16:29.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:29.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:29.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:30.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.093 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:30.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2063063848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:29 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 1315/13743 objects degraded (9.569%), 19 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:29 smithi118 ceph-mon[33862]: osdmap e310: 8 total, 7 up, 7 in 2024-03-21T14:16:30.185 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2063063848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:30.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:29 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 1315/13743 objects degraded (9.569%), 19 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:30.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:29 smithi003 ceph-mon[29576]: osdmap e310: 8 total, 7 up, 7 in 2024-03-21T14:16:30.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2063063848' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:30.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:29 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 1315/13743 objects degraded (9.569%), 19 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:30.186 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:29 smithi003 ceph-mon[40510]: osdmap e310: 8 total, 7 up, 7 in 2024-03-21T14:16:30.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:30.218 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:30.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:30.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:16:30.320 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:30.345 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:30.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:30.473 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:30.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:30.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:30.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:30.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:30.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:30.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.855 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:30.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:30.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:30.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:31.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:30 smithi118 ceph-mon[33862]: pgmap v632: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 769 B/s rd, 0 B/s wr, 1.21k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:31.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:30 smithi003 ceph-mon[29576]: pgmap v632: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 769 B/s rd, 0 B/s wr, 1.21k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:30 smithi003 ceph-mon[40510]: pgmap v632: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 769 B/s rd, 0 B/s wr, 1.21k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:31.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.363 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:31.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:31.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:31.491 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:31.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:31.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:31.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:31.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:31.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:31.851 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:31.878 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:31.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:31.979 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.007 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:32.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:32.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:32.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:32.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:32.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.389 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:16:32.451 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:16:32.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:32.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:32.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:32.644 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:32.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.773 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:32.874 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:32.903 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:32.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:33.003 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:33.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:32 smithi003 ceph-mon[29576]: pgmap v634: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 671 B/s rd, 0 B/s wr, 1.05k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:33.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:32 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:33.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:33.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:32 smithi003 ceph-mon[40510]: pgmap v634: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 671 B/s rd, 0 B/s wr, 1.05k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:33.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:32 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:33.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:33.033 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:33.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:33.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:32 smithi118 ceph-mon[33862]: pgmap v634: 158 pgs: 7 active+clean+remapped, 19 active+undersized+degraded, 39 active+undersized, 93 active+clean; 274 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 671 B/s rd, 0 B/s wr, 1.05k op/s; 1315/13743 objects degraded (9.569%); 6/13743 objects misplaced (0.044%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:32 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:16:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:33.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:33.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:33.289 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:16:33.289 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:16:33.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:33.415 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:33.427 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-21T14:16:33.428 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-21T14:16:33.517 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:33.543 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:33.645 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:33.671 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:33.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:33.799 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:33.833 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 0.616514 (8404032) 2024-03-21T14:16:33.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:33.926 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:33.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:34.028 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:34.052 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:34.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3587068474' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]: dispatch 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]: dispatch 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3587068474' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]: dispatch 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]: dispatch 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:16:34.074 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:33 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:16:34.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:34.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:33 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3587068474' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]: dispatch 2024-03-21T14:16:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:33 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]: dispatch 2024-03-21T14:16:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:33 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:16:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:33 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:16:34.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:34.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:34.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:34.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:34.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:16:34.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:34.558 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:34.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:34.660 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:34.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:34.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:34.810 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:34.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:34.938 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:34.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:34.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:35.040 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:35.065 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:35.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:35.167 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:34 smithi118 ceph-mon[33862]: pgmap v635: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 41 active+undersized, 93 active+clean; 431 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 62/591 objects degraded (10.491%); 6/591 objects misplaced (1.015%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]': finished 2024-03-21T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:34 smithi118 ceph-mon[33862]: osdmap e311: 8 total, 7 up, 7 in 2024-03-21T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:34 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 62/591 objects degraded (10.491%), 17 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:34 smithi118 ceph-mon[33862]: osdmap e312: 8 total, 7 up, 7 in 2024-03-21T14:16:35.194 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[29576]: pgmap v635: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 41 active+undersized, 93 active+clean; 431 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 62/591 objects degraded (10.491%); 6/591 objects misplaced (1.015%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]': finished 2024-03-21T14:16:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[29576]: osdmap e311: 8 total, 7 up, 7 in 2024-03-21T14:16:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 62/591 objects degraded (10.491%), 17 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[29576]: osdmap e312: 8 total, 7 up, 7 in 2024-03-21T14:16:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[40510]: pgmap v635: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 41 active+undersized, 93 active+clean; 431 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.55k op/s; 62/591 objects degraded (10.491%); 6/591 objects misplaced (1.015%); 0 B/s, 0 objects/s recovering 2024-03-21T14:16:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.6165136924402305}]': finished 2024-03-21T14:16:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[40510]: osdmap e311: 8 total, 7 up, 7 in 2024-03-21T14:16:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 62/591 objects degraded (10.491%), 17 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:34 smithi003 ceph-mon[40510]: osdmap e312: 8 total, 7 up, 7 in 2024-03-21T14:16:35.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:35.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:35.423 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.451 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:35.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:35.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.579 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:35.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:35.680 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:35.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.708 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:35.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:35.809 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:35.838 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:35.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:35.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:35.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:35.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:35.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:36.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.098 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:36.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:35 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:16:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:35 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:16:36.200 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:36.227 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:36.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:35 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:16:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:35 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:16:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:35 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:16:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:35 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:16:36.329 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:36.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.356 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:36.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:36.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:36.485 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:36.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:36.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:36.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:36.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:36.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:16:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:16:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:16:36.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:36.872 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:36.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:36.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:36.919 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:36.962 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:36.970 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:36.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.001 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:37.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:37.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.130 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:37.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:36 smithi118 ceph-mon[33862]: pgmap v638: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 41 active+undersized, 93 active+clean; 431 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 731 op/s; 62/591 objects degraded (10.491%); 6/591 objects misplaced (1.015%) 2024-03-21T14:16:37.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:16:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:16:37.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:36 smithi003 ceph-mon[29576]: pgmap v638: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 41 active+undersized, 93 active+clean; 431 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 731 op/s; 62/591 objects degraded (10.491%); 6/591 objects misplaced (1.015%) 2024-03-21T14:16:37.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:36 smithi003 ceph-mon[40510]: pgmap v638: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 41 active+undersized, 93 active+clean; 431 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 731 op/s; 62/591 objects degraded (10.491%); 6/591 objects misplaced (1.015%) 2024-03-21T14:16:37.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:37.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:37.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:37.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:37.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:37.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:37.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:37.633 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:37.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:37.760 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:37.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:37.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:37.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:37.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:37.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:38.013 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:38.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:38.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:38.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:38.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:38.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:38.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:38.366 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:38.393 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:38.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:38.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:38.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:38.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:38.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:38.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:38.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:38.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:38.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:38.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:38.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:38.906 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:38.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:39.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:39.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:39.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:39.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:38 smithi118 ceph-mon[33862]: pgmap v639: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 44 active+undersized, 93 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 28 KiB/s wr, 556 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:38 smithi003 ceph-mon[29576]: pgmap v639: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 44 active+undersized, 93 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 28 KiB/s wr, 556 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:38 smithi003 ceph-mon[40510]: pgmap v639: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 44 active+undersized, 93 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 28 KiB/s wr, 556 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:39.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:39.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:39.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:39.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:39.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:39.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:39.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:39.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:39.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.675 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:39.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:39.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:39.793 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:39.801 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:39.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:39.903 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:39.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:39.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:40.029 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:40.057 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:40.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:40.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:39 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 62/594 objects degraded (10.438%), 14 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:40.182 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:40.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:40.209 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:39 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 62/594 objects degraded (10.438%), 14 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:40.209 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:39 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 62/594 objects degraded (10.438%), 14 pgs degraded, 40 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:40.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:40.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:40.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:40.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:40.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:40.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:40.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:40.565 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:40.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:40.694 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:40.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.820 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:40.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:40.850 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_2' removed 2024-03-21T14:16:40.921 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:40.947 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:40.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:41.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:41.073 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:41.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:40 smithi118 ceph-mon[33862]: pgmap v640: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 44 active+undersized, 93 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 27 KiB/s wr, 549 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:40 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2614116740' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:16:41.198 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:40 smithi003 ceph-mon[29576]: pgmap v640: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 44 active+undersized, 93 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 27 KiB/s wr, 549 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:40 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2614116740' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:16:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:40 smithi003 ceph-mon[40510]: pgmap v640: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 44 active+undersized, 93 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 27 KiB/s wr, 549 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2614116740' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:16:41.300 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.324 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:41.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:41.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:41.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:41.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:41.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:41.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:41.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:41.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:41.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:41.826 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:41.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:41.872 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:16:41.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:16:41.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:41.952 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:41.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:42.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:42.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:42.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2614116740' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:16:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:41 smithi118 ceph-mon[33862]: osdmap e313: 8 total, 7 up, 7 in 2024-03-21T14:16:42.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2614116740' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:16:42.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:41 smithi003 ceph-mon[29576]: osdmap e313: 8 total, 7 up, 7 in 2024-03-21T14:16:42.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2614116740' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:16:42.189 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:41 smithi003 ceph-mon[40510]: osdmap e313: 8 total, 7 up, 7 in 2024-03-21T14:16:42.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:42.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:42.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:42.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:42.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:42.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.455 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:42.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:42.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:42.579 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:42.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:42.657 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:16:42.657 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:16:42.680 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:42.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:42.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:42.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:42.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:42.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:43.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:43.081 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:43.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:42 smithi118 ceph-mon[33862]: pgmap v642: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 34 KiB/s, 0 objects/s recovering 2024-03-21T14:16:43.182 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:43.205 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:43.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:42 smithi003 ceph-mon[29576]: pgmap v642: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 34 KiB/s, 0 objects/s recovering 2024-03-21T14:16:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:42 smithi003 ceph-mon[40510]: pgmap v642: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 34 KiB/s, 0 objects/s recovering 2024-03-21T14:16:43.307 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:43.331 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:43.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:43.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:43.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:43.558 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:43.582 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:43.684 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:43.709 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:43.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.835 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:43.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:43.888 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:16:43.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:43.960 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:43.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[29576]: osdmap e314: 8 total, 7 up, 7 in 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3370074153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[40510]: osdmap e314: 8 total, 7 up, 7 in 2024-03-21T14:16:44.001 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:43 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3370074153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:16:44.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:44.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:43 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:16:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:43 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:16:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:43 smithi118 ceph-mon[33862]: osdmap e314: 8 total, 7 up, 7 in 2024-03-21T14:16:44.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:43 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3370074153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:16:44.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:44.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:44.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:44.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:44.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:44.466 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T14:16:44.467 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T14:16:44.467 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-21T14:16:44.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-21T14:16:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:44.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:44.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:44.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.596 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:44.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:44.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:44.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:44.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:44.847 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:44.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:44.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:44.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: pgmap v644: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3370074153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:16:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: osdmap e315: 8 total, 7 up, 7 in 2024-03-21T14:16:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1748921823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 62/594 objects degraded (10.438%), 14 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[40510]: osdmap e316: 8 total, 7 up, 7 in 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: pgmap v644: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3370074153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: osdmap e315: 8 total, 7 up, 7 in 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1748921823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 62/594 objects degraded (10.438%), 14 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:45.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:44 smithi003 ceph-mon[29576]: osdmap e316: 8 total, 7 up, 7 in 2024-03-21T14:16:45.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:45.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.102 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:45.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: pgmap v644: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%); 33 KiB/s, 0 objects/s recovering 2024-03-21T14:16:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3370074153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:16:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: osdmap e315: 8 total, 7 up, 7 in 2024-03-21T14:16:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1748921823' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:45.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 62/594 objects degraded (10.438%), 14 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:16:45.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:44 smithi118 ceph-mon[33862]: osdmap e316: 8 total, 7 up, 7 in 2024-03-21T14:16:45.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:45.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:16:45.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:16:45.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:16:45.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:16:45.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:16:45.216 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:16:45.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:16:45.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:45.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:45.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:45.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:45.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:45.481 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:45.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:45.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:45.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:45.732 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:45.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:45.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:45.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:16:45.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:46.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:46.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:46.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:46.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:16:46.322 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:16:46.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:46.361 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.386 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:46.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:46.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:46.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:46.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:46.611 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:16:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:16:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:16:46.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:46.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:46.743 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:46.769 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:46.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:46.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:46.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:46.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:47.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:47.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:16:47.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:47.150 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:46 smithi118 ceph-mon[33862]: pgmap v647: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:46 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:47.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:16:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:16:47.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:46 smithi003 ceph-mon[29576]: pgmap v647: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:47.187 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:46 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:47.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:46 smithi003 ceph-mon[40510]: pgmap v647: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:47.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:46 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:16:47.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:47.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:47.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:47.409 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:47.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:47.539 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:47.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:47.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:47.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:47.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:47.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:47.925 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:47.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:48.027 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:48.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:48.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:48.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:48.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:48.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:48.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:48.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:16:48.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:48.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:48.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:48.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:48.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:48.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:48.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:48.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:48.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:48.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:48.920 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:48.948 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:48.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:49.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:49.074 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:49.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:49.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:48 smithi003 ceph-mon[29576]: pgmap v648: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:49.156 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:49.156 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:48 smithi003 ceph-mon[40510]: pgmap v648: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:49.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:48 smithi118 ceph-mon[33862]: pgmap v648: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:49.204 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:49.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:49.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:49.333 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:49.354 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:16:49.354 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:16:49.354 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:16:49.355 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:16:49.355 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:16:49.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:49.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.463 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:49.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:49.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:49.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:49.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:49.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:49.718 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:49.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:49.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:49.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:49.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:49.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:50.045 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_3' created 2024-03-21T14:16:50.071 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:50.097 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:50.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2769334979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T14:16:50.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:50 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T14:16:50.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2769334979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T14:16:50.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:50 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T14:16:50.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:16:50.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:50.226 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:50.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:50.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2769334979' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T14:16:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:50 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-21T14:16:50.456 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:50.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.480 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:50.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:50.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:50.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:50.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:50.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:50.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.860 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:50.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:50.890 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-21T14:16:50.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:50.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:50.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:50.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:51.087 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:51.114 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:51.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[29576]: pgmap v649: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 769 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[29576]: osdmap e317: 8 total, 7 up, 7 in 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[29576]: osdmap e318: 8 total, 7 up, 7 in 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[40510]: pgmap v649: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 769 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[40510]: osdmap e317: 8 total, 7 up, 7 in 2024-03-21T14:16:51.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:51 smithi003 ceph-mon[40510]: osdmap e318: 8 total, 7 up, 7 in 2024-03-21T14:16:51.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:51.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:51.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:51.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:51.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:51 smithi118 ceph-mon[33862]: pgmap v649: 142 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 39 active+undersized, 82 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 769 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:51 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-21T14:16:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:51 smithi118 ceph-mon[33862]: osdmap e317: 8 total, 7 up, 7 in 2024-03-21T14:16:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:51 smithi118 ceph-mon[33862]: osdmap e318: 8 total, 7 up, 7 in 2024-03-21T14:16:51.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:51.496 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:51.574 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:51.598 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:51.607 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:51.626 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:51.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:51.755 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:51.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:51.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:51.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:51.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:51.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:52.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:52.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:52.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:52.131 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:52.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:52.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:52 smithi003 ceph-mon[29576]: pgmap v652: 158 pgs: 9 unknown, 7 active+clean+remapped, 14 active+undersized+degraded, 42 active+undersized, 86 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 740 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:52 smithi003 ceph-mon[29576]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:16:52.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:52 smithi003 ceph-mon[29576]: osdmap e319: 8 total, 7 up, 7 in 2024-03-21T14:16:52.356 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:52 smithi003 ceph-mon[40510]: pgmap v652: 158 pgs: 9 unknown, 7 active+clean+remapped, 14 active+undersized+degraded, 42 active+undersized, 86 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 740 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:52.356 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:52 smithi003 ceph-mon[40510]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:16:52.356 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:52 smithi003 ceph-mon[40510]: osdmap e319: 8 total, 7 up, 7 in 2024-03-21T14:16:52.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:52.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:52 smithi118 ceph-mon[33862]: pgmap v652: 158 pgs: 9 unknown, 7 active+clean+remapped, 14 active+undersized+degraded, 42 active+undersized, 86 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 740 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:52 smithi118 ceph-mon[33862]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:16:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:52 smithi118 ceph-mon[33862]: osdmap e319: 8 total, 7 up, 7 in 2024-03-21T14:16:52.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:52.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:52.617 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:52.643 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:52.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:52.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:52.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:52.871 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:52.898 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:52.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:16:52.999 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:53.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:53.124 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:53.152 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:53.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:53.281 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:53.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:53.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:53.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:53.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.540 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:53.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:53.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:53.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:53.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:53.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:53.794 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:53.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:53.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:53.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:53.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:53.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:54.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:54.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:54.152 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:54.176 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:54.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:54.305 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:54.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:54.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:54.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:54.602 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:54.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:54.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:54.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:54 smithi003 ceph-mon[29576]: pgmap v654: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:54.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:54 smithi003 ceph-mon[40510]: pgmap v654: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:54.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:54.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:54.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:54 smithi118 ceph-mon[33862]: pgmap v654: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:54.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:54.986 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:54.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:55.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:55.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:55.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:55.237 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:55.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:55.339 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:55.368 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:55.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:55.471 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:55.498 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:55.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:55.623 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:16:55.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:55.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:16:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:55.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:16:55.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:55.882 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:55.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:55 smithi118 ceph-mon[33862]: osdmap e320: 8 total, 7 up, 7 in 2024-03-21T14:16:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:55 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3653675190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:55 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:55.984 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:55.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:55 smithi003 ceph-mon[29576]: osdmap e320: 8 total, 7 up, 7 in 2024-03-21T14:16:55.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:55 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3653675190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:55.990 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:55 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:55.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:55 smithi003 ceph-mon[40510]: osdmap e320: 8 total, 7 up, 7 in 2024-03-21T14:16:55.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:55 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3653675190' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:55.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:55 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:16:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:56.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:56.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:56.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:56.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:56.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:56.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:56.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:56.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:56.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:56.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:56.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.637 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:56.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:56.674 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:16:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[29576]: pgmap v656: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 932 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[29576]: osdmap e321: 8 total, 7 up, 7 in 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[40510]: pgmap v656: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 932 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[40510]: osdmap e321: 8 total, 7 up, 7 in 2024-03-21T14:16:56.675 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:56 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:16:56.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:56.761 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:56.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:56.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:56.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:56 smithi118 ceph-mon[33862]: pgmap v656: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 932 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:56 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:16:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:56 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:16:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:56 smithi118 ceph-mon[33862]: osdmap e321: 8 total, 7 up, 7 in 2024-03-21T14:16:56.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:56 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:16:56.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:57.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:57.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:57.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:57.240 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:57.264 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:57.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:57.366 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:57.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:57.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:16:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:16:57.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:57.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:57.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:57.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:57.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:57.744 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:57.772 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:57.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:57.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:57 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:16:57.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:57 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:16:57.834 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:57 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:16:57.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:57 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:16:57.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:57 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:16:57.834 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:57 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:16:57.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:57.897 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:57.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:57 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:16:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:57 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:16:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:57 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:16:57.962 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219636 8.3 8:c2dee2e9:::benchmark_data_smithi003_110242_object88719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.40985784499998, 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99404183000000002, 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:57.963 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219636 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:57.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550757+0000", 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.02e-07 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550761+0000", 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0810000000000004e-06 2024-03-21T14:16:57.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550771+0000", 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4530000000000008e-06 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550778+0000", 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550827+0000", 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8690000000000003e-05 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550833+0000", 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1410000000000002e-06 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539253+0000", 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98841965700000001 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539380+0000", 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012699099999999999 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539666+0000", 2024-03-21T14:16:57.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028561599999999998 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.542830+0000", 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031648290000000001 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544576+0000", 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001746039 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544691+0000", 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011476000000000001 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544746+0000", 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4617999999999998e-05 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544798+0000", 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2462e-05 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219640 8.7 8:e03a6842:::benchmark_data_smithi003_110242_object88723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.39988806299999, 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.000192481, 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:57.971 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219640 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560728+0000", 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.548e-06 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560734+0000", 2024-03-21T14:16:57.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5189999999999998e-06 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560752+0000", 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7683999999999999e-05 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560764+0000", 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1558e-05 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560819+0000", 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5161000000000003e-05 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560826+0000", 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8390000000000001e-06 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551363+0000", 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99053792699999998 2024-03-21T14:16:57.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551445+0000", 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1199999999999995e-05 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551758+0000", 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031323899999999999 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558010+0000", 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062516619999999998 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558022+0000", 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2751e-05 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560813+0000", 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027905650000000001 2024-03-21T14:16:57.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560868+0000", 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5031999999999998e-05 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:57.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560919+0000", 2024-03-21T14:16:57.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0796e-05 2024-03-21T14:16:57.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219647 8.7 8:e325f337:::benchmark_data_smithi003_110242_object88730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.394490265, 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0098845869999999, 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:57.980 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219647 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:57.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566125+0000", 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7699999999999992e-07 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566131+0000", 2024-03-21T14:16:57.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5400000000000003e-06 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566142+0000", 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1029000000000001e-05 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566150+0000", 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8179999999999993e-06 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566259+0000", 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109064 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566268+0000", 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1850000000000004e-06 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554008+0000", 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98774052300000004 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554113+0000", 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104929 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554390+0000", 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027710000000000001 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560968+0000", 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065778310000000001 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569693+0000", 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087252330000000006 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575871+0000", 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061776979999999997 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575961+0000", 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9869000000000003e-05 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:57.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576009+0000", 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7790999999999997e-05 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219661 8.3 8:c78a9c88:::benchmark_data_smithi003_110242_object88744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.38289207299999, 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96728133199999999, 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:57.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219661 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577723+0000", 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.218e-06 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:57.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577729+0000", 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4469999999999997e-06 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577741+0000", 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1929e-05 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577749+0000", 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0420000000000003e-06 2024-03-21T14:16:57.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577810+0000", 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0661000000000001e-05 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577818+0000", 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5720000000000003e-06 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:57.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541402+0000", 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96358385700000004 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541513+0000", 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110608 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541765+0000", 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025253 2024-03-21T14:16:57.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544855+0000", 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030902249999999998 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544866+0000", 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0219e-05 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544900+0000", 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4406999999999998e-05 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544963+0000", 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3510999999999996e-05 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:57.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545004+0000", 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0105999999999999e-05 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219664 8.3 8:c1f39ff4:::benchmark_data_smithi003_110242_object88747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.38095679499997, 2024-03-21T14:16:57.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96554897500000003, 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219664 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:57.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579659+0000", 2024-03-21T14:16:57.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0979999999999999e-06 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579663+0000", 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6879999999999998e-06 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579675+0000", 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1701e-05 2024-03-21T14:16:57.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579684+0000", 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3670000000000003e-06 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579738+0000", 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3353999999999998e-05 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579746+0000", 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1349999999999992e-06 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.542881+0000", 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96313484800000004 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.543021+0000", 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014037899999999999 2024-03-21T14:16:58.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.543251+0000", 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000229776 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545019+0000", 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001768541 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545064+0000", 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4567999999999997e-05 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545127+0000", 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3037e-05 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545169+0000", 2024-03-21T14:16:58.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1650000000000003e-05 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545206+0000", 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7833000000000002e-05 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219669 8.7 8:e640187a:::benchmark_data_smithi003_110242_object88752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:16:58.004 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.37810483300001, 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99373624599999999, 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219669 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:16:58.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582510+0000", 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3500000000000005e-07 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582515+0000", 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6190000000000002e-06 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582527+0000", 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1663e-05 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582535+0000", 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9719999999999992e-06 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582589+0000", 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4258000000000001e-05 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582594+0000", 2024-03-21T14:16:58.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5650000000000001e-06 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556001+0000", 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97340653200000005 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556118+0000", 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116953 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556396+0000", 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027774400000000002 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569603+0000", 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013207429 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569729+0000", 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126198 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576017+0000", 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062882509999999999 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576174+0000", 2024-03-21T14:16:58.011 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015680300000000001 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576246+0000", 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1423999999999999e-05 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219677 8.7 8:e3d98eec:::benchmark_data_smithi003_110242_object88760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:58.012 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.37213361099998, 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98785003400000004, 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219677 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.013 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.014 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588482+0000", 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7999999999999993e-07 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588487+0000", 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2270000000000001e-06 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588498+0000", 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1511e-05 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.015 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588507+0000", 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6759999999999996e-06 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588580+0000", 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3046000000000005e-05 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.016 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588589+0000", 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8140000000000007e-06 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558336+0000", 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96974661200000001 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558447+0000", 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111644 2024-03-21T14:16:58.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558737+0000", 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028978899999999999 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569659+0000", 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010921759 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569764+0000", 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105388 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576252+0000", 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064880720000000001 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.019 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576293+0000", 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1300000000000001e-05 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576331+0000", 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7215999999999997e-05 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219708 8.7 8:e065f095:::benchmark_data_smithi003_110242_object88791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:16:58.020 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 287.343179276, 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96333364799999999, 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219708 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.021 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.022 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617436+0000", 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2160000000000001e-06 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617442+0000", 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4369999999999998e-06 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617453+0000", 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1109e-05 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617460+0000", 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6179999999999997e-06 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617520+0000", 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9939000000000003e-05 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617530+0000", 2024-03-21T14:16:58.024 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1679999999999993e-06 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561003+0000", 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94347382000000002 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561126+0000", 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012228799999999999 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561456+0000", 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033023800000000001 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575680+0000", 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014224647999999999 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.026 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575752+0000", 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1209000000000002e-05 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580677+0000", 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049251019999999998 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580721+0000", 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4159000000000002e-05 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580769+0000", 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7697000000000001e-05 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146548 9.0 9:0dd0b1d4:::benchmark_data_smithi003_135814_object5814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.717159+0000", 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.243455547, 2024-03-21T14:16:58.028 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.029799326, 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146548 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717159+0000", 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717159+0000", 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717160+0000", 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4890000000000001e-06 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.030 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717167+0000", 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1060000000000002e-06 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717180+0000", 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2979000000000001e-05 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717189+0000", 2024-03-21T14:16:58.031 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0010000000000007e-06 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.732497+0000", 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015307731999999999 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.732505+0000", 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4260000000000003e-06 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.032 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.741415+0000", 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0089090789999999 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.741552+0000", 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013709200000000001 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0", 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.741683+0000", 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013169899999999999 2024-03-21T14:16:58.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.745161+0000", 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003477876 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746810+0000", 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016483299999999999 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.034 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746868+0000", 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8176e-05 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746958+0000", 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0340999999999994e-05 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.035 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146549 9.2 9:444f3cf6:::benchmark_data_smithi003_135814_object5815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.243408393, 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.035862668, 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146549 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.036 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.037 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-07 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717209+0000", 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.915e-06 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717216+0000", 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9530000000000002e-06 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.038 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717222+0000", 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1330000000000004e-06 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.727749+0000", 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010526862999999999 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.729296+0000", 2024-03-21T14:16:58.039 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001546415 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.745813+0000", 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0165168 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746169+0000", 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035624300000000001 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.040 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746303+0000", 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013459600000000001 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750301+0000", 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039974140000000003 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.041 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.752966+0000", 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026654920000000002 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753001+0000", 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4690000000000002e-05 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.042 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753069+0000", 2024-03-21T14:16:58.043 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:58.044 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7613000000000005e-05 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146566 9.2 9:478951ba:::benchmark_data_smithi003_135814_object5832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.738747+0000", 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.221866965, 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0144947470000001, 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.045 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146566 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738747+0000", 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738747+0000", 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738749+0000", 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5570000000000001e-06 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.047 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738755+0000", 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0839999999999997e-06 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738766+0000", 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0776e-05 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738772+0000", 2024-03-21T14:16:58.048 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2129999999999996e-06 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738824+0000", 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1814000000000002e-05 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738830+0000", 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8089999999999998e-06 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.049 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.748308+0000", 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0094779439999999 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.748412+0000", 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104036 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.748522+0000", 2024-03-21T14:16:58.050 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011043 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.751822+0000", 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033004729999999999 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753120+0000", 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00129707 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.051 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753161+0000", 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1202000000000003e-05 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753242+0000", 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1339000000000002e-05 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.052 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146570 9.2 9:454dc127:::benchmark_data_smithi003_135814_object5836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.742486+0000", 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.218128284, 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0111842019999999, 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146570 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.053 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742486+0000", 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742486+0000", 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742487+0000", 2024-03-21T14:16:58.054 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.229e-06 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742492+0000", 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9459999999999997e-06 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742502+0000", 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8090000000000004e-06 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.055 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742508+0000", 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9020000000000004e-06 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742560+0000", 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2111000000000003e-05 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742567+0000", 2024-03-21T14:16:58.056 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0500000000000003e-06 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750335+0000", 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.007768346 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750438+0000", 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103065 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.057 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750506+0000", 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7508000000000005e-05 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.751919+0000", 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001412817 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.058 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753569+0000", 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016506050000000001 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753600+0000", 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0420999999999999e-05 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753670+0000", 2024-03-21T14:16:58.059 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0393000000000002e-05 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146575 9.0 9:09bb5e96:::benchmark_data_smithi003_135814_object5841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.750996+0000", 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 124.20961840299999, 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99901830800000002, 2024-03-21T14:16:58.060 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146575 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.750996+0000", 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.061 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.750996+0000", 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.750998+0000", 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.759e-06 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.062 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751002+0000", 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8230000000000002e-06 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751014+0000", 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1202e-05 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751020+0000", 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8640000000000001e-06 2024-03-21T14:16:58.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751064+0000", 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4823000000000001e-05 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751072+0000", 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2459999999999997e-06 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.064 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.747328+0000", 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.996256751 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.747439+0000", 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011100899999999999 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0", 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.747553+0000", 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011396999999999999 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.749343+0000", 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017892769999999999 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.749904+0000", 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056149800000000001 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.066 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.749945+0000", 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1069999999999998e-05 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750014+0000", 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9016000000000005e-05 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.067 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268614 9.f 9:f836142a:::benchmark_data_smithi003_135814_object127880:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863347+0000", 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.097266980999997, 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0564403689999999, 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268614 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.068 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863347+0000", 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863347+0000", 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863349+0000", 2024-03-21T14:16:58.069 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2589999999999999e-06 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863352+0000", 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8670000000000004e-06 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863362+0000", 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6830000000000003e-06 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.070 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863372+0000", 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8260000000000005e-06 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864619+0000", 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012469849999999999 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864629+0000", 2024-03-21T14:16:58.071 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0193999999999999e-05 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869854+0000", 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052248629999999997 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869864+0000", 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0397999999999999e-05 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.072 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.917770+0000", 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.047905265 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.919788+0000", 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020180290000000002 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.073 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268616 9.f 9:f6a2344d:::benchmark_data_smithi003_135814_object127882:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.097222453000001, 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0566032329999999, 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268616 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.074 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.075 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7899999999999999e-07 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863395+0000", 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8279999999999999e-06 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863402+0000", 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3709999999999996e-06 2024-03-21T14:16:58.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863407+0000", 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9529999999999996e-06 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864661+0000", 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00125344 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-03-21T14:16:58.077 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864667+0000", 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9050000000000002e-06 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869887+0000", 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052205400000000001 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869894+0000", 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6959999999999999e-06 2024-03-21T14:16:58.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.919906+0000", 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0500118490000001 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.919995+0000", 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9172000000000006e-05 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268618 9.6 9:6081da32:::benchmark_data_smithi003_135814_object127884:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.097199301000003, 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.047590408, 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.080 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268618 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:16:58.081 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.46e-07 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863418+0000", 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1940000000000001e-06 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.082 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863423+0000", 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4079999999999997e-06 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863429+0000", 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7649999999999999e-06 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.083 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864328+0000", 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089905200000000001 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864335+0000", 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0129999999999996e-06 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876237+0000", 2024-03-21T14:16:58.084 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011902233 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876247+0000", 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8160000000000005e-06 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.908396+0000", 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.032149414 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.085 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911005+0000", 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026091669999999999 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268622 9.6 9:654a3d90:::benchmark_data_smithi003_135814_object127888:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863436+0000", 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.097177840999997, 2024-03-21T14:16:58.086 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0484016620000001, 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268622 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863436+0000", 2024-03-21T14:16:58.087 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863436+0000", 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863437+0000", 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8299999999999998e-07 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.088 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863439+0000", 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1710000000000001e-06 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863445+0000", 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5140000000000001e-06 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863450+0000", 2024-03-21T14:16:58.089 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3009999999999997e-06 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864386+0000", 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00093625900000000005 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864392+0000", 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6679999999999998e-06 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.090 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876274+0000", 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011882622000000001 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876279+0000", 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0180000000000001e-06 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.091 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911726+0000", 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.035446259 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911838+0000", 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011246700000000001 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.092 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268634 9.b 9:d4935d58:::benchmark_data_smithi003_135814_object127900:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863456+0000", 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.097157881999998, 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0497115850000001, 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268634 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.093 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863456+0000", 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863456+0000", 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.094 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863457+0000", 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7099999999999997e-07 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863459+0000", 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4059999999999999e-06 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.095 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863465+0000", 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.327e-06 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863470+0000", 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0669999999999999e-06 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864720+0000", 2024-03-21T14:16:58.096 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00125074 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864726+0000", 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5990000000000001e-06 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.870446+0000", 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057203009999999997 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.870455+0000", 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1789999999999997e-06 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.902031+0000", 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0315752300000001 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.098 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.902166+0000", 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013569 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5", 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.902575+0000", 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040849799999999998 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913018+0000", 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010443295 2024-03-21T14:16:58.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913034+0000", 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5814000000000001e-05 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913086+0000", 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1659000000000001e-05 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.100 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913168+0000", 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2409000000000003e-05 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268638 9.6 9:651f0ab7:::benchmark_data_smithi003_135814_object127904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.864113+0000", 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.096501609999997, 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0622911719999999, 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.101 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268638 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864113+0000", 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.102 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864113+0000", 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864114+0000", 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7700000000000003e-07 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864118+0000", 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.831e-06 2024-03-21T14:16:58.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864129+0000", 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0737000000000001e-05 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864138+0000", 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5420000000000005e-06 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.104 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864623+0000", 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048560200000000002 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864631+0000", 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9389999999999997e-06 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876315+0000", 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011683567000000001 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876320+0000", 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7609999999999998e-06 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911865+0000", 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0355454129999999 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.912392+0000", 2024-03-21T14:16:58.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052676199999999996 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5", 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.912628+0000", 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023668499999999999 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926185+0000", 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013556456 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926228+0000", 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3429e-05 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926310+0000", 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1829999999999994e-05 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926404+0000", 2024-03-21T14:16:58.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3740999999999996e-05 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268641 9.b 9:dd564974:::benchmark_data_smithi003_135814_object127907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.877108+0000", 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.083505875, 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0432787109999999, 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:16:58.109 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268641 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877108+0000", 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877108+0000", 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877110+0000", 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.668e-06 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877118+0000", 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4699999999999996e-06 2024-03-21T14:16:58.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877127+0000", 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9329999999999996e-06 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877134+0000", 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7229999999999998e-06 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877185+0000", 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0804999999999999e-05 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877193+0000", 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913205+0000", 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0360120150000001 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913311+0000", 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105954 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5", 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913425+0000", 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113713 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:16:58.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920031+0000", 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066069259999999999 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920240+0000", 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020825800000000001 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920294+0000", 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4106e-05 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:16:58.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920387+0000", 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3314999999999997e-05 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:16:58.116 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:16:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.118 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:58.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:58.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:58.246 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:58.347 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.375 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:58.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:58.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:58.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:58.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:58.624 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-21T14:16:58.630 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:58.731 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:58.760 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:58.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:58.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:58.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:16:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:58 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:58 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:16:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:58 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:16:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:58 smithi118 ceph-mon[33862]: pgmap v658: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2263083343' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:16:58.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:58 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:16:58.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:16:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[29576]: pgmap v658: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2263083343' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[40510]: pgmap v658: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2263083343' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:16:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:58 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:16:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:59.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:59.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T14:16:59.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:16:59.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:59.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:16:59.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:16:59.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:16:59.363 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:16:59.383 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:16:59.395 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:59.403 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:16:59.427 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:16:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.428 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:59.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:59.444 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:16:59.465 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:16:59.488 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:16:59.488 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:16:59.488 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:16:59.488 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:16:59.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:16:59.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:16:59.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:16:59.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:16:59.489 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:16:59.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:16:59.561 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:16:59.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:16:59.663 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:16:59.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:16:59.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:16:59.825 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:16:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:16:59.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: 1.2 scrub starts 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: 1.2 scrub ok 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: osdmap e322: 8 total, 7 up, 7 in 2024-03-21T14:16:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:16:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/223150364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:16:59.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:16:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:16:59.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:17:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:17:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:17:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:17:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: 1.2 scrub starts 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: 1.2 scrub ok 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: osdmap e322: 8 total, 7 up, 7 in 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/223150364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: 1.2 scrub starts 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: 1.2 scrub ok 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:17:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: osdmap e322: 8 total, 7 up, 7 in 2024-03-21T14:17:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:16:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/223150364' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:00.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:00.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.082 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:00.184 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:00.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:00.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:00.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:00.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:00.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:00.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:00.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:00.595 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:00.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:00.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[29576]: pgmap v660: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[29576]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:17:00.727 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:17:00.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[40510]: pgmap v660: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:17:00.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[40510]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:17:00.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:17:00.728 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:00 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:17:00.734 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:00.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:17:00.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:17:00.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:00.854 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:00 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:17:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:00 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:17:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:00 smithi118 ceph-mon[33862]: pgmap v660: 158 pgs: 7 active+clean+remapped, 14 active+undersized+degraded, 47 active+undersized, 90 active+clean; 483 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 852 B/s rd, 0 op/s; 62/594 objects degraded (10.438%); 6/594 objects misplaced (1.010%) 2024-03-21T14:17:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:00 smithi118 ceph-mon[33862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:17:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:00 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:17:00.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:00 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:17:00.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:00.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:00.983 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:00.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:01.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:01.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:01.122 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-21T14:17:01.122 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-21T14:17:01.122 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-21T14:17:01.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:17:01.125 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T14:17:01.125 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T14:17:01.200 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-03-21T14:17:01.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T14:17:01.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_160672 2024-03-21T14:17:01.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:17:01.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-03-21T14:17:01.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:01.250 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:01.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:01.352 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:01.389 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:01.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:01.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:01.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:01.663 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:01.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:01.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:01.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:01.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:01.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:01.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:01.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:02.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:17:02.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:17:02.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 281/2058 objects degraded (13.654%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:02.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[29576]: osdmap e323: 8 total, 7 up, 7 in 2024-03-21T14:17:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:17:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:17:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 281/2058 objects degraded (13.654%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:02.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:01 smithi003 ceph-mon[40510]: osdmap e323: 8 total, 7 up, 7 in 2024-03-21T14:17:02.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:02.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:02.165 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:01 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:17:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:01 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:17:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:01 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 281/2058 objects degraded (13.654%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:01 smithi118 ceph-mon[33862]: osdmap e323: 8 total, 7 up, 7 in 2024-03-21T14:17:02.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:02.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 16 3924 3908 244.237 244.25 0.00474393 0.00407948 2024-03-21T14:17:02.224 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:02.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:02.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:02.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:02.465 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:02.501 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:02.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:02.603 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.641 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:02.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:02.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:02.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:17:02.772 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:02.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:02.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:02.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:02.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:17:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:17:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[29576]: 1.6 scrub starts 2024-03-21T14:17:03.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[29576]: 1.6 scrub ok 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[29576]: pgmap v661: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 44 active+undersized, 90 active+clean; 31 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 4.4 MiB/s wr, 68 op/s; 281/2058 objects degraded (13.654%); 6/2058 objects misplaced (0.292%) 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[40510]: 1.6 scrub starts 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[40510]: 1.6 scrub ok 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[40510]: pgmap v661: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 44 active+undersized, 90 active+clean; 31 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 4.4 MiB/s wr, 68 op/s; 281/2058 objects degraded (13.654%); 6/2058 objects misplaced (0.292%) 2024-03-21T14:17:03.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:03.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:03.054 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:03.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:02 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:17:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:02 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:17:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:02 smithi118 ceph-mon[33862]: 1.6 scrub starts 2024-03-21T14:17:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:02 smithi118 ceph-mon[33862]: 1.6 scrub ok 2024-03-21T14:17:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:02 smithi118 ceph-mon[33862]: pgmap v661: 158 pgs: 7 active+clean+remapped, 17 active+undersized+degraded, 44 active+undersized, 90 active+clean; 31 MiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 4.4 MiB/s wr, 68 op/s; 281/2058 objects degraded (13.654%); 6/2058 objects misplaced (0.292%) 2024-03-21T14:17:03.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:03.193 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:03.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 16 7577 7561 236.25 228.312 0.00436767 0.00422593 2024-03-21T14:17:03.257 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:17:03.257 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:17:03.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:03.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:03.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:03.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:03.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:03.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:03.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.762 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:03.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:03.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:03.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:03.901 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:03.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:03.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:17:03.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:17:03.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:17:03.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:17:03.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:17:03.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:17:03.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:17:03.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:03 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:17:04.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:04.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:04.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:04.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:03 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:17:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:03 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:17:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:03 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:17:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:03 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:17:04.199 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:04.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 11275 11259 234.535 231.125 0.00250849 0.00425534 2024-03-21T14:17:04.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:04.337 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:04.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:04.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:04.474 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:04.576 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:04.612 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:04.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:04.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:04.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:04.883 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:04.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:04.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:04.936 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T14:17:04.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:17:04.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:17:04.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[29576]: pgmap v663: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 158 MiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 20 MiB/s wr, 315 op/s; 1206/8136 objects degraded (14.823%); 6/8136 objects misplaced (0.074%) 2024-03-21T14:17:04.976 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[29576]: osdmap e324: 8 total, 7 up, 7 in 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[40510]: pgmap v663: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 158 MiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 20 MiB/s wr, 315 op/s; 1206/8136 objects degraded (14.823%); 6/8136 objects misplaced (0.074%) 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:17:04.977 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:04 smithi003 ceph-mon[40510]: osdmap e324: 8 total, 7 up, 7 in 2024-03-21T14:17:04.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:05.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:05.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:05.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:05.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:05.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:04 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:17:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:04 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:17:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:04 smithi118 ceph-mon[33862]: pgmap v663: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 158 MiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 20 MiB/s wr, 315 op/s; 1206/8136 objects degraded (14.823%); 6/8136 objects misplaced (0.074%) 2024-03-21T14:17:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:04 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:17:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:04 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:17:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:04 smithi118 ceph-mon[33862]: osdmap e324: 8 total, 7 up, 7 in 2024-03-21T14:17:05.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 14511 14495 226.459 202.25 0.00276639 0.00440909 2024-03-21T14:17:05.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.302 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:05.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:05.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:05.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:05.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.578 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:05.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:05.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:05.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:05.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.857 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:05.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:05.958 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:05.994 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:05.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:05.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:06.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:05 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:17:06.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:05 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:17:06.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:06.131 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:06.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:05 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:17:06.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 17737 17721 221.489 201.625 0.00425488 0.00450983 2024-03-21T14:17:06.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:06.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:06.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:06.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:06.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:06.413 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:06.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:06.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:06.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:06.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:06.677 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:06.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:06.711 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:17:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:17:06.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:06.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:17:06.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:17:06.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:17:06.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:17:06.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:17:06.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: pgmap v665: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 158 MiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 23 MiB/s wr, 361 op/s; 1206/8136 objects degraded (14.823%); 6/8136 objects misplaced (0.074%) 2024-03-21T14:17:06.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:17:06.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: pgmap v665: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 158 MiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 23 MiB/s wr, 361 op/s; 1206/8136 objects degraded (14.823%); 6/8136 objects misplaced (0.074%) 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:17:06.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:06 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:17:07.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:17:07.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:17:07.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:17:07.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:17:07.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:17:07.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: pgmap v665: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 158 MiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 23 MiB/s wr, 361 op/s; 1206/8136 objects degraded (14.823%); 6/8136 objects misplaced (0.074%) 2024-03-21T14:17:07.111 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:17:07.111 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:06 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:17:07.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:07.201 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 15 20589 20574 214.29 178.312 0.00550963 0.0046617 2024-03-21T14:17:07.248 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:07.285 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:07.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:07.431 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:07.431 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:07.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:17:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:17:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:07.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:07.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:07.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:07.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:07.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.707 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:07.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:17:07.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:07.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:07.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:07.905 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:17:07.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:17:07.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2527210795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:07.906 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:07.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:17:07.906 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:17:07.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2527210795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:07.907 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:07 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:07.912 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:17:07.917 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:17:07.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:07.983 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:07.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:08.085 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:08.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:08.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:08.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:07 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:17:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:07 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:17:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2527210795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:07 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:08.202 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 23056 23040 205.692 154.125 0.00177845 0.00480294 2024-03-21T14:17:08.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:08.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:08.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:08.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:08.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:08.385 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:08.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:08.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:08.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:08.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:08.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:08.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:08.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:08.968 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:08.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: pgmap v666: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 152 MiB/s wr, 2.43k op/s; 9635/59013 objects degraded (16.327%); 6/59013 objects misplaced (0.010%) 2024-03-21T14:17:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 9635/59013 objects degraded (16.327%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:09.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: osdmap e325: 8 total, 7 up, 7 in 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[29576]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: pgmap v666: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 152 MiB/s wr, 2.43k op/s; 9635/59013 objects degraded (16.327%); 6/59013 objects misplaced (0.010%) 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 9635/59013 objects degraded (16.327%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: osdmap e325: 8 total, 7 up, 7 in 2024-03-21T14:17:09.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:17:09.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:08 smithi003 ceph-mon[40510]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:17:09.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:09.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: pgmap v666: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 152 MiB/s wr, 2.43k op/s; 9635/59013 objects degraded (16.327%); 6/59013 objects misplaced (0.010%) 2024-03-21T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 9635/59013 objects degraded (16.327%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: osdmap e325: 8 total, 7 up, 7 in 2024-03-21T14:17:09.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:17:09.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:08 smithi118 ceph-mon[33862]: from='client.15678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:17:09.202 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 25490 25474 198.976 152.125 0.00480705 0.0050206 2024-03-21T14:17:09.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:09.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:09.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:09.378 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:09.479 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:09.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:09.616 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:09.655 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:17:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:17:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:17:09.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[29576]: osdmap e326: 8 total, 7 up, 7 in 2024-03-21T14:17:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:17:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:17:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:17:09.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:09 smithi003 ceph-mon[40510]: osdmap e326: 8 total, 7 up, 7 in 2024-03-21T14:17:09.757 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:09.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:09.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:09.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:09.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:09.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:10.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:10.085 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:09 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:17:10.085 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:09 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:17:10.086 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:09 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:17:10.086 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:09 smithi118 ceph-mon[33862]: osdmap e326: 8 total, 7 up, 7 in 2024-03-21T14:17:10.202 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 28026 28010 194.478 158.5 0.00646312 0.00513618 2024-03-21T14:17:10.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:10.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:10.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:10.259 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":667,"stamp":"2024-03-21T14:17:07.725219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1276689107,"num_objects":19671,"num_object_clones":0,"num_object_copies":59013,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9635,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":19671,"num_whiteouts":0,"num_read":1976,"num_read_kb":1745,"num_write":20228,"num_write_kb":1246819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":432208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21755,"ondisk_log_size":21755,"up":406,"acting":413,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":466,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9473980,"kb_used_data":5785020,"kb_used_omap":10,"kb_used_meta":3688885,"kb_avail":646742084,"statfs":{"total":671965249536,"available":662263894016,"internally_reserved":0,"allocated":5923860480,"data_stored":5919573091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":3777418346},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1244200960,"num_objects":18985,"num_object_clones":0,"num_object_copies":56955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9354,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18985,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18985,"num_write_kb":1215040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.193845"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":561,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.617626+0000","last_change":"2024-03-21T14:16:18.479867+0000","last_active":"2024-03-21T14:17:04.617626+0000","last_peered":"2024-03-21T14:17:04.617626+0000","last_clean":"2024-03-21T14:17:04.617626+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:17:04.617626+0000","last_undegraded":"2024-03-21T14:17:04.617626+0000","last_fullsized":"2024-03-21T14:17:04.617626+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:18.479698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:18.479698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:20.878647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":581,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626530+0000","last_change":"2024-03-21T14:16:13.706591+0000","last_active":"2024-03-21T14:17:04.626530+0000","last_peered":"2024-03-21T14:17:04.626530+0000","last_clean":"2024-03-21T14:17:04.626530+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:17:04.626530+0000","last_undegraded":"2024-03-21T14:17:04.626530+0000","last_fullsized":"2024-03-21T14:17:04.626530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:13.706475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.706475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.837131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":384,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.617721+0000","last_change":"2024-03-21T14:16:21.861900+0000","last_active":"2024-03-21T14:17:04.617721+0000","last_peered":"2024-03-21T14:17:04.617721+0000","last_clean":"2024-03-21T14:14:52.709607+0000","last_became_active":"2024-03-21T14:16:21.861900+0000","last_became_peered":"2024-03-21T14:16:21.861900+0000","last_unstale":"2024-03-21T14:17:04.617721+0000","last_undegraded":"2024-03-21T14:17:04.617721+0000","last_fullsized":"2024-03-21T14:16:21.853133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":606,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.617438+0000","last_change":"2024-03-21T14:14:54.733655+0000","last_active":"2024-03-21T14:17:04.617438+0000","last_peered":"2024-03-21T14:17:04.617438+0000","last_clean":"2024-03-21T14:14:52.715690+0000","last_became_active":"2024-03-21T14:14:54.733655+0000","last_became_peered":"2024-03-21T14:14:54.733655+0000","last_unstale":"2024-03-21T14:17:04.617438+0000","last_undegraded":"2024-03-21T14:14:54.724620+0000","last_fullsized":"2024-03-21T14:14:54.724501+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:12:52.533687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:52.533687+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":435,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.622970+0000","last_change":"2024-03-21T14:14:54.740287+0000","last_active":"2024-03-21T14:17:04.622970+0000","last_peered":"2024-03-21T14:17:04.622970+0000","last_clean":"2024-03-21T14:14:52.711070+0000","last_became_active":"2024-03-21T14:14:54.740287+0000","last_became_peered":"2024-03-21T14:14:54.740287+0000","last_unstale":"2024-03-21T14:17:04.622970+0000","last_undegraded":"2024-03-21T14:17:04.622970+0000","last_fullsized":"2024-03-21T14:14:54.732333+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:49.783238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:49.783238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686737+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":574,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.616989+0000","last_change":"2024-03-21T14:14:54.733806+0000","last_active":"2024-03-21T14:17:04.616989+0000","last_peered":"2024-03-21T14:17:04.616989+0000","last_clean":"2024-03-21T14:14:52.715590+0000","last_became_active":"2024-03-21T14:14:54.733806+0000","last_became_peered":"2024-03-21T14:14:54.733806+0000","last_unstale":"2024-03-21T14:17:04.616989+0000","last_undegraded":"2024-03-21T14:17:04.616989+0000","last_fullsized":"2024-03-21T14:14:54.723805+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347548+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.627988+0000","last_change":"2024-03-21T14:14:54.731504+0000","last_active":"2024-03-21T14:17:04.627988+0000","last_peered":"2024-03-21T14:17:04.627988+0000","last_clean":"2024-03-21T14:14:52.721005+0000","last_became_active":"2024-03-21T14:14:54.731504+0000","last_became_peered":"2024-03-21T14:14:54.731504+0000","last_unstale":"2024-03-21T14:17:04.627988+0000","last_undegraded":"2024-03-21T14:17:04.627988+0000","last_fullsized":"2024-03-21T14:14:54.723631+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.147407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.147407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.761994+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":622,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.626656+0000","last_change":"2024-03-21T14:14:54.733569+0000","last_active":"2024-03-21T14:17:04.626656+0000","last_peered":"2024-03-21T14:17:04.626656+0000","last_clean":"2024-03-21T14:14:52.725358+0000","last_became_active":"2024-03-21T14:14:54.733569+0000","last_became_peered":"2024-03-21T14:14:54.733569+0000","last_unstale":"2024-03-21T14:17:04.626656+0000","last_undegraded":"2024-03-21T14:14:54.724152+0000","last_fullsized":"2024-03-21T14:14:54.724039+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:12:48.244225+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:12:48.244225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.762022+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":430,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624861+0000","last_change":"2024-03-21T14:14:54.746386+0000","last_active":"2024-03-21T14:17:04.624861+0000","last_peered":"2024-03-21T14:17:04.624861+0000","last_clean":"2024-03-21T14:14:52.715750+0000","last_became_active":"2024-03-21T14:14:54.746386+0000","last_became_peered":"2024-03-21T14:14:54.746386+0000","last_unstale":"2024-03-21T14:17:04.624861+0000","last_undegraded":"2024-03-21T14:17:04.624861+0000","last_fullsized":"2024-03-21T14:14:54.729653+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.510830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.510830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":572,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.623039+0000","last_change":"2024-03-21T14:16:25.570159+0000","last_active":"2024-03-21T14:17:04.623039+0000","last_peered":"2024-03-21T14:17:04.623039+0000","last_clean":"2024-03-21T14:17:04.623039+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:17:04.623039+0000","last_undegraded":"2024-03-21T14:17:04.623039+0000","last_fullsized":"2024-03-21T14:17:04.623039+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:25.570054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:16:25.570054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:44.999312+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":382,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.617733+0000","last_change":"2024-03-21T14:16:21.871252+0000","last_active":"2024-03-21T14:17:04.617733+0000","last_peered":"2024-03-21T14:17:04.617733+0000","last_clean":"2024-03-21T14:17:04.617733+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:17:04.617733+0000","last_undegraded":"2024-03-21T14:17:04.617733+0000","last_fullsized":"2024-03-21T14:17:04.617733+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:57.333382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:14:57.333382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":586,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618426+0000","last_change":"2024-03-21T14:14:49.050746+0000","last_active":"2024-03-21T14:17:04.618426+0000","last_peered":"2024-03-21T14:17:04.618426+0000","last_clean":"2024-03-21T14:17:04.618426+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:17:04.618426+0000","last_undegraded":"2024-03-21T14:17:04.618426+0000","last_fullsized":"2024-03-21T14:17:04.618426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:14:49.050627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:49.050627+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":519,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.626845+0000","last_change":"2024-03-21T14:16:21.867853+0000","last_active":"2024-03-21T14:17:04.626845+0000","last_peered":"2024-03-21T14:17:04.626845+0000","last_clean":"2024-03-21T14:14:52.705556+0000","last_became_active":"2024-03-21T14:16:21.867853+0000","last_became_peered":"2024-03-21T14:16:21.867853+0000","last_unstale":"2024-03-21T14:17:04.626845+0000","last_undegraded":"2024-03-21T14:17:04.626845+0000","last_fullsized":"2024-03-21T14:16:21.860202+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:41.341673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.341673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":573,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.623320+0000","last_change":"2024-03-21T14:16:22.034648+0000","last_active":"2024-03-21T14:17:04.623320+0000","last_peered":"2024-03-21T14:17:04.623320+0000","last_clean":"2024-03-21T14:17:04.623320+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:17:04.623320+0000","last_undegraded":"2024-03-21T14:17:04.623320+0000","last_fullsized":"2024-03-21T14:17:04.623320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:22.033938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:22.033938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:26.975170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":48,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618248+0000","last_change":"2024-03-21T14:16:21.862467+0000","last_active":"2024-03-21T14:17:04.618248+0000","last_peered":"2024-03-21T14:17:04.618248+0000","last_clean":"2024-03-21T14:17:04.618248+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:17:04.618248+0000","last_undegraded":"2024-03-21T14:17:04.618248+0000","last_fullsized":"2024-03-21T14:17:04.618248+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:54.278647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:54.278647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":582,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.621461+0000","last_change":"2024-03-21T14:15:38.173272+0000","last_active":"2024-03-21T14:17:04.621461+0000","last_peered":"2024-03-21T14:17:04.621461+0000","last_clean":"2024-03-21T14:17:04.621461+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:17:04.621461+0000","last_undegraded":"2024-03-21T14:17:04.621461+0000","last_fullsized":"2024-03-21T14:17:04.621461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:15:38.173089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.173089+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:06.425181+0000","last_change":"2024-03-21T14:17:06.425215+0000","last_active":"2024-03-21T14:17:06.425181+0000","last_peered":"2024-03-21T14:17:06.425181+0000","last_clean":"2024-03-21T14:17:06.425181+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:17:06.425181+0000","last_undegraded":"2024-03-21T14:17:06.425181+0000","last_fullsized":"2024-03-21T14:17:06.425181+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:06.425104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:06.425104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:21.128928+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":573,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627249+0000","last_change":"2024-03-21T14:15:29.462169+0000","last_active":"2024-03-21T14:17:04.627249+0000","last_peered":"2024-03-21T14:17:04.627249+0000","last_clean":"2024-03-21T14:17:04.627249+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:17:04.627249+0000","last_undegraded":"2024-03-21T14:17:04.627249+0000","last_fullsized":"2024-03-21T14:17:04.627249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:29.462055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:29.462055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":426,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.621296+0000","last_change":"2024-03-21T14:15:37.108767+0000","last_active":"2024-03-21T14:17:04.621296+0000","last_peered":"2024-03-21T14:17:04.621296+0000","last_clean":"2024-03-21T14:17:04.621296+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:17:04.621296+0000","last_undegraded":"2024-03-21T14:17:04.621296+0000","last_fullsized":"2024-03-21T14:17:04.621296+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.108651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'27","reported_seq":549,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.628305+0000","last_change":"2024-03-21T14:14:54.733568+0000","last_active":"2024-03-21T14:17:04.628305+0000","last_peered":"2024-03-21T14:17:04.628305+0000","last_clean":"2024-03-21T14:12:42.334846+0000","last_became_active":"2024-03-21T14:14:54.733568+0000","last_became_peered":"2024-03-21T14:14:54.733568+0000","last_unstale":"2024-03-21T14:17:04.628305+0000","last_undegraded":"2024-03-21T14:14:54.727590+0000","last_fullsized":"2024-03-21T14:14:54.727474+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:14:48.508532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:48.508532+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:05.017808+0000","last_change":"2024-03-21T14:17:05.017847+0000","last_active":"2024-03-21T14:17:05.017808+0000","last_peered":"2024-03-21T14:17:05.017808+0000","last_clean":"2024-03-21T14:17:05.017808+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:17:05.017808+0000","last_undegraded":"2024-03-21T14:17:05.017808+0000","last_fullsized":"2024-03-21T14:17:05.017808+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.017731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.017731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:13.574877+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.618831+0000","last_change":"2024-03-21T14:14:54.732684+0000","last_active":"2024-03-21T14:17:04.618831+0000","last_peered":"2024-03-21T14:17:04.618831+0000","last_clean":"2024-03-21T14:14:52.716513+0000","last_became_active":"2024-03-21T14:14:54.732684+0000","last_became_peered":"2024-03-21T14:14:54.732684+0000","last_unstale":"2024-03-21T14:17:04.618831+0000","last_undegraded":"2024-03-21T14:17:04.618831+0000","last_fullsized":"2024-03-21T14:14:54.724942+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.679529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.679529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":568,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.618676+0000","last_change":"2024-03-21T14:14:54.738998+0000","last_active":"2024-03-21T14:17:04.618676+0000","last_peered":"2024-03-21T14:17:04.618676+0000","last_clean":"2024-03-21T14:14:52.716365+0000","last_became_active":"2024-03-21T14:14:54.738998+0000","last_became_peered":"2024-03-21T14:14:54.738998+0000","last_unstale":"2024-03-21T14:17:04.618676+0000","last_undegraded":"2024-03-21T14:17:04.618676+0000","last_fullsized":"2024-03-21T14:14:54.726991+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:49.866989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.866989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347393+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":609,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627762+0000","last_change":"2024-03-21T14:15:26.792016+0000","last_active":"2024-03-21T14:17:04.627762+0000","last_peered":"2024-03-21T14:17:04.627762+0000","last_clean":"2024-03-21T14:17:04.627762+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:17:04.627762+0000","last_undegraded":"2024-03-21T14:17:04.627762+0000","last_fullsized":"2024-03-21T14:17:04.627762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:15:26.791862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:26.791862+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":574,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.624960+0000","last_change":"2024-03-21T14:14:54.743535+0000","last_active":"2024-03-21T14:17:04.624960+0000","last_peered":"2024-03-21T14:17:04.624960+0000","last_clean":"2024-03-21T14:14:52.711655+0000","last_became_active":"2024-03-21T14:14:54.743535+0000","last_became_peered":"2024-03-21T14:14:54.743535+0000","last_unstale":"2024-03-21T14:17:04.624960+0000","last_undegraded":"2024-03-21T14:14:54.725946+0000","last_fullsized":"2024-03-21T14:14:54.725819+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.547775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686798+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"298'10","reported_seq":119,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.620217+0000","last_change":"2024-03-21T14:16:59.343397+0000","last_active":"2024-03-21T14:17:04.620217+0000","last_peered":"2024-03-21T14:17:04.620217+0000","last_clean":"2024-03-21T14:17:04.620217+0000","last_became_active":"2024-03-21T14:16:00.605304+0000","last_became_peered":"2024-03-21T14:16:00.605304+0000","last_unstale":"2024-03-21T14:17:04.620217+0000","last_undegraded":"2024-03-21T14:17:04.620217+0000","last_fullsized":"2024-03-21T14:17:04.620217+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'10","last_scrub_stamp":"2024-03-21T14:16:59.343213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:59.343213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:05.718685+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626471+0000","last_change":"2024-03-21T14:14:59.645994+0000","last_active":"2024-03-21T14:17:04.626471+0000","last_peered":"2024-03-21T14:17:04.626471+0000","last_clean":"2024-03-21T14:17:04.626471+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:17:04.626471+0000","last_undegraded":"2024-03-21T14:17:04.626471+0000","last_fullsized":"2024-03-21T14:17:04.626471+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:14:59.645838+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:14:59.645838+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":271,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.625204+0000","last_change":"2024-03-21T14:14:54.738663+0000","last_active":"2024-03-21T14:17:04.625204+0000","last_peered":"2024-03-21T14:17:04.625204+0000","last_clean":"2024-03-21T14:14:52.711713+0000","last_became_active":"2024-03-21T14:14:54.738663+0000","last_became_peered":"2024-03-21T14:14:54.738663+0000","last_unstale":"2024-03-21T14:17:04.625204+0000","last_undegraded":"2024-03-21T14:14:54.726825+0000","last_fullsized":"2024-03-21T14:14:54.726705+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:13:41.601807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.601807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686767+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626965+0000","last_change":"2024-03-21T14:15:27.529054+0000","last_active":"2024-03-21T14:17:04.626965+0000","last_peered":"2024-03-21T14:17:04.626965+0000","last_clean":"2024-03-21T14:17:04.626965+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:17:04.626965+0000","last_undegraded":"2024-03-21T14:17:04.626965+0000","last_fullsized":"2024-03-21T14:17:04.626965+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:27.528922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:27.528922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":223,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625196+0000","last_change":"2024-03-21T14:14:54.739116+0000","last_active":"2024-03-21T14:17:04.625196+0000","last_peered":"2024-03-21T14:17:04.625196+0000","last_clean":"2024-03-21T14:14:52.711613+0000","last_became_active":"2024-03-21T14:14:54.739116+0000","last_became_peered":"2024-03-21T14:14:54.739116+0000","last_unstale":"2024-03-21T14:17:04.625196+0000","last_undegraded":"2024-03-21T14:17:04.625196+0000","last_fullsized":"2024-03-21T14:14:54.732137+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.819350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.819350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686724+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625910+0000","last_change":"2024-03-21T14:14:54.733221+0000","last_active":"2024-03-21T14:17:04.625910+0000","last_peered":"2024-03-21T14:17:04.625910+0000","last_clean":"2024-03-21T14:14:52.721722+0000","last_became_active":"2024-03-21T14:14:54.733221+0000","last_became_peered":"2024-03-21T14:14:54.733221+0000","last_unstale":"2024-03-21T14:17:04.625910+0000","last_undegraded":"2024-03-21T14:17:04.625910+0000","last_fullsized":"2024-03-21T14:14:54.723133+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:59.621530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.621530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.762062+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":305,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627028+0000","last_change":"2024-03-21T14:16:21.869793+0000","last_active":"2024-03-21T14:17:04.627028+0000","last_peered":"2024-03-21T14:17:04.627028+0000","last_clean":"2024-03-21T14:17:04.627028+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:17:04.627028+0000","last_undegraded":"2024-03-21T14:17:04.627028+0000","last_fullsized":"2024-03-21T14:17:04.627028+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:14:59.343608+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:14:59.343608+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":510,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.617241+0000","last_change":"2024-03-21T14:16:34.611932+0000","last_active":"2024-03-21T14:17:04.617241+0000","last_peered":"2024-03-21T14:17:04.617241+0000","last_clean":"2024-03-21T14:14:52.711440+0000","last_became_active":"2024-03-21T14:16:34.611932+0000","last_became_peered":"2024-03-21T14:16:34.611932+0000","last_unstale":"2024-03-21T14:17:04.617241+0000","last_undegraded":"2024-03-21T14:17:04.617241+0000","last_fullsized":"2024-03-21T14:16:34.605066+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:49.859204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:49.859204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":256,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.616994+0000","last_change":"2024-03-21T14:16:19.440863+0000","last_active":"2024-03-21T14:17:04.616994+0000","last_peered":"2024-03-21T14:17:04.616994+0000","last_clean":"2024-03-21T14:17:04.616994+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:17:04.616994+0000","last_undegraded":"2024-03-21T14:17:04.616994+0000","last_fullsized":"2024-03-21T14:17:04.616994+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:19.440733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:19.440733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:21.533807+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.629706+0000","last_change":"2024-03-21T14:16:56.847010+0000","last_active":"2024-03-21T14:17:04.629706+0000","last_peered":"2024-03-21T14:17:04.629706+0000","last_clean":"2024-03-21T14:17:04.629706+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:17:04.629706+0000","last_undegraded":"2024-03-21T14:17:04.629706+0000","last_fullsized":"2024-03-21T14:17:04.629706+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:56.846916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:56.846916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:04.434680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"275'10","reported_seq":612,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624894+0000","last_change":"2024-03-21T14:16:19.562633+0000","last_active":"2024-03-21T14:17:04.624894+0000","last_peered":"2024-03-21T14:17:04.624894+0000","last_clean":"2024-03-21T14:17:04.624894+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:17:04.624894+0000","last_undegraded":"2024-03-21T14:17:04.624894+0000","last_fullsized":"2024-03-21T14:17:04.624894+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'10","last_scrub_stamp":"2024-03-21T14:16:19.562507+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:16:19.562507+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:25.547387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626002+0000","last_change":"2024-03-21T14:14:57.860174+0000","last_active":"2024-03-21T14:17:04.626002+0000","last_peered":"2024-03-21T14:17:04.626002+0000","last_clean":"2024-03-21T14:17:04.626002+0000","last_became_active":"2024-03-21T14:12:00.536387+0000","last_became_peered":"2024-03-21T14:12:00.536387+0000","last_unstale":"2024-03-21T14:17:04.626002+0000","last_undegraded":"2024-03-21T14:17:04.626002+0000","last_fullsized":"2024-03-21T14:17:04.626002+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:57.859470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:14:57.859470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624328+0000","last_change":"2024-03-21T14:16:06.800831+0000","last_active":"2024-03-21T14:17:04.624328+0000","last_peered":"2024-03-21T14:17:04.624328+0000","last_clean":"2024-03-21T14:17:04.624328+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:17:04.624328+0000","last_undegraded":"2024-03-21T14:17:04.624328+0000","last_fullsized":"2024-03-21T14:17:04.624328+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:06.800708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:06.800708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:16.698112+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":52,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.906626+0000","last_change":"2024-03-21T14:17:04.906653+0000","last_active":"2024-03-21T14:17:04.906626+0000","last_peered":"2024-03-21T14:17:04.906626+0000","last_clean":"2024-03-21T14:17:04.906626+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:17:04.906626+0000","last_undegraded":"2024-03-21T14:17:04.906626+0000","last_fullsized":"2024-03-21T14:17:04.906626+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:17:04.906566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:04.906566+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:07.800918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":640,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626563+0000","last_change":"2024-03-21T14:15:38.609735+0000","last_active":"2024-03-21T14:17:04.626563+0000","last_peered":"2024-03-21T14:17:04.626563+0000","last_clean":"2024-03-21T14:17:04.626563+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:17:04.626563+0000","last_undegraded":"2024-03-21T14:17:04.626563+0000","last_fullsized":"2024-03-21T14:17:04.626563+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:15:38.609594+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.609594+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":50,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624723+0000","last_change":"2024-03-21T14:16:21.866876+0000","last_active":"2024-03-21T14:17:04.624723+0000","last_peered":"2024-03-21T14:17:04.624723+0000","last_clean":"2024-03-21T14:17:04.624723+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:17:04.624723+0000","last_undegraded":"2024-03-21T14:17:04.624723+0000","last_fullsized":"2024-03-21T14:17:04.624723+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:46.242766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:46.242766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":574,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625903+0000","last_change":"2024-03-21T14:15:33.728687+0000","last_active":"2024-03-21T14:17:04.625903+0000","last_peered":"2024-03-21T14:17:04.625903+0000","last_clean":"2024-03-21T14:17:04.625903+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:17:04.625903+0000","last_undegraded":"2024-03-21T14:17:04.625903+0000","last_fullsized":"2024-03-21T14:17:04.625903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:33.728434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:15:33.728434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:05.617955+0000","last_change":"2024-03-21T14:17:05.617992+0000","last_active":"2024-03-21T14:17:05.617955+0000","last_peered":"2024-03-21T14:17:05.617955+0000","last_clean":"2024-03-21T14:17:05.617955+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:17:05.617955+0000","last_undegraded":"2024-03-21T14:17:05.617955+0000","last_fullsized":"2024-03-21T14:17:05.617955+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.617865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.617865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:09.660691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":622,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.628529+0000","last_change":"2024-03-21T14:14:54.733024+0000","last_active":"2024-03-21T14:17:04.628529+0000","last_peered":"2024-03-21T14:17:04.628529+0000","last_clean":"2024-03-21T14:14:52.723132+0000","last_became_active":"2024-03-21T14:14:54.733024+0000","last_became_peered":"2024-03-21T14:14:54.733024+0000","last_unstale":"2024-03-21T14:17:04.628529+0000","last_undegraded":"2024-03-21T14:14:54.728446+0000","last_fullsized":"2024-03-21T14:14:54.727848+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.492830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.492830+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":260,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624861+0000","last_change":"2024-03-21T14:16:21.862369+0000","last_active":"2024-03-21T14:17:04.624861+0000","last_peered":"2024-03-21T14:17:04.624861+0000","last_clean":"2024-03-21T14:14:52.705631+0000","last_became_active":"2024-03-21T14:16:21.862369+0000","last_became_peered":"2024-03-21T14:16:21.862369+0000","last_unstale":"2024-03-21T14:17:04.624861+0000","last_undegraded":"2024-03-21T14:17:04.624861+0000","last_fullsized":"2024-03-21T14:16:21.852836+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:01.256711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:01.256711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.630472+0000","last_change":"2024-03-21T14:15:36.366086+0000","last_active":"2024-03-21T14:17:04.630472+0000","last_peered":"2024-03-21T14:17:04.630472+0000","last_clean":"2024-03-21T14:17:04.630472+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:17:04.630472+0000","last_undegraded":"2024-03-21T14:17:04.630472+0000","last_fullsized":"2024-03-21T14:17:04.630472+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:36.365963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:36.365963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.618403+0000","last_change":"2024-03-21T14:14:54.734170+0000","last_active":"2024-03-21T14:17:04.618403+0000","last_peered":"2024-03-21T14:17:04.618403+0000","last_clean":"2024-03-21T14:14:52.716386+0000","last_became_active":"2024-03-21T14:14:54.734170+0000","last_became_peered":"2024-03-21T14:14:54.734170+0000","last_unstale":"2024-03-21T14:17:04.618403+0000","last_undegraded":"2024-03-21T14:17:04.618403+0000","last_fullsized":"2024-03-21T14:14:54.727989+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:47.020235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:47.020235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":644,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.621835+0000","last_change":"2024-03-21T14:16:13.416665+0000","last_active":"2024-03-21T14:17:04.621835+0000","last_peered":"2024-03-21T14:17:04.621835+0000","last_clean":"2024-03-21T14:17:04.621835+0000","last_became_active":"2024-03-21T14:16:00.609375+0000","last_became_peered":"2024-03-21T14:16:00.609375+0000","last_unstale":"2024-03-21T14:17:04.621835+0000","last_undegraded":"2024-03-21T14:17:04.621835+0000","last_fullsized":"2024-03-21T14:17:04.621835+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:16:13.416403+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.416403+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:31.227020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":435,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624998+0000","last_change":"2024-03-21T14:16:59.705337+0000","last_active":"2024-03-21T14:17:04.624998+0000","last_peered":"2024-03-21T14:17:04.624998+0000","last_clean":"2024-03-21T14:17:04.624998+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:17:04.624998+0000","last_undegraded":"2024-03-21T14:17:04.624998+0000","last_fullsized":"2024-03-21T14:17:04.624998+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:59.705219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:59.705219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:08.017501+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":730,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627086+0000","last_change":"2024-03-21T14:16:21.870976+0000","last_active":"2024-03-21T14:17:04.627086+0000","last_peered":"2024-03-21T14:17:04.627086+0000","last_clean":"2024-03-21T14:17:04.627086+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:17:04.627086+0000","last_undegraded":"2024-03-21T14:17:04.627086+0000","last_fullsized":"2024-03-21T14:17:04.627086+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'9","last_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.540505+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":586,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.623161+0000","last_change":"2024-03-21T14:14:54.736534+0000","last_active":"2024-03-21T14:17:04.623161+0000","last_peered":"2024-03-21T14:17:04.623161+0000","last_clean":"2024-03-21T14:14:52.711154+0000","last_became_active":"2024-03-21T14:14:54.736534+0000","last_became_peered":"2024-03-21T14:14:54.736534+0000","last_unstale":"2024-03-21T14:17:04.623161+0000","last_undegraded":"2024-03-21T14:17:04.623161+0000","last_fullsized":"2024-03-21T14:14:54.732339+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:34.724192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:13:34.724192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686777+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":613,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627327+0000","last_change":"2024-03-21T14:16:12.543045+0000","last_active":"2024-03-21T14:17:04.627327+0000","last_peered":"2024-03-21T14:17:04.627327+0000","last_clean":"2024-03-21T14:17:04.627327+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:17:04.627327+0000","last_undegraded":"2024-03-21T14:17:04.627327+0000","last_fullsized":"2024-03-21T14:17:04.627327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:16:12.542909+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:16:12.542909+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:36.540422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":556,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.620035+0000","last_change":"2024-03-21T14:14:54.741413+0000","last_active":"2024-03-21T14:17:04.620035+0000","last_peered":"2024-03-21T14:17:04.620035+0000","last_clean":"2024-03-21T14:14:52.716802+0000","last_became_active":"2024-03-21T14:14:54.741413+0000","last_became_peered":"2024-03-21T14:14:54.741413+0000","last_unstale":"2024-03-21T14:17:04.620035+0000","last_undegraded":"2024-03-21T14:17:04.620035+0000","last_fullsized":"2024-03-21T14:14:54.729363+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347441+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"312'11","reported_seq":66,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.627709+0000","last_change":"2024-03-21T14:16:34.613524+0000","last_active":"2024-03-21T14:17:04.627709+0000","last_peered":"2024-03-21T14:17:04.627709+0000","last_clean":"2024-03-21T14:02:40.635115+0000","last_became_active":"2024-03-21T14:16:34.613524+0000","last_became_peered":"2024-03-21T14:16:34.613524+0000","last_unstale":"2024-03-21T14:17:04.627709+0000","last_undegraded":"2024-03-21T14:16:34.608081+0000","last_fullsized":"2024-03-21T14:16:34.607957+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:13:03.638324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.638324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":194,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627693+0000","last_change":"2024-03-21T14:16:34.620542+0000","last_active":"2024-03-21T14:17:04.627693+0000","last_peered":"2024-03-21T14:17:04.627693+0000","last_clean":"2024-03-21T14:17:04.627693+0000","last_became_active":"2024-03-21T14:16:34.620151+0000","last_became_peered":"2024-03-21T14:16:34.620151+0000","last_unstale":"2024-03-21T14:17:04.627693+0000","last_undegraded":"2024-03-21T14:17:04.627693+0000","last_fullsized":"2024-03-21T14:17:04.627693+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:38.968899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.968899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":454,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624803+0000","last_change":"2024-03-21T14:16:21.889160+0000","last_active":"2024-03-21T14:17:04.624803+0000","last_peered":"2024-03-21T14:17:04.624803+0000","last_clean":"2024-03-21T14:17:04.624803+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:17:04.624803+0000","last_undegraded":"2024-03-21T14:17:04.624803+0000","last_fullsized":"2024-03-21T14:17:04.624803+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:16:08.564278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:16:08.564278+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:23.996276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":99,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.620176+0000","last_change":"2024-03-21T14:14:54.738499+0000","last_active":"2024-03-21T14:17:04.620176+0000","last_peered":"2024-03-21T14:17:04.620176+0000","last_clean":"2024-03-21T14:02:38.441204+0000","last_became_active":"2024-03-21T14:14:54.738499+0000","last_became_peered":"2024-03-21T14:14:54.738499+0000","last_unstale":"2024-03-21T14:17:04.620176+0000","last_undegraded":"2024-03-21T14:17:04.620176+0000","last_fullsized":"2024-03-21T14:14:54.731067+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:43.502435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:43.502435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":435,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627013+0000","last_change":"2024-03-21T14:16:57.987097+0000","last_active":"2024-03-21T14:17:04.627013+0000","last_peered":"2024-03-21T14:17:04.627013+0000","last_clean":"2024-03-21T14:17:04.627013+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:17:04.627013+0000","last_undegraded":"2024-03-21T14:17:04.627013+0000","last_fullsized":"2024-03-21T14:17:04.627013+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.986983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.986983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:10.230755+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:05.374024+0000","last_change":"2024-03-21T14:17:05.374072+0000","last_active":"2024-03-21T14:17:05.374024+0000","last_peered":"2024-03-21T14:17:05.374024+0000","last_clean":"2024-03-21T14:17:05.374024+0000","last_became_active":"2024-03-21T14:16:34.616172+0000","last_became_peered":"2024-03-21T14:16:34.616172+0000","last_unstale":"2024-03-21T14:17:05.374024+0000","last_undegraded":"2024-03-21T14:17:05.374024+0000","last_fullsized":"2024-03-21T14:17:05.374024+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.373895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.373895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:35.036037+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":270,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.628230+0000","last_change":"2024-03-21T14:15:24.741252+0000","last_active":"2024-03-21T14:17:04.628230+0000","last_peered":"2024-03-21T14:17:04.628230+0000","last_clean":"2024-03-21T14:17:04.628230+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:17:04.628230+0000","last_undegraded":"2024-03-21T14:17:04.628230+0000","last_fullsized":"2024-03-21T14:17:04.628230+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:15:24.741119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:24.741119+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":576,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.623838+0000","last_change":"2024-03-21T14:15:38.063435+0000","last_active":"2024-03-21T14:17:04.623838+0000","last_peered":"2024-03-21T14:17:04.623838+0000","last_clean":"2024-03-21T14:17:04.623838+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:17:04.623838+0000","last_undegraded":"2024-03-21T14:17:04.623838+0000","last_fullsized":"2024-03-21T14:17:04.623838+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:38.059436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.059436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":56,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.628138+0000","last_change":"2024-03-21T14:17:03.781272+0000","last_active":"2024-03-21T14:17:04.628138+0000","last_peered":"2024-03-21T14:17:04.628138+0000","last_clean":"2024-03-21T14:17:04.628138+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:17:04.628138+0000","last_undegraded":"2024-03-21T14:17:04.628138+0000","last_fullsized":"2024-03-21T14:17:04.628138+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:03.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.517464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":212,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.624665+0000","last_change":"2024-03-21T14:16:22.893949+0000","last_active":"2024-03-21T14:17:04.624665+0000","last_peered":"2024-03-21T14:17:04.624665+0000","last_clean":"2024-03-21T14:17:04.624665+0000","last_became_active":"2024-03-21T14:16:22.893331+0000","last_became_peered":"2024-03-21T14:16:22.893331+0000","last_unstale":"2024-03-21T14:17:04.624665+0000","last_undegraded":"2024-03-21T14:17:04.624665+0000","last_fullsized":"2024-03-21T14:17:04.624665+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:28.983112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:28.983112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,6],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":318,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.627137+0000","last_change":"2024-03-21T14:16:34.615269+0000","last_active":"2024-03-21T14:17:04.627137+0000","last_peered":"2024-03-21T14:17:04.627137+0000","last_clean":"2024-03-21T14:12:42.522855+0000","last_became_active":"2024-03-21T14:16:34.615269+0000","last_became_peered":"2024-03-21T14:16:34.615269+0000","last_unstale":"2024-03-21T14:17:04.627137+0000","last_undegraded":"2024-03-21T14:16:34.611042+0000","last_fullsized":"2024-03-21T14:16:34.610902+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:13:59.567870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.567870+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"324'1211","reported_seq":1235,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:06.452684+0000","last_change":"2024-03-21T14:16:51.057507+0000","last_active":"2024-03-21T14:17:06.452684+0000","last_peered":"2024-03-21T14:17:06.452684+0000","last_clean":"2024-03-21T14:17:06.452684+0000","last_became_active":"2024-03-21T14:16:51.057016+0000","last_became_peered":"2024-03-21T14:16:51.057016+0000","last_unstale":"2024-03-21T14:17:06.452684+0000","last_undegraded":"2024-03-21T14:17:06.452684+0000","last_fullsized":"2024-03-21T14:17:06.452684+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1211,"log_dups_size":0,"ondisk_log_size":1211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:18.774227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79364096,"num_objects":1211,"num_object_clones":0,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1211,"num_write_kb":77504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":455,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627535+0000","last_change":"2024-03-21T14:16:21.869140+0000","last_active":"2024-03-21T14:17:04.627535+0000","last_peered":"2024-03-21T14:17:04.627535+0000","last_clean":"2024-03-21T14:17:04.627535+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:17:04.627535+0000","last_undegraded":"2024-03-21T14:17:04.627535+0000","last_fullsized":"2024-03-21T14:17:04.627535+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:14.598756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:16:14.598756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:26.510077+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":606,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625730+0000","last_change":"2024-03-21T14:15:04.004989+0000","last_active":"2024-03-21T14:17:04.625730+0000","last_peered":"2024-03-21T14:17:04.625730+0000","last_clean":"2024-03-21T14:17:04.625730+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:17:04.625730+0000","last_undegraded":"2024-03-21T14:17:04.625730+0000","last_fullsized":"2024-03-21T14:17:04.625730+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:15:04.004853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:04.004853+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":502,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624212+0000","last_change":"2024-03-21T14:17:02.712012+0000","last_active":"2024-03-21T14:17:04.624212+0000","last_peered":"2024-03-21T14:17:04.624212+0000","last_clean":"2024-03-21T14:17:04.624212+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:17:04.624212+0000","last_undegraded":"2024-03-21T14:17:04.624212+0000","last_fullsized":"2024-03-21T14:17:04.624212+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:17:02.711916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:02.711916+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:19.694933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"214'9","reported_seq":767,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.623756+0000","last_change":"2024-03-21T14:16:14.902618+0000","last_active":"2024-03-21T14:17:04.623756+0000","last_peered":"2024-03-21T14:17:04.623756+0000","last_clean":"2024-03-21T14:17:04.623756+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:17:04.623756+0000","last_undegraded":"2024-03-21T14:17:04.623756+0000","last_fullsized":"2024-03-21T14:17:04.623756+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'9","last_scrub_stamp":"2024-03-21T14:16:14.902143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:14.902143+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:39.064216+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"324'1173","reported_seq":1194,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:06.464343+0000","last_change":"2024-03-21T14:17:01.220613+0000","last_active":"2024-03-21T14:17:06.464343+0000","last_peered":"2024-03-21T14:17:06.464343+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.055518+0000","last_became_peered":"2024-03-21T14:16:51.055518+0000","last_unstale":"2024-03-21T14:17:06.464343+0000","last_undegraded":"2024-03-21T14:17:01.207316+0000","last_fullsized":"2024-03-21T14:16:51.047099+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1173,"log_dups_size":0,"ondisk_log_size":1173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:05.018647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76873728,"num_objects":1173,"num_object_clones":0,"num_object_copies":3519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1173,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1173,"num_write_kb":75072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1173}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":312,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625662+0000","last_change":"2024-03-21T14:15:40.753578+0000","last_active":"2024-03-21T14:17:04.625662+0000","last_peered":"2024-03-21T14:17:04.625662+0000","last_clean":"2024-03-21T14:17:04.625662+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:17:04.625662+0000","last_undegraded":"2024-03-21T14:17:04.625662+0000","last_fullsized":"2024-03-21T14:17:04.625662+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:40.753276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:40.753276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625108+0000","last_change":"2024-03-21T14:15:41.992210+0000","last_active":"2024-03-21T14:17:04.625108+0000","last_peered":"2024-03-21T14:17:04.625108+0000","last_clean":"2024-03-21T14:17:04.625108+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:17:04.625108+0000","last_undegraded":"2024-03-21T14:17:04.625108+0000","last_fullsized":"2024-03-21T14:17:04.625108+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:41.991958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:41.991958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":638,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618440+0000","last_change":"2024-03-21T14:15:43.183428+0000","last_active":"2024-03-21T14:17:04.618440+0000","last_peered":"2024-03-21T14:17:04.618440+0000","last_clean":"2024-03-21T14:17:04.618440+0000","last_became_active":"2024-03-21T14:12:00.540768+0000","last_became_peered":"2024-03-21T14:12:00.540768+0000","last_unstale":"2024-03-21T14:17:04.618440+0000","last_undegraded":"2024-03-21T14:17:04.618440+0000","last_fullsized":"2024-03-21T14:17:04.618440+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:15:43.183132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:43.183132+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":100,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624942+0000","last_change":"2024-03-21T14:14:54.745296+0000","last_active":"2024-03-21T14:17:04.624942+0000","last_peered":"2024-03-21T14:17:04.624942+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:14:54.745296+0000","last_became_peered":"2024-03-21T14:14:54.745296+0000","last_unstale":"2024-03-21T14:17:04.624942+0000","last_undegraded":"2024-03-21T14:17:04.624942+0000","last_fullsized":"2024-03-21T14:14:54.730905+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:39.425147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:14:39.425147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"324'1143","reported_seq":1167,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:06.465871+0000","last_change":"2024-03-21T14:16:51.060563+0000","last_active":"2024-03-21T14:17:06.465871+0000","last_peered":"2024-03-21T14:17:06.465871+0000","last_clean":"2024-03-21T14:17:06.465871+0000","last_became_active":"2024-03-21T14:16:51.060300+0000","last_became_peered":"2024-03-21T14:16:51.060300+0000","last_unstale":"2024-03-21T14:17:06.465871+0000","last_undegraded":"2024-03-21T14:17:06.465871+0000","last_fullsized":"2024-03-21T14:17:06.465871+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1143,"log_dups_size":0,"ondisk_log_size":1143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:55.863045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":3429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1143,"num_write_kb":73152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":501,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624707+0000","last_change":"2024-03-21T14:16:34.615853+0000","last_active":"2024-03-21T14:17:04.624707+0000","last_peered":"2024-03-21T14:17:04.624707+0000","last_clean":"2024-03-21T14:14:52.709870+0000","last_became_active":"2024-03-21T14:16:34.615853+0000","last_became_peered":"2024-03-21T14:16:34.615853+0000","last_unstale":"2024-03-21T14:17:04.624707+0000","last_undegraded":"2024-03-21T14:17:04.624707+0000","last_fullsized":"2024-03-21T14:16:34.611199+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:54.779479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.779479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.761981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":47,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625383+0000","last_change":"2024-03-21T14:16:21.862254+0000","last_active":"2024-03-21T14:17:04.625383+0000","last_peered":"2024-03-21T14:17:04.625383+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:16:21.862254+0000","last_became_peered":"2024-03-21T14:16:21.862254+0000","last_unstale":"2024-03-21T14:17:04.625383+0000","last_undegraded":"2024-03-21T14:17:04.625383+0000","last_fullsized":"2024-03-21T14:16:21.852757+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:43.154506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.154506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.628090+0000","last_change":"2024-03-21T14:16:34.619006+0000","last_active":"2024-03-21T14:17:04.628090+0000","last_peered":"2024-03-21T14:17:04.628090+0000","last_clean":"2024-03-21T14:17:04.628090+0000","last_became_active":"2024-03-21T14:16:34.618652+0000","last_became_peered":"2024-03-21T14:16:34.618652+0000","last_unstale":"2024-03-21T14:17:04.628090+0000","last_undegraded":"2024-03-21T14:17:04.628090+0000","last_fullsized":"2024-03-21T14:17:04.628090+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:25.963838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:25.963838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":531,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627948+0000","last_change":"2024-03-21T14:15:34.404267+0000","last_active":"2024-03-21T14:17:04.627948+0000","last_peered":"2024-03-21T14:17:04.627948+0000","last_clean":"2024-03-21T14:17:04.627948+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:17:04.627948+0000","last_undegraded":"2024-03-21T14:17:04.627948+0000","last_fullsized":"2024-03-21T14:17:04.627948+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:15:34.404114+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:11:10.962498+0000","last_clean_scrub_stamp":"2024-03-21T14:15:34.404114+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"324'1330","reported_seq":1351,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:07.168869+0000","last_change":"2024-03-21T14:17:01.213763+0000","last_active":"2024-03-21T14:17:07.168869+0000","last_peered":"2024-03-21T14:17:07.168869+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.048501+0000","last_became_peered":"2024-03-21T14:16:51.048501+0000","last_unstale":"2024-03-21T14:17:07.168869+0000","last_undegraded":"2024-03-21T14:17:01.209817+0000","last_fullsized":"2024-03-21T14:16:51.043684+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1330,"log_dups_size":0,"ondisk_log_size":1330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:10.283522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87162880,"num_objects":1330,"num_object_clones":0,"num_object_copies":3990,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1330,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1330,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1330,"num_write_kb":85120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":1330}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":366,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624362+0000","last_change":"2024-03-21T14:16:34.617543+0000","last_active":"2024-03-21T14:17:04.624362+0000","last_peered":"2024-03-21T14:17:04.624362+0000","last_clean":"2024-03-21T14:14:52.711889+0000","last_became_active":"2024-03-21T14:16:34.617543+0000","last_became_peered":"2024-03-21T14:16:34.617543+0000","last_unstale":"2024-03-21T14:17:04.624362+0000","last_undegraded":"2024-03-21T14:17:04.624362+0000","last_fullsized":"2024-03-21T14:16:34.611916+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.865488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.865488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.762090+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":100,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.620099+0000","last_change":"2024-03-21T14:14:54.731341+0000","last_active":"2024-03-21T14:17:04.620099+0000","last_peered":"2024-03-21T14:17:04.620099+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:14:54.731341+0000","last_became_peered":"2024-03-21T14:14:54.731341+0000","last_unstale":"2024-03-21T14:17:04.620099+0000","last_undegraded":"2024-03-21T14:17:04.620099+0000","last_fullsized":"2024-03-21T14:14:54.724706+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.532653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.532653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":382,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627210+0000","last_change":"2024-03-21T14:16:21.865674+0000","last_active":"2024-03-21T14:17:04.627210+0000","last_peered":"2024-03-21T14:17:04.627210+0000","last_clean":"2024-03-21T14:17:04.627210+0000","last_became_active":"2024-03-21T14:16:21.865349+0000","last_became_peered":"2024-03-21T14:16:21.865349+0000","last_unstale":"2024-03-21T14:17:04.627210+0000","last_undegraded":"2024-03-21T14:17:04.627210+0000","last_fullsized":"2024-03-21T14:17:04.627210+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:34.477257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:34.477257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":180,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.618555+0000","last_change":"2024-03-21T14:14:54.734284+0000","last_active":"2024-03-21T14:17:04.618555+0000","last_peered":"2024-03-21T14:17:04.618555+0000","last_clean":"2024-03-21T14:12:42.578861+0000","last_became_active":"2024-03-21T14:14:54.734284+0000","last_became_peered":"2024-03-21T14:14:54.734284+0000","last_unstale":"2024-03-21T14:17:04.618555+0000","last_undegraded":"2024-03-21T14:14:54.726445+0000","last_fullsized":"2024-03-21T14:14:54.726319+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.159386+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.a","version":"324'1136","reported_seq":1157,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:06.781024+0000","last_change":"2024-03-21T14:17:01.214830+0000","last_active":"2024-03-21T14:17:06.781024+0000","last_peered":"2024-03-21T14:17:06.781024+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.052354+0000","last_became_peered":"2024-03-21T14:16:51.052354+0000","last_unstale":"2024-03-21T14:17:06.781024+0000","last_undegraded":"2024-03-21T14:17:01.206891+0000","last_fullsized":"2024-03-21T14:16:51.046025+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1136,"log_dups_size":0,"ondisk_log_size":1136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:51.001045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74448896,"num_objects":1136,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1136,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1136,"num_write_kb":72704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1136}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":640,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.622962+0000","last_change":"2024-03-21T14:14:54.737456+0000","last_active":"2024-03-21T14:17:04.622962+0000","last_peered":"2024-03-21T14:17:04.622962+0000","last_clean":"2024-03-21T14:14:52.705212+0000","last_became_active":"2024-03-21T14:14:54.737456+0000","last_became_peered":"2024-03-21T14:14:54.737456+0000","last_unstale":"2024-03-21T14:17:04.622962+0000","last_undegraded":"2024-03-21T14:17:04.622962+0000","last_fullsized":"2024-03-21T14:14:54.720914+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:38.263320+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:38.263320+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.111183+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":570,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.628322+0000","last_change":"2024-03-21T14:16:10.542371+0000","last_active":"2024-03-21T14:17:04.628322+0000","last_peered":"2024-03-21T14:17:04.628322+0000","last_clean":"2024-03-21T14:17:04.628322+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:17:04.628322+0000","last_undegraded":"2024-03-21T14:17:04.628322+0000","last_fullsized":"2024-03-21T14:17:04.628322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:10.542193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:10.542193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:32.264925+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":82,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627785+0000","last_change":"2024-03-21T14:16:23.920822+0000","last_active":"2024-03-21T14:17:04.627785+0000","last_peered":"2024-03-21T14:17:04.627785+0000","last_clean":"2024-03-21T14:17:04.627785+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:17:04.627785+0000","last_undegraded":"2024-03-21T14:17:04.627785+0000","last_fullsized":"2024-03-21T14:17:04.627785+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:16:23.920695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:16:23.920695+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:33.452138+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":580,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.620341+0000","last_change":"2024-03-21T14:16:57.382377+0000","last_active":"2024-03-21T14:17:04.620341+0000","last_peered":"2024-03-21T14:17:04.620341+0000","last_clean":"2024-03-21T14:17:04.620341+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:17:04.620341+0000","last_undegraded":"2024-03-21T14:17:04.620341+0000","last_fullsized":"2024-03-21T14:17:04.620341+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.382291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.382291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:25.063945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"307'53","reported_seq":587,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.627451+0000","last_change":"2024-03-21T14:16:21.860558+0000","last_active":"2024-03-21T14:17:04.627451+0000","last_peered":"2024-03-21T14:17:04.627451+0000","last_clean":"2024-03-21T14:12:42.663015+0000","last_became_active":"2024-03-21T14:16:21.860558+0000","last_became_peered":"2024-03-21T14:16:21.860558+0000","last_unstale":"2024-03-21T14:17:04.627451+0000","last_undegraded":"2024-03-21T14:16:21.858369+0000","last_fullsized":"2024-03-21T14:16:21.858165+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-03-21T14:12:51.290920+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:12:51.290920+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.762008+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":442,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627484+0000","last_change":"2024-03-21T14:15:31.430590+0000","last_active":"2024-03-21T14:17:04.627484+0000","last_peered":"2024-03-21T14:17:04.627484+0000","last_clean":"2024-03-21T14:17:04.627484+0000","last_became_active":"2024-03-21T14:12:00.526762+0000","last_became_peered":"2024-03-21T14:12:00.526762+0000","last_unstale":"2024-03-21T14:17:04.627484+0000","last_undegraded":"2024-03-21T14:17:04.627484+0000","last_fullsized":"2024-03-21T14:17:04.627484+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:31.430514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:31.430514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"324'1212","reported_seq":1233,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:06.777961+0000","last_change":"2024-03-21T14:17:01.216005+0000","last_active":"2024-03-21T14:17:06.777961+0000","last_peered":"2024-03-21T14:17:06.777961+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.048295+0000","last_became_peered":"2024-03-21T14:16:51.048295+0000","last_unstale":"2024-03-21T14:17:06.777961+0000","last_undegraded":"2024-03-21T14:17:01.206551+0000","last_fullsized":"2024-03-21T14:16:51.045802+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1212,"log_dups_size":0,"ondisk_log_size":1212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:58.330943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79429632,"num_objects":1212,"num_object_clones":0,"num_object_copies":3636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1212,"num_write_kb":77568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1212}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":805,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.617134+0000","last_change":"2024-03-21T14:16:58.345839+0000","last_active":"2024-03-21T14:17:04.617134+0000","last_peered":"2024-03-21T14:17:04.617134+0000","last_clean":"2024-03-21T14:17:04.617134+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:17:04.617134+0000","last_undegraded":"2024-03-21T14:17:04.617134+0000","last_fullsized":"2024-03-21T14:17:04.617134+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:58.345739+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:58.345739+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.806247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624898+0000","last_change":"2024-03-21T14:14:54.732344+0000","last_active":"2024-03-21T14:17:04.624898+0000","last_peered":"2024-03-21T14:17:04.624898+0000","last_clean":"2024-03-21T14:14:52.711965+0000","last_became_active":"2024-03-21T14:14:54.732344+0000","last_became_peered":"2024-03-21T14:14:54.732344+0000","last_unstale":"2024-03-21T14:17:04.624898+0000","last_undegraded":"2024-03-21T14:17:04.624898+0000","last_fullsized":"2024-03-21T14:14:54.726496+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:37.711376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:37.711376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686758+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625430+0000","last_change":"2024-03-21T14:14:54.746014+0000","last_active":"2024-03-21T14:17:04.625430+0000","last_peered":"2024-03-21T14:17:04.625430+0000","last_clean":"2024-03-21T14:14:52.716087+0000","last_became_active":"2024-03-21T14:14:54.746014+0000","last_became_peered":"2024-03-21T14:14:54.746014+0000","last_unstale":"2024-03-21T14:17:04.625430+0000","last_undegraded":"2024-03-21T14:17:04.625430+0000","last_fullsized":"2024-03-21T14:14:54.731030+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.667132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.667132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":99,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.626206+0000","last_change":"2024-03-21T14:14:54.730388+0000","last_active":"2024-03-21T14:17:04.626206+0000","last_peered":"2024-03-21T14:17:04.626206+0000","last_clean":"2024-03-21T14:02:31.423682+0000","last_became_active":"2024-03-21T14:14:54.730388+0000","last_became_peered":"2024-03-21T14:14:54.730388+0000","last_unstale":"2024-03-21T14:17:04.626206+0000","last_undegraded":"2024-03-21T14:17:04.626206+0000","last_fullsized":"2024-03-21T14:14:54.723417+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.212763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.212763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.762076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":650,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626753+0000","last_change":"2024-03-21T14:16:10.652519+0000","last_active":"2024-03-21T14:17:04.626753+0000","last_peered":"2024-03-21T14:17:04.626753+0000","last_clean":"2024-03-21T14:17:04.626753+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:17:04.626753+0000","last_undegraded":"2024-03-21T14:17:04.626753+0000","last_fullsized":"2024-03-21T14:17:04.626753+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:16:10.652373+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:16:10.652373+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:13.400287+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":46,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625398+0000","last_change":"2024-03-21T14:16:21.863083+0000","last_active":"2024-03-21T14:17:04.625398+0000","last_peered":"2024-03-21T14:17:04.625398+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:16:21.863083+0000","last_became_peered":"2024-03-21T14:16:21.863083+0000","last_unstale":"2024-03-21T14:17:04.625398+0000","last_undegraded":"2024-03-21T14:17:04.625398+0000","last_fullsized":"2024-03-21T14:16:21.855618+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:07.244640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.244640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.b","version":"324'1320","reported_seq":1344,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:07.388582+0000","last_change":"2024-03-21T14:16:51.054881+0000","last_active":"2024-03-21T14:17:07.388582+0000","last_peered":"2024-03-21T14:17:07.388582+0000","last_clean":"2024-03-21T14:17:07.388582+0000","last_became_active":"2024-03-21T14:16:51.054471+0000","last_became_peered":"2024-03-21T14:16:51.054471+0000","last_unstale":"2024-03-21T14:17:07.388582+0000","last_undegraded":"2024-03-21T14:17:07.388582+0000","last_fullsized":"2024-03-21T14:17:07.388582+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1320,"log_dups_size":0,"ondisk_log_size":1320,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:52.373755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86507520,"num_objects":1320,"num_object_clones":0,"num_object_copies":3960,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1320,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1320,"num_write_kb":84480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":55,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.629960+0000","last_change":"2024-03-21T14:16:55.947536+0000","last_active":"2024-03-21T14:17:04.629960+0000","last_peered":"2024-03-21T14:17:04.629960+0000","last_clean":"2024-03-21T14:17:04.629960+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:17:04.629960+0000","last_undegraded":"2024-03-21T14:17:04.629960+0000","last_fullsized":"2024-03-21T14:17:04.629960+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:55.947438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:55.947438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:11.457883+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":881,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.623976+0000","last_change":"2024-03-21T14:14:54.746367+0000","last_active":"2024-03-21T14:17:04.623976+0000","last_peered":"2024-03-21T14:17:04.623976+0000","last_clean":"2024-03-21T14:14:52.710400+0000","last_became_active":"2024-03-21T14:14:54.746367+0000","last_became_peered":"2024-03-21T14:14:54.746367+0000","last_unstale":"2024-03-21T14:17:04.623976+0000","last_undegraded":"2024-03-21T14:17:04.623976+0000","last_fullsized":"2024-03-21T14:14:54.729618+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:37.747359+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:37.747359+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.709576+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":597,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625367+0000","last_change":"2024-03-21T14:16:20.564440+0000","last_active":"2024-03-21T14:17:04.625367+0000","last_peered":"2024-03-21T14:17:04.625367+0000","last_clean":"2024-03-21T14:17:04.625367+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:17:04.625367+0000","last_undegraded":"2024-03-21T14:17:04.625367+0000","last_fullsized":"2024-03-21T14:17:04.625367+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:16:20.564321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:28.547951+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":388,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.617632+0000","last_change":"2024-03-21T14:14:57.048261+0000","last_active":"2024-03-21T14:17:04.617632+0000","last_peered":"2024-03-21T14:17:04.617632+0000","last_clean":"2024-03-21T14:17:04.617632+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:17:04.617632+0000","last_undegraded":"2024-03-21T14:17:04.617632+0000","last_fullsized":"2024-03-21T14:17:04.617632+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'29","last_scrub_stamp":"2024-03-21T14:14:57.048131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:57.048131+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625048+0000","last_change":"2024-03-21T14:14:54.743075+0000","last_active":"2024-03-21T14:17:04.625048+0000","last_peered":"2024-03-21T14:17:04.625048+0000","last_clean":"2024-03-21T14:14:52.711749+0000","last_became_active":"2024-03-21T14:14:54.743075+0000","last_became_peered":"2024-03-21T14:14:54.743075+0000","last_unstale":"2024-03-21T14:17:04.625048+0000","last_undegraded":"2024-03-21T14:17:04.625048+0000","last_fullsized":"2024-03-21T14:14:54.725771+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.820408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.820408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"324'1193","reported_seq":1214,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:06.792137+0000","last_change":"2024-03-21T14:17:01.206813+0000","last_active":"2024-03-21T14:17:06.792137+0000","last_peered":"2024-03-21T14:17:06.792137+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.048170+0000","last_became_peered":"2024-03-21T14:16:51.048170+0000","last_unstale":"2024-03-21T14:17:06.792137+0000","last_undegraded":"2024-03-21T14:17:01.206430+0000","last_fullsized":"2024-03-21T14:16:51.045507+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1193,"log_dups_size":0,"ondisk_log_size":1193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:15.453539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78184448,"num_objects":1193,"num_object_clones":0,"num_object_copies":3579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1193,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1193,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1193,"num_write_kb":76352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":1193}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":652,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626199+0000","last_change":"2024-03-21T14:16:02.693105+0000","last_active":"2024-03-21T14:17:04.626199+0000","last_peered":"2024-03-21T14:17:04.626199+0000","last_clean":"2024-03-21T14:17:04.626199+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:17:04.626199+0000","last_undegraded":"2024-03-21T14:17:04.626199+0000","last_fullsized":"2024-03-21T14:17:04.626199+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:02.692989+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:02.692989+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.572551+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627917+0000","last_change":"2024-03-21T14:15:59.603291+0000","last_active":"2024-03-21T14:17:04.627917+0000","last_peered":"2024-03-21T14:17:04.627917+0000","last_clean":"2024-03-21T14:17:04.627917+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:17:04.627917+0000","last_undegraded":"2024-03-21T14:17:04.627917+0000","last_fullsized":"2024-03-21T14:17:04.627917+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:00.656879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:15:00.656879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":376,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618325+0000","last_change":"2024-03-21T14:15:01.061413+0000","last_active":"2024-03-21T14:17:04.618325+0000","last_peered":"2024-03-21T14:17:04.618325+0000","last_clean":"2024-03-21T14:17:04.618325+0000","last_became_active":"2024-03-21T14:12:20.559683+0000","last_became_peered":"2024-03-21T14:12:20.559683+0000","last_unstale":"2024-03-21T14:17:04.618325+0000","last_undegraded":"2024-03-21T14:17:04.618325+0000","last_fullsized":"2024-03-21T14:17:04.618325+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:01.061102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:01.061102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1367,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:05.786251+0000","last_change":"2024-03-21T14:16:34.619088+0000","last_active":"2024-03-21T14:17:05.786251+0000","last_peered":"2024-03-21T14:17:05.786251+0000","last_clean":"2024-03-21T14:17:05.786251+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:17:05.786251+0000","last_undegraded":"2024-03-21T14:17:05.786251+0000","last_fullsized":"2024-03-21T14:17:05.786251+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.009777+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":808,"num_read_kb":803,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":499,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.627150+0000","last_change":"2024-03-21T14:14:54.741139+0000","last_active":"2024-03-21T14:17:04.627150+0000","last_peered":"2024-03-21T14:17:04.627150+0000","last_clean":"2024-03-21T14:14:52.718881+0000","last_became_active":"2024-03-21T14:14:54.741139+0000","last_became_peered":"2024-03-21T14:14:54.741139+0000","last_unstale":"2024-03-21T14:17:04.627150+0000","last_undegraded":"2024-03-21T14:14:54.731232+0000","last_fullsized":"2024-03-21T14:14:54.731117+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:13:43.544671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.544671+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":49,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627615+0000","last_change":"2024-03-21T14:16:21.868921+0000","last_active":"2024-03-21T14:17:04.627615+0000","last_peered":"2024-03-21T14:17:04.627615+0000","last_clean":"2024-03-21T14:17:04.627615+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:17:04.627615+0000","last_undegraded":"2024-03-21T14:17:04.627615+0000","last_fullsized":"2024-03-21T14:17:04.627615+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:40.570508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:15:40.570508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"324'1075","reported_seq":1099,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:06.466688+0000","last_change":"2024-03-21T14:16:51.055172+0000","last_active":"2024-03-21T14:17:06.466688+0000","last_peered":"2024-03-21T14:17:06.466688+0000","last_clean":"2024-03-21T14:17:06.466688+0000","last_became_active":"2024-03-21T14:16:51.054796+0000","last_became_peered":"2024-03-21T14:16:51.054796+0000","last_unstale":"2024-03-21T14:17:06.466688+0000","last_undegraded":"2024-03-21T14:17:06.466688+0000","last_fullsized":"2024-03-21T14:17:06.466688+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1075,"log_dups_size":0,"ondisk_log_size":1075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.729411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70451200,"num_objects":1075,"num_object_clones":0,"num_object_copies":3225,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1075,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1075,"num_write_kb":68800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":646,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618339+0000","last_change":"2024-03-21T14:17:01.305286+0000","last_active":"2024-03-21T14:17:04.618339+0000","last_peered":"2024-03-21T14:17:04.618339+0000","last_clean":"2024-03-21T14:17:04.618339+0000","last_became_active":"2024-03-21T14:14:35.622383+0000","last_became_peered":"2024-03-21T14:14:35.622383+0000","last_unstale":"2024-03-21T14:17:04.618339+0000","last_undegraded":"2024-03-21T14:17:04.618339+0000","last_fullsized":"2024-03-21T14:17:04.618339+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:17:01.305155+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:17:01.305155+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:03.446459+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":579,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.626628+0000","last_change":"2024-03-21T14:14:54.741160+0000","last_active":"2024-03-21T14:17:04.626628+0000","last_peered":"2024-03-21T14:17:04.626628+0000","last_clean":"2024-03-21T14:14:52.714204+0000","last_became_active":"2024-03-21T14:14:54.741160+0000","last_became_peered":"2024-03-21T14:14:54.741160+0000","last_unstale":"2024-03-21T14:17:04.626628+0000","last_undegraded":"2024-03-21T14:17:04.626628+0000","last_fullsized":"2024-03-21T14:14:54.728056+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:43.550498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:14:43.550498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":585,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.628539+0000","last_change":"2024-03-21T14:14:54.741947+0000","last_active":"2024-03-21T14:17:04.628539+0000","last_peered":"2024-03-21T14:17:04.628539+0000","last_clean":"2024-03-21T14:14:52.716388+0000","last_became_active":"2024-03-21T14:14:54.741947+0000","last_became_peered":"2024-03-21T14:14:54.741947+0000","last_unstale":"2024-03-21T14:17:04.628539+0000","last_undegraded":"2024-03-21T14:17:04.628539+0000","last_fullsized":"2024-03-21T14:14:54.727020+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:55.416068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.416068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":305,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.623223+0000","last_change":"2024-03-21T14:16:58.126142+0000","last_active":"2024-03-21T14:17:04.623223+0000","last_peered":"2024-03-21T14:17:04.623223+0000","last_clean":"2024-03-21T14:17:04.623223+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:17:04.623223+0000","last_undegraded":"2024-03-21T14:17:04.623223+0000","last_fullsized":"2024-03-21T14:17:04.623223+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:16:58.125989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:16:58.125989+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:09.891261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":173,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.626248+0000","last_change":"2024-03-21T14:16:21.862283+0000","last_active":"2024-03-21T14:17:04.626248+0000","last_peered":"2024-03-21T14:17:04.626248+0000","last_clean":"2024-03-21T14:14:52.705626+0000","last_became_active":"2024-03-21T14:16:21.862283+0000","last_became_peered":"2024-03-21T14:16:21.862283+0000","last_unstale":"2024-03-21T14:17:04.626248+0000","last_undegraded":"2024-03-21T14:17:04.626248+0000","last_fullsized":"2024-03-21T14:16:21.852923+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.222353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.222353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"324'1298","reported_seq":1322,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:06.811074+0000","last_change":"2024-03-21T14:16:51.057360+0000","last_active":"2024-03-21T14:17:06.811074+0000","last_peered":"2024-03-21T14:17:06.811074+0000","last_clean":"2024-03-21T14:17:06.811074+0000","last_became_active":"2024-03-21T14:16:51.056996+0000","last_became_peered":"2024-03-21T14:16:51.056996+0000","last_unstale":"2024-03-21T14:17:06.811074+0000","last_undegraded":"2024-03-21T14:17:06.811074+0000","last_fullsized":"2024-03-21T14:17:06.811074+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1298,"log_dups_size":0,"ondisk_log_size":1298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.215097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85065728,"num_objects":1298,"num_object_clones":0,"num_object_copies":3894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1298,"num_write_kb":83072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":889,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.625506+0000","last_change":"2024-03-21T14:14:54.743500+0000","last_active":"2024-03-21T14:17:04.625506+0000","last_peered":"2024-03-21T14:17:04.625506+0000","last_clean":"2024-03-21T14:14:52.711337+0000","last_became_active":"2024-03-21T14:14:54.743500+0000","last_became_peered":"2024-03-21T14:14:54.743500+0000","last_unstale":"2024-03-21T14:17:04.625506+0000","last_undegraded":"2024-03-21T14:17:04.625506+0000","last_fullsized":"2024-03-21T14:14:54.728617+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:36.741153+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:36.741153+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686837+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":71,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.624289+0000","last_change":"2024-03-21T14:16:09.598659+0000","last_active":"2024-03-21T14:17:04.624289+0000","last_peered":"2024-03-21T14:17:04.624289+0000","last_clean":"2024-03-21T14:17:04.624289+0000","last_became_active":"2024-03-21T14:16:00.605705+0000","last_became_peered":"2024-03-21T14:16:00.605705+0000","last_unstale":"2024-03-21T14:17:04.624289+0000","last_undegraded":"2024-03-21T14:17:04.624289+0000","last_fullsized":"2024-03-21T14:17:04.624289+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:09.598439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:09.598439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.377213+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,6],"avail_no_missing":["3","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":516,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.627509+0000","last_change":"2024-03-21T14:16:21.865096+0000","last_active":"2024-03-21T14:17:04.627509+0000","last_peered":"2024-03-21T14:17:04.627509+0000","last_clean":"2024-03-21T14:14:52.706578+0000","last_became_active":"2024-03-21T14:16:21.865096+0000","last_became_peered":"2024-03-21T14:16:21.865096+0000","last_unstale":"2024-03-21T14:17:04.627509+0000","last_undegraded":"2024-03-21T14:17:04.627509+0000","last_fullsized":"2024-03-21T14:16:21.859801+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:45.295537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:45.295537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.761964+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":621,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.628419+0000","last_change":"2024-03-21T14:15:39.434842+0000","last_active":"2024-03-21T14:17:04.628419+0000","last_peered":"2024-03-21T14:17:04.628419+0000","last_clean":"2024-03-21T14:17:04.628419+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:17:04.628419+0000","last_undegraded":"2024-03-21T14:17:04.628419+0000","last_fullsized":"2024-03-21T14:17:04.628419+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:15:39.434689+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:15:39.434689+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":514,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625561+0000","last_change":"2024-03-21T14:16:21.864198+0000","last_active":"2024-03-21T14:17:04.625561+0000","last_peered":"2024-03-21T14:17:04.625561+0000","last_clean":"2024-03-21T14:17:04.625561+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:17:04.625561+0000","last_undegraded":"2024-03-21T14:17:04.625561+0000","last_fullsized":"2024-03-21T14:17:04.625561+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T14:16:07.563840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:07.563840+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.400268+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"324'1240","reported_seq":1261,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:07.162901+0000","last_change":"2024-03-21T14:17:01.207904+0000","last_active":"2024-03-21T14:17:07.162901+0000","last_peered":"2024-03-21T14:17:07.162901+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.052312+0000","last_became_peered":"2024-03-21T14:16:51.052312+0000","last_unstale":"2024-03-21T14:17:07.162901+0000","last_undegraded":"2024-03-21T14:17:01.207684+0000","last_fullsized":"2024-03-21T14:16:51.043676+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1240,"log_dups_size":0,"ondisk_log_size":1240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.036790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1240,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":79360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1240}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":644,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627004+0000","last_change":"2024-03-21T14:16:13.521253+0000","last_active":"2024-03-21T14:17:04.627004+0000","last_peered":"2024-03-21T14:17:04.627004+0000","last_clean":"2024-03-21T14:17:04.627004+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:17:04.627004+0000","last_undegraded":"2024-03-21T14:17:04.627004+0000","last_fullsized":"2024-03-21T14:17:04.627004+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:13.521115+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.521115+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:33.283687+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.623439+0000","last_change":"2024-03-21T14:14:54.743852+0000","last_active":"2024-03-21T14:17:04.623439+0000","last_peered":"2024-03-21T14:17:04.623439+0000","last_clean":"2024-03-21T14:14:52.711546+0000","last_became_active":"2024-03-21T14:14:54.743852+0000","last_became_peered":"2024-03-21T14:14:54.743852+0000","last_unstale":"2024-03-21T14:17:04.623439+0000","last_undegraded":"2024-03-21T14:17:04.623439+0000","last_fullsized":"2024-03-21T14:14:54.732762+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:51.612367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:02.083143+0000","last_clean_scrub_stamp":"2024-03-21T14:13:51.612367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686787+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":440,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626979+0000","last_change":"2024-03-21T14:15:02.631337+0000","last_active":"2024-03-21T14:17:04.626979+0000","last_peered":"2024-03-21T14:17:04.626979+0000","last_clean":"2024-03-21T14:17:04.626979+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:17:04.626979+0000","last_undegraded":"2024-03-21T14:17:04.626979+0000","last_fullsized":"2024-03-21T14:17:04.626979+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:15:02.631076+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:15:02.631076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":634,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618217+0000","last_change":"2024-03-21T14:15:59.624035+0000","last_active":"2024-03-21T14:17:04.618217+0000","last_peered":"2024-03-21T14:17:04.618217+0000","last_clean":"2024-03-21T14:17:04.618217+0000","last_became_active":"2024-03-21T14:15:59.607826+0000","last_became_peered":"2024-03-21T14:15:59.607826+0000","last_unstale":"2024-03-21T14:17:04.618217+0000","last_undegraded":"2024-03-21T14:17:04.618217+0000","last_fullsized":"2024-03-21T14:17:04.618217+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'28","last_scrub_stamp":"2024-03-21T14:14:58.038582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:58.038582+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":758,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625018+0000","last_change":"2024-03-21T14:15:31.699369+0000","last_active":"2024-03-21T14:17:04.625018+0000","last_peered":"2024-03-21T14:17:04.625018+0000","last_clean":"2024-03-21T14:17:04.625018+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:17:04.625018+0000","last_undegraded":"2024-03-21T14:17:04.625018+0000","last_fullsized":"2024-03-21T14:17:04.625018+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:15:31.699239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:31.699239+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"324'1161","reported_seq":1182,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:06.474049+0000","last_change":"2024-03-21T14:17:01.210297+0000","last_active":"2024-03-21T14:17:06.474049+0000","last_peered":"2024-03-21T14:17:06.474049+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.053795+0000","last_became_peered":"2024-03-21T14:16:51.053795+0000","last_unstale":"2024-03-21T14:17:06.474049+0000","last_undegraded":"2024-03-21T14:17:01.206110+0000","last_fullsized":"2024-03-21T14:16:51.048364+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1161,"log_dups_size":0,"ondisk_log_size":1161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:07.322836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76087296,"num_objects":1161,"num_object_clones":0,"num_object_copies":3483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1161,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1161,"num_write_kb":74304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1161}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":641,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.617117+0000","last_change":"2024-03-21T14:14:54.739215+0000","last_active":"2024-03-21T14:17:04.617117+0000","last_peered":"2024-03-21T14:17:04.617117+0000","last_clean":"2024-03-21T14:14:52.716527+0000","last_became_active":"2024-03-21T14:14:54.739215+0000","last_became_peered":"2024-03-21T14:14:54.739215+0000","last_unstale":"2024-03-21T14:17:04.617117+0000","last_undegraded":"2024-03-21T14:17:04.617117+0000","last_fullsized":"2024-03-21T14:14:54.724221+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.954782+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.954782+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":580,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618130+0000","last_change":"2024-03-21T14:15:04.115339+0000","last_active":"2024-03-21T14:17:04.618130+0000","last_peered":"2024-03-21T14:17:04.618130+0000","last_clean":"2024-03-21T14:17:04.618130+0000","last_became_active":"2024-03-21T14:13:18.543885+0000","last_became_peered":"2024-03-21T14:13:18.543885+0000","last_unstale":"2024-03-21T14:17:04.618130+0000","last_undegraded":"2024-03-21T14:17:04.618130+0000","last_fullsized":"2024-03-21T14:17:04.618130+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:15:04.115222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.629792+0000","last_change":"2024-03-21T14:17:02.767201+0000","last_active":"2024-03-21T14:17:04.629792+0000","last_peered":"2024-03-21T14:17:04.629792+0000","last_clean":"2024-03-21T14:17:04.629792+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:17:04.629792+0000","last_undegraded":"2024-03-21T14:17:04.629792+0000","last_fullsized":"2024-03-21T14:17:04.629792+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:02.767079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:17:02.767079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:31.502000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.624320+0000","last_change":"2024-03-21T14:16:08.848200+0000","last_active":"2024-03-21T14:17:04.624320+0000","last_peered":"2024-03-21T14:17:04.624320+0000","last_clean":"2024-03-21T14:17:04.624320+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:17:04.624320+0000","last_undegraded":"2024-03-21T14:17:04.624320+0000","last_fullsized":"2024-03-21T14:17:04.624320+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:16:08.847957+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:16:08.847957+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:15.262553+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":558,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618054+0000","last_change":"2024-03-21T14:14:52.054130+0000","last_active":"2024-03-21T14:17:04.618054+0000","last_peered":"2024-03-21T14:17:04.618054+0000","last_clean":"2024-03-21T14:17:04.618054+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:17:04.618054+0000","last_undegraded":"2024-03-21T14:17:04.618054+0000","last_fullsized":"2024-03-21T14:17:04.618054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:52.053970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:52.053970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"324'1178","reported_seq":1202,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:06.466656+0000","last_change":"2024-03-21T14:16:51.052270+0000","last_active":"2024-03-21T14:17:06.466656+0000","last_peered":"2024-03-21T14:17:06.466656+0000","last_clean":"2024-03-21T14:17:06.466656+0000","last_became_active":"2024-03-21T14:16:51.051942+0000","last_became_peered":"2024-03-21T14:16:51.051942+0000","last_unstale":"2024-03-21T14:17:06.466656+0000","last_undegraded":"2024-03-21T14:17:06.466656+0000","last_fullsized":"2024-03-21T14:17:06.466656+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1178,"log_dups_size":0,"ondisk_log_size":1178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:02.736930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77201408,"num_objects":1178,"num_object_clones":0,"num_object_copies":3534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1178,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1178,"num_write_kb":75392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":561,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.617430+0000","last_change":"2024-03-21T14:16:21.860397+0000","last_active":"2024-03-21T14:17:04.617430+0000","last_peered":"2024-03-21T14:17:04.617430+0000","last_clean":"2024-03-21T14:09:26.509792+0000","last_became_active":"2024-03-21T14:16:21.860397+0000","last_became_peered":"2024-03-21T14:16:21.860397+0000","last_unstale":"2024-03-21T14:17:04.617430+0000","last_undegraded":"2024-03-21T14:17:04.617430+0000","last_fullsized":"2024-03-21T14:16:21.852037+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:37.210371+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:37.210371+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:06.298213+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":579,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.626748+0000","last_change":"2024-03-21T14:17:00.985782+0000","last_active":"2024-03-21T14:17:04.626748+0000","last_peered":"2024-03-21T14:17:04.626748+0000","last_clean":"2024-03-21T14:17:04.626748+0000","last_became_active":"2024-03-21T14:16:22.893432+0000","last_became_peered":"2024-03-21T14:16:22.893432+0000","last_unstale":"2024-03-21T14:17:04.626748+0000","last_undegraded":"2024-03-21T14:17:04.626748+0000","last_fullsized":"2024-03-21T14:17:04.626748+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:00.985637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:17:00.985637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:09.581194+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":757,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618547+0000","last_change":"2024-03-21T14:17:03.353514+0000","last_active":"2024-03-21T14:17:04.618547+0000","last_peered":"2024-03-21T14:17:04.618547+0000","last_clean":"2024-03-21T14:17:04.618547+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:17:04.618547+0000","last_undegraded":"2024-03-21T14:17:04.618547+0000","last_fullsized":"2024-03-21T14:17:04.618547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:17:03.353394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.353394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:30.866059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":428,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625672+0000","last_change":"2024-03-21T14:14:53.682726+0000","last_active":"2024-03-21T14:17:04.625672+0000","last_peered":"2024-03-21T14:17:04.625672+0000","last_clean":"2024-03-21T14:17:04.625672+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:17:04.625672+0000","last_undegraded":"2024-03-21T14:17:04.625672+0000","last_fullsized":"2024-03-21T14:17:04.625672+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:53.682640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:53.682640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":621,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.626868+0000","last_change":"2024-03-21T14:14:40.572475+0000","last_active":"2024-03-21T14:17:04.626868+0000","last_peered":"2024-03-21T14:17:04.626868+0000","last_clean":"2024-03-21T14:17:04.626868+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:17:04.626868+0000","last_undegraded":"2024-03-21T14:17:04.626868+0000","last_fullsized":"2024-03-21T14:17:04.626868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:14:40.572343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:40.572343+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"324'1345","reported_seq":1369,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:07.395150+0000","last_change":"2024-03-21T14:16:51.054830+0000","last_active":"2024-03-21T14:17:07.395150+0000","last_peered":"2024-03-21T14:17:07.395150+0000","last_clean":"2024-03-21T14:17:07.395150+0000","last_became_active":"2024-03-21T14:16:51.054526+0000","last_became_peered":"2024-03-21T14:16:51.054526+0000","last_unstale":"2024-03-21T14:17:07.395150+0000","last_undegraded":"2024-03-21T14:17:07.395150+0000","last_fullsized":"2024-03-21T14:17:07.395150+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1345,"log_dups_size":0,"ondisk_log_size":1345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.076219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88145920,"num_objects":1345,"num_object_clones":0,"num_object_copies":4035,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1345,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1345,"num_write_kb":86080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":576,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.627112+0000","last_change":"2024-03-21T14:15:37.646778+0000","last_active":"2024-03-21T14:17:04.627112+0000","last_peered":"2024-03-21T14:17:04.627112+0000","last_clean":"2024-03-21T14:17:04.627112+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:17:04.627112+0000","last_undegraded":"2024-03-21T14:17:04.627112+0000","last_fullsized":"2024-03-21T14:17:04.627112+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:37.646668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.646668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":555,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.619144+0000","last_change":"2024-03-21T14:16:22.893583+0000","last_active":"2024-03-21T14:17:04.619144+0000","last_peered":"2024-03-21T14:17:04.619144+0000","last_clean":"2024-03-21T14:17:04.619144+0000","last_became_active":"2024-03-21T14:16:22.893178+0000","last_became_peered":"2024-03-21T14:16:22.893178+0000","last_unstale":"2024-03-21T14:17:04.619144+0000","last_undegraded":"2024-03-21T14:17:04.619144+0000","last_fullsized":"2024-03-21T14:17:04.619144+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:16.566202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:16.566202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:29.193550+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.618142+0000","last_change":"2024-03-21T14:14:54.739307+0000","last_active":"2024-03-21T14:17:04.618142+0000","last_peered":"2024-03-21T14:17:04.618142+0000","last_clean":"2024-03-21T14:14:52.715781+0000","last_became_active":"2024-03-21T14:14:54.739307+0000","last_became_peered":"2024-03-21T14:14:54.739307+0000","last_unstale":"2024-03-21T14:17:04.618142+0000","last_undegraded":"2024-03-21T14:17:04.618142+0000","last_fullsized":"2024-03-21T14:14:54.726477+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:45.602994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:45.602994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347488+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":619,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618249+0000","last_change":"2024-03-21T14:15:32.019352+0000","last_active":"2024-03-21T14:17:04.618249+0000","last_peered":"2024-03-21T14:17:04.618249+0000","last_clean":"2024-03-21T14:17:04.618249+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:17:04.618249+0000","last_undegraded":"2024-03-21T14:17:04.618249+0000","last_fullsized":"2024-03-21T14:17:04.618249+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:15:32.019116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:32.019116+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"324'1128","reported_seq":1149,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:06.476196+0000","last_change":"2024-03-21T14:17:01.207830+0000","last_active":"2024-03-21T14:17:06.476196+0000","last_peered":"2024-03-21T14:17:06.476196+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.052954+0000","last_became_peered":"2024-03-21T14:16:51.052954+0000","last_unstale":"2024-03-21T14:17:06.476196+0000","last_undegraded":"2024-03-21T14:17:01.207619+0000","last_fullsized":"2024-03-21T14:16:51.046136+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1128,"log_dups_size":0,"ondisk_log_size":1128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.656424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73924608,"num_objects":1128,"num_object_clones":0,"num_object_copies":3384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1128,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1128,"num_write_kb":72192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1128}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":560,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.624501+0000","last_change":"2024-03-21T14:16:21.860630+0000","last_active":"2024-03-21T14:17:04.624501+0000","last_peered":"2024-03-21T14:17:04.624501+0000","last_clean":"2024-03-21T14:09:26.509792+0000","last_became_active":"2024-03-21T14:16:21.860630+0000","last_became_peered":"2024-03-21T14:16:21.860630+0000","last_unstale":"2024-03-21T14:17:04.624501+0000","last_undegraded":"2024-03-21T14:17:04.624501+0000","last_fullsized":"2024-03-21T14:16:21.852422+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:36.244697+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:36.244697+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:05.813250+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":109,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.629491+0000","last_change":"2024-03-21T14:14:54.726685+0000","last_active":"2024-03-21T14:17:04.629491+0000","last_peered":"2024-03-21T14:17:04.629491+0000","last_clean":"2024-03-21T14:02:40.461300+0000","last_became_active":"2024-03-21T14:14:54.726685+0000","last_became_peered":"2024-03-21T14:14:54.726685+0000","last_unstale":"2024-03-21T14:17:04.629491+0000","last_undegraded":"2024-03-21T14:14:54.724018+0000","last_fullsized":"2024-03-21T14:14:54.723895+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:13:36.334881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.334881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:07.762036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":582,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.628906+0000","last_change":"2024-03-21T14:15:30.432822+0000","last_active":"2024-03-21T14:17:04.628906+0000","last_peered":"2024-03-21T14:17:04.628906+0000","last_clean":"2024-03-21T14:17:04.628906+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:17:04.628906+0000","last_undegraded":"2024-03-21T14:17:04.628906+0000","last_fullsized":"2024-03-21T14:17:04.628906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:30.432721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_clean_scrub_stamp":"2024-03-21T14:15:30.432721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":248,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618231+0000","last_change":"2024-03-21T14:15:05.093514+0000","last_active":"2024-03-21T14:17:04.618231+0000","last_peered":"2024-03-21T14:17:04.618231+0000","last_clean":"2024-03-21T14:17:04.618231+0000","last_became_active":"2024-03-21T14:12:00.530704+0000","last_became_peered":"2024-03-21T14:12:00.530704+0000","last_unstale":"2024-03-21T14:17:04.618231+0000","last_undegraded":"2024-03-21T14:17:04.618231+0000","last_fullsized":"2024-03-21T14:17:04.618231+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:05.093393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:05.093393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":614,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.618227+0000","last_change":"2024-03-21T14:14:54.732979+0000","last_active":"2024-03-21T14:17:04.618227+0000","last_peered":"2024-03-21T14:17:04.618227+0000","last_clean":"2024-03-21T14:14:52.716824+0000","last_became_active":"2024-03-21T14:14:54.732979+0000","last_became_peered":"2024-03-21T14:14:54.732979+0000","last_unstale":"2024-03-21T14:17:04.618227+0000","last_undegraded":"2024-03-21T14:14:54.728543+0000","last_fullsized":"2024-03-21T14:14:54.728421+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:12:53.529362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.529362+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:08.347474+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"324'1331","reported_seq":1355,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:07.375353+0000","last_change":"2024-03-21T14:16:51.056904+0000","last_active":"2024-03-21T14:17:07.375353+0000","last_peered":"2024-03-21T14:17:07.375353+0000","last_clean":"2024-03-21T14:17:07.375353+0000","last_became_active":"2024-03-21T14:16:51.056577+0000","last_became_peered":"2024-03-21T14:16:51.056577+0000","last_unstale":"2024-03-21T14:17:07.375353+0000","last_undegraded":"2024-03-21T14:17:07.375353+0000","last_fullsized":"2024-03-21T14:17:07.375353+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":1331,"log_dups_size":0,"ondisk_log_size":1331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:02.310663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87228416,"num_objects":1331,"num_object_clones":0,"num_object_copies":3993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1331,"num_write_kb":85184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":889,"reported_epoch":324,"state":"active+undersized","last_fresh":"2024-03-21T14:17:04.622629+0000","last_change":"2024-03-21T14:14:54.737618+0000","last_active":"2024-03-21T14:17:04.622629+0000","last_peered":"2024-03-21T14:17:04.622629+0000","last_clean":"2024-03-21T14:14:52.710614+0000","last_became_active":"2024-03-21T14:14:54.737618+0000","last_became_peered":"2024-03-21T14:14:54.737618+0000","last_unstale":"2024-03-21T14:17:04.622629+0000","last_undegraded":"2024-03-21T14:17:04.622629+0000","last_fullsized":"2024-03-21T14:14:54.727680+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.696891+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.696891+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686827+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":573,"reported_epoch":324,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:04.618684+0000","last_change":"2024-03-21T14:16:00.603039+0000","last_active":"2024-03-21T14:17:04.618684+0000","last_peered":"2024-03-21T14:17:04.618684+0000","last_clean":"2024-03-21T14:17:04.618684+0000","last_became_active":"2024-03-21T14:16:00.602424+0000","last_became_peered":"2024-03-21T14:16:00.602424+0000","last_unstale":"2024-03-21T14:17:04.618684+0000","last_undegraded":"2024-03-21T14:17:04.618684+0000","last_fullsized":"2024-03-21T14:17:04.618684+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:33.112232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:33.112232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"279'10","reported_seq":262,"reported_epoch":324,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:04.624451+0000","last_change":"2024-03-21T14:14:54.743410+0000","last_active":"2024-03-21T14:17:04.624451+0000","last_peered":"2024-03-21T14:17:04.624451+0000","last_clean":"2024-03-21T14:05:32.338633+0000","last_became_active":"2024-03-21T14:14:54.743410+0000","last_became_peered":"2024-03-21T14:14:54.743410+0000","last_unstale":"2024-03-21T14:17:04.624451+0000","last_undegraded":"2024-03-21T14:14:54.727338+0000","last_fullsized":"2024-03-21T14:14:54.727093+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:14:05.166173+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:14:05.166173+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:04.686816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.625818+0000","last_change":"2024-03-21T14:15:28.709389+0000","last_active":"2024-03-21T14:17:04.625818+0000","last_peered":"2024-03-21T14:17:04.625818+0000","last_clean":"2024-03-21T14:17:04.625818+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:17:04.625818+0000","last_undegraded":"2024-03-21T14:17:04.625818+0000","last_fullsized":"2024-03-21T14:17:04.625818+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:28.709149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:28.709149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":623,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-21T14:17:04.618773+0000","last_change":"2024-03-21T14:15:40.120088+0000","last_active":"2024-03-21T14:17:04.618773+0000","last_peered":"2024-03-21T14:17:04.618773+0000","last_clean":"2024-03-21T14:17:04.618773+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:17:04.618773+0000","last_undegraded":"2024-03-21T14:17:04.618773+0000","last_fullsized":"2024-03-21T14:17:04.618773+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:15:40.119580+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:15:40.119580+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":1276248064,"num_objects":19474,"num_object_clones":0,"num_object_copies":58422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19474,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19474,"num_write_kb":1246336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3236364288,"data_stored":3236364288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19474,"ondisk_log_size":19474,"up":40,"acting":40,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":808,"num_read_kb":803,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":435019,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":55,"num_write":32,"num_write_kb":438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":429358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":26,"acting":26,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":85,"acting":86,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":754,"ondisk_log_size":754,"up":85,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":83,"acting":85,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":79,"acting":82,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103347,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1046420,"kb_used_data":455756,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92698732,"statfs":{"total":95995035648,"available":94923501568,"internally_reserved":0,"allocated":466694144,"data_stored":466044470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":604830154},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430392,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575059,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952424,"kb_used_data":317340,"kb_used_omap":1,"kb_used_meta":635070,"kb_avail":92792728,"statfs":{"total":95995035648,"available":95019753472,"internally_reserved":0,"allocated":324956160,"data_stored":324290409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":650312133},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738584,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1125764,"kb_used_data":653116,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":92619388,"statfs":{"total":95995035648,"available":94842253312,"internally_reserved":0,"allocated":668790784,"data_stored":668156094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":483981765},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3212408,"kb_used_data":2753648,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":90532744,"statfs":{"total":95995035648,"available":92705529856,"internally_reserved":0,"allocated":2819735552,"data_stored":2819353006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":469760453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182599,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":828048,"kb_used_data":309576,"kb_used_omap":1,"kb_used_meta":518462,"kb_avail":92917104,"statfs":{"total":95995035648,"available":95147114496,"internally_reserved":0,"allocated":317005824,"data_stored":316359461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":530905543},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019075,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1329684,"kb_used_data":879244,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":92415468,"statfs":{"total":95995035648,"available":94633439232,"internally_reserved":0,"allocated":900345856,"data_stored":899695007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692030,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":979232,"kb_used_data":416340,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":92765920,"statfs":{"total":95995035648,"available":94992302080,"internally_reserved":0,"allocated":426332160,"data_stored":425674644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":683343872,"data_stored":683343872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":346816512,"data_stored":346816512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":328990720,"data_stored":328990720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":915472384,"data_stored":915472384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":482410496,"data_stored":482410496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":479330304,"data_stored":479330304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:17:10.266 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-21T14:17:10.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:10.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:10.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:10.455 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:10.490 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:10.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:10.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:10.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:10.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:10.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:10.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:10.766 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:10.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:10.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:10.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:10.902 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:10.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:10.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:11.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:17:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:17:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:17:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:17:11.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[29576]: pgmap v668: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 150 MiB/s wr, 2.40k op/s; 9635/59013 objects degraded (16.327%); 6/59013 objects misplaced (0.010%) 2024-03-21T14:17:11.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:17:11.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:17:11.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:17:11.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:17:11.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:10 smithi003 ceph-mon[40510]: pgmap v668: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 150 MiB/s wr, 2.40k op/s; 9635/59013 objects degraded (16.327%); 6/59013 objects misplaced (0.010%) 2024-03-21T14:17:11.041 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:11.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:11.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:11.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:11.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:10 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:17:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:10 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:17:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:10 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:17:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:10 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:17:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:10 smithi118 ceph-mon[33862]: pgmap v668: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.2 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.3 KiB/s rd, 150 MiB/s wr, 2.40k op/s; 9635/59013 objects degraded (16.327%); 6/59013 objects misplaced (0.010%) 2024-03-21T14:17:11.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 15 30630 30615 191.311 162.812 0.00669909 0.00522199 2024-03-21T14:17:11.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:11.318 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:11.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.461 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:11.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:11.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:11.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:11.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:11.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:11.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:11.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:11.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:11.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:11.973 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:11 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:17:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:11 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:17:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:11 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:17:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:11 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:17:12.009 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:12.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:12.148 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:12.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:11 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:17:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:11 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:17:12.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 33180 33164 188.401 159.312 0.00679114 0.00530353 2024-03-21T14:17:12.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:12.286 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:12.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:12.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:12.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:12.559 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:12.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:12.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:12.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:12.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:12.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:12.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:17:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:12.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:12.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:12.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:12.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:17:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:17:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[29576]: 6.2 scrub starts 2024-03-21T14:17:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[29576]: 6.2 scrub ok 2024-03-21T14:17:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[29576]: pgmap v670: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 740 B/s rd, 203 MiB/s wr, 3.25k op/s; 11996/75606 objects degraded (15.866%); 6/75606 objects misplaced (0.008%) 2024-03-21T14:17:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:17:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:17:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[40510]: 6.2 scrub starts 2024-03-21T14:17:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[40510]: 6.2 scrub ok 2024-03-21T14:17:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:12 smithi003 ceph-mon[40510]: pgmap v670: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 740 B/s rd, 203 MiB/s wr, 3.25k op/s; 11996/75606 objects degraded (15.866%); 6/75606 objects misplaced (0.008%) 2024-03-21T14:17:13.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:13.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:12 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:17:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:12 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:17:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:12 smithi118 ceph-mon[33862]: 6.2 scrub starts 2024-03-21T14:17:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:12 smithi118 ceph-mon[33862]: 6.2 scrub ok 2024-03-21T14:17:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:12 smithi118 ceph-mon[33862]: pgmap v670: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 1.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 740 B/s rd, 203 MiB/s wr, 3.25k op/s; 11996/75606 objects degraded (15.866%); 6/75606 objects misplaced (0.008%) 2024-03-21T14:17:13.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 35600 35584 185.304 151.25 0.00488067 0.00539121 2024-03-21T14:17:13.221 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:13.257 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:13.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:13.395 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:13.497 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:13.530 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:13.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:13.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:13.754 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:17:13.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:13.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:17:13.792 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:13.794 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:17:13.881 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:17:13.894 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:13.918 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:17:13.928 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:13.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:13.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:13.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/582041711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:13.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/582041711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:13.957 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:17:13.998 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:17:13.999 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:17:14.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:14.050 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:17:14.050 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:17:14.050 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:17:14.050 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:17:14.050 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:17:14.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:14.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/582041711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:14.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 37361 37345 179.516 110.062 0.00527977 0.00556568 2024-03-21T14:17:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:14.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:14.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:14.452 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.489 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:14.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:14.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:14.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.629 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:14.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:14.731 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.766 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:14.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:14.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:14.904 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:14.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:17:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[29576]: pgmap v671: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 232 MiB/s wr, 3.71k op/s; 15944/97299 objects degraded (16.387%); 6/97299 objects misplaced (0.006%) 2024-03-21T14:17:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:17:15.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 15944/97299 objects degraded (16.387%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:17:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[40510]: pgmap v671: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 232 MiB/s wr, 3.71k op/s; 15944/97299 objects degraded (16.387%); 6/97299 objects misplaced (0.006%) 2024-03-21T14:17:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:17:15.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:14 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 15944/97299 objects degraded (16.387%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:15.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:15.039 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:17:15.062 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:17:15.141 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:15.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:15.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:14 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:17:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:14 smithi118 ceph-mon[33862]: pgmap v671: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 232 MiB/s wr, 3.71k op/s; 15944/97299 objects degraded (16.387%); 6/97299 objects misplaced (0.006%) 2024-03-21T14:17:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:14 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:17:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:14 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 15944/97299 objects degraded (16.387%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:15.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 39586 39570 176.626 139.062 0.00326423 0.00565693 2024-03-21T14:17:15.267 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:17:15.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:17:15.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:17:15.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:15.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:15.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:15.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:15.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:15.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:15.718 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:15.728 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:15.760 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T14:17:15.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:15.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:15.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:15.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:16.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:16.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:17:16.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:17:16.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:17:16.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:17:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:17:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:17:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:17:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:15 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:17:16.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:16.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:16.173 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:15 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:17:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:15 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:17:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:15 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:17:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:15 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:17:16.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 41687 41671 173.605 131.312 0.00827112 0.0057547 2024-03-21T14:17:16.275 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:16.311 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:16.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:16.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.451 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:16.542 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:16.552 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:16.588 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:16.650 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:17:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:17:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:17:16.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:16.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:16.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:16.873 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:16.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:16.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:16 smithi003 ceph-mon[29576]: pgmap v672: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.59k op/s; 15944/97299 objects degraded (16.387%); 6/97299 objects misplaced (0.006%) 2024-03-21T14:17:16.966 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:16 smithi003 ceph-mon[40510]: pgmap v672: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.59k op/s; 15944/97299 objects degraded (16.387%); 6/97299 objects misplaced (0.006%) 2024-03-21T14:17:16.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:17.011 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:17.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:17:17.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:17.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:16 smithi118 ceph-mon[33862]: pgmap v672: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 100 MiB/s wr, 1.59k op/s; 15944/97299 objects degraded (16.387%); 6/97299 objects misplaced (0.006%) 2024-03-21T14:17:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:17.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:17.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 16 43821 43805 171.09 133.375 0.00960257 0.00584064 2024-03-21T14:17:17.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:17.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:17.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:17.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:17.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:17.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:17:17.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:17.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:17.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:17.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:17.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:17.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:17.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:17.840 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:17.866 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:17.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:17.979 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:17.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:18.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:17:18.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:17:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:17:18.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:17 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:17:18.081 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:18.116 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:18.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:17:18.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:17 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:17:18.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 45913 45897 168.716 130.75 0.00573704 0.00592247 2024-03-21T14:17:18.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:18.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:18.358 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:18.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:18.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:18.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:18.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:18.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:18.636 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:18.672 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:18.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:18.774 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:18.804 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:18.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:18.906 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:18.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:18.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:18.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:19.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:19.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:19.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:19.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:18 smithi118 ceph-mon[33862]: pgmap v673: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 150 MiB/s wr, 2.40k op/s; 21389/129804 objects degraded (16.478%); 6/129804 objects misplaced (0.005%) 2024-03-21T14:17:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:18 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:17:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:18 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:17:19.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 16 47850 47834 166.068 121.062 0.00862553 0.00601689 2024-03-21T14:17:19.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[29576]: pgmap v673: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 150 MiB/s wr, 2.40k op/s; 21389/129804 objects degraded (16.478%); 6/129804 objects misplaced (0.005%) 2024-03-21T14:17:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:17:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:17:19.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:19.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[40510]: pgmap v673: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.0 KiB/s rd, 150 MiB/s wr, 2.40k op/s; 21389/129804 objects degraded (16.478%); 6/129804 objects misplaced (0.005%) 2024-03-21T14:17:19.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:17:19.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:18 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:17:19.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:19.349 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:19.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:19.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:19.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:19.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:19.612 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:19.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:19.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:19.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:19.887 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:19.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:19.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:19.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:20.065 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:19 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:17:20.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:19 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:17:20.066 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:19 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 21389/129804 objects degraded (16.478%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:20.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:19 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:17:20.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:19 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:17:20.066 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:19 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 21389/129804 objects degraded (16.478%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:20.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:20.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:19 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:17:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:19 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:17:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:19 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 21389/129804 objects degraded (16.478%), 22 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:20.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 49758 49742 163.604 119.25 0.00590151 0.00610734 2024-03-21T14:17:20.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:20.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:20.399 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:20.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:20.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:20.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:20.707 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:20.809 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:20.839 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:20.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:20.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:20.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:21.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.116 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:21.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:20 smithi118 ceph-mon[33862]: pgmap v674: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.36k op/s; 21389/129804 objects degraded (16.478%); 6/129804 objects misplaced (0.005%) 2024-03-21T14:17:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:20 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:17:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:20 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:17:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:20 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:17:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:20 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:17:21.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:17:21.204595+0000 min lat: 0.00112315 max lat: 0.150126 avg lat: 0.00619788 2024-03-21T14:17:21.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:17:21.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 51616 51600 161.23 116.125 0.00343731 0.00619788 2024-03-21T14:17:21.217 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[29576]: pgmap v674: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.36k op/s; 21389/129804 objects degraded (16.478%); 6/129804 objects misplaced (0.005%) 2024-03-21T14:17:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:17:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:17:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:17:21.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:17:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[40510]: pgmap v674: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 147 MiB/s wr, 2.36k op/s; 21389/129804 objects degraded (16.478%); 6/129804 objects misplaced (0.005%) 2024-03-21T14:17:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:17:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:17:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:17:21.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:20 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:17:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:21.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:21.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:21.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:21.497 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:21.533 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:21.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:21.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:21.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:21.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:21.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:21.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:21.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:21.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:22.041 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:22.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:22.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:22.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:21 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:17:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:21 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:17:22.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 53459 53443 159.037 115.188 0.00556972 0.00628281 2024-03-21T14:17:22.216 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:22.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:21 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:17:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:21 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:17:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:21 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:17:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:21 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:17:22.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:22.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:22.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:22.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:22.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:22.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:22.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:22.771 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:22.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:22.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:22.909 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:22.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:23.011 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:23.045 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:23.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:23.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:23.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:22 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:17:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:22 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:17:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:22 smithi118 ceph-mon[33862]: pgmap v675: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 858 B/s rd, 146 MiB/s wr, 2.34k op/s; 23231/142845 objects degraded (16.263%); 6/142845 objects misplaced (0.004%) 2024-03-21T14:17:23.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:23.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:23.203 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 15 55198 55183 156.751 108.75 0.0100488 0.00637524 2024-03-21T14:17:23.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:22 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:17:23.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:22 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:17:23.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:22 smithi003 ceph-mon[29576]: pgmap v675: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 858 B/s rd, 146 MiB/s wr, 2.34k op/s; 23231/142845 objects degraded (16.263%); 6/142845 objects misplaced (0.004%) 2024-03-21T14:17:23.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:22 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:17:23.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:22 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:17:23.238 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:22 smithi003 ceph-mon[40510]: pgmap v675: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 2.9 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 858 B/s rd, 146 MiB/s wr, 2.34k op/s; 23231/142845 objects degraded (16.263%); 6/142845 objects misplaced (0.004%) 2024-03-21T14:17:23.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:23.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:23.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:23.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:23.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:23.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:23.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:23.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:23.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:23.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:23.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:23.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221713 7.f 7:fc6b6dba:::benchmark_data_smithi003_80362_object100696:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.38586283500001, 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93373331800000003, 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.719 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221713 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197978+0000", 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197980+0000", 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.705e-06 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.197988+0000", 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2139999999999996e-06 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198007+0000", 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9247999999999999e-05 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198018+0000", 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0268000000000001e-05 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198077+0000", 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9280999999999997e-05 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.198085+0000", 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7959999999999996e-06 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119714+0000", 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92162893099999998 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.119872+0000", 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000158045 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.120011+0000", 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013935900000000001 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131462+0000", 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011450777000000001 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131476+0000", 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4005999999999999e-05 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131627+0000", 2024-03-21T14:17:23.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000150791 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131673+0000", 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6919000000000003e-05 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131711+0000", 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7978000000000001e-05 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221725 7.f 7:fe480980:::benchmark_data_smithi003_80362_object100708:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.358047833, 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90603860999999997, 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221725 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225793+0000", 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225795+0000", 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0959999999999999e-06 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.728 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225804+0000", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8510000000000005e-06 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225818+0000", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3557000000000001e-05 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225828+0000", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0644e-05 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225939+0000", 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110461 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225950+0000", 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1712999999999999e-05 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123657+0000", 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.897706589 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123780+0000", 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012321000000000001 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.123890+0000", 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110142 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131508+0000", 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0076180889999999998 2024-03-21T14:17:23.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131594+0000", 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5677e-05 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131747+0000", 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015268700000000001 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131788+0000", 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1032999999999998e-05 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131832+0000", 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3860999999999999e-05 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221726 7.f 7:fbed2f09:::benchmark_data_smithi003_80362_object100709:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:17:23.733 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.357991578, 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91368115900000002, 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221726 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225849+0000", 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225850+0000", 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7000000000000005e-07 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225854+0000", 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5920000000000001e-06 2024-03-21T14:17:23.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225861+0000", 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4329999999999998e-06 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225867+0000", 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2999999999999998e-06 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225979+0000", 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111437 2024-03-21T14:17:23.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.225986+0000", 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9e-06 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127686+0000", 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90170028599999996 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127815+0000", 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012929099999999999 2024-03-21T14:17:23.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.127949+0000", 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013351500000000001 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139334+0000", 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011385072 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139378+0000", 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3952999999999997e-05 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139425+0000", 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6764999999999997e-05 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139479+0000", 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4245e-05 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139531+0000", 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1799999999999999e-05 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221737 7.f 7:f1651cbf:::benchmark_data_smithi003_80362_object100720:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.346326127, 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90934260099999997, 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.740 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221737 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237515+0000", 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237516+0000", 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0100000000000001e-06 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237521+0000", 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5430000000000001e-06 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237538+0000", 2024-03-21T14:17:23.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7096e-05 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237547+0000", 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9870000000000005e-06 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237607+0000", 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9367000000000003e-05 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.237616+0000", 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5820000000000008e-06 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131857+0000", 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89424055899999999 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.131984+0000", 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012695299999999999 2024-03-21T14:17:23.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.132098+0000", 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113804 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139390+0000", 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072926889999999998 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139610+0000", 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000219626 2024-03-21T14:17:23.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146690+0000", 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0070798809999999997 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146801+0000", 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110599 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146857+0000", 2024-03-21T14:17:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6904999999999998e-05 2024-03-21T14:17:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221741 7.f 7:fa1cb433:::benchmark_data_smithi003_80362_object100724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.34240727099996, 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90562954600000001, 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221741 2024-03-21T14:17:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241434+0000", 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241435+0000", 2024-03-21T14:17:23.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4470000000000001e-06 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241442+0000", 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2640000000000002e-06 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241460+0000", 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7186000000000001e-05 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241469+0000", 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9719999999999998e-06 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241541+0000", 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2210000000000002e-05 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.241551+0000", 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9399999999999997e-06 2024-03-21T14:17:23.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135494+0000", 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89394319300000002 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135628+0000", 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013400099999999999 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.135788+0000", 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016006200000000001 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139568+0000", 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0037803009999999998 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139659+0000", 2024-03-21T14:17:23.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0662999999999996e-05 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146934+0000", 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0072749190000000004 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.147017+0000", 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2694000000000005e-05 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.147063+0000", 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6693999999999997e-05 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221746 7.f 7:fec8ebdb:::benchmark_data_smithi003_80362_object100729:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.33784928700004, 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90374667900000005, 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.755 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221746 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.756 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245992+0000", 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245993+0000", 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.432e-06 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.757 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.245999+0000", 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.088e-06 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246013+0000", 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4158999999999999e-05 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.758 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246022+0000", 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7050000000000005e-06 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246089+0000", 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6668999999999995e-05 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.759 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.246098+0000", 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6609999999999995e-06 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139691+0000", 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89359230599999995 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.760 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139805+0000", 2024-03-21T14:17:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114026 2024-03-21T14:17:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.761 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.139913+0000", 2024-03-21T14:17:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108721 2024-03-21T14:17:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.762 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.146871+0000", 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069575490000000004 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149464+0000", 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025927440000000001 2024-03-21T14:17:23.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149513+0000", 2024-03-21T14:17:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8949000000000002e-05 2024-03-21T14:17:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.764 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149686+0000", 2024-03-21T14:17:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000173308 2024-03-21T14:17:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.765 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149738+0000", 2024-03-21T14:17:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2361999999999998e-05 2024-03-21T14:17:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221753 7.f 7:f88a5a14:::benchmark_data_smithi003_80362_object100736:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:17:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.33177651699998, 2024-03-21T14:17:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89806965699999997, 2024-03-21T14:17:23.767 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221753 2024-03-21T14:17:23.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.769 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252064+0000", 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.770 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252065+0000", 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0200000000000002e-07 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.771 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252069+0000", 2024-03-21T14:17:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7479999999999999e-06 2024-03-21T14:17:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252076+0000", 2024-03-21T14:17:23.772 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252083+0000", 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5230000000000001e-06 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.774 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252137+0000", 2024-03-21T14:17:23.774 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4317999999999997e-05 2024-03-21T14:17:23.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.774 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.774 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.252147+0000", 2024-03-21T14:17:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0257e-05 2024-03-21T14:17:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.775 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143316+0000", 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89116910800000004 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143431+0000", 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011446300000000001 2024-03-21T14:17:23.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.777 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.777 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.143540+0000", 2024-03-21T14:17:23.777 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108909 2024-03-21T14:17:23.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.777 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.778 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.778 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149748+0000", 2024-03-21T14:17:23.778 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062084599999999998 2024-03-21T14:17:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149786+0000", 2024-03-21T14:17:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7997000000000002e-05 2024-03-21T14:17:23.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.780 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.780 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.149826+0000", 2024-03-21T14:17:23.780 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9373999999999999e-05 2024-03-21T14:17:23.780 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.780 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150000+0000", 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017394 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.781 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150134+0000", 2024-03-21T14:17:23.782 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013446700000000001 2024-03-21T14:17:23.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.782 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.782 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.782 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.783 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221755 7.f 7:f52429fb:::benchmark_data_smithi003_80362_object100738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.783 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:17:23.783 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.32711876300004, 2024-03-21T14:17:23.783 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.90123331200000001, 2024-03-21T14:17:23.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.784 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.784 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.785 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.785 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221755 2024-03-21T14:17:23.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.785 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.785 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.785 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:17:23.786 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.786 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.786 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.786 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256722+0000", 2024-03-21T14:17:23.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.787 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.787 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.787 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256723+0000", 2024-03-21T14:17:23.787 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.223e-06 2024-03-21T14:17:23.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.788 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.788 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.788 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256729+0000", 2024-03-21T14:17:23.788 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7960000000000001e-06 2024-03-21T14:17:23.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256742+0000", 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2394e-05 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.789 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256750+0000", 2024-03-21T14:17:23.790 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9270000000000005e-06 2024-03-21T14:17:23.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.790 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.790 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.790 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256802+0000", 2024-03-21T14:17:23.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2578999999999999e-05 2024-03-21T14:17:23.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.791 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.791 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.791 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.256811+0000", 2024-03-21T14:17:23.791 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1349999999999998e-06 2024-03-21T14:17:23.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.792 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.792 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.792 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150166+0000", 2024-03-21T14:17:23.792 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89335426299999998 2024-03-21T14:17:23.792 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150275+0000", 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109422 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.150383+0000", 2024-03-21T14:17:23.793 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108297 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157790+0000", 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074066669999999996 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157802+0000", 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2245e-05 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157842+0000", 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0120000000000002e-05 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157908+0000", 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6168999999999996e-05 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.157956+0000", 2024-03-21T14:17:23.796 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7074999999999999e-05 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.24692.0:221760 7.f 7:f2fb3067:::benchmark_data_smithi003_80362_object100743:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 554.299659744, 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.89272659300000001, 2024-03-21T14:17:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.24692", 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2094866467", 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 221760 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.798 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284181+0000", 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284183+0000", 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.711e-06 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284190+0000", 2024-03-21T14:17:23.799 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5209999999999997e-06 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284207+0000", 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6325e-05 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284215+0000", 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4319999999999999e-06 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.800 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284280+0000", 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5042000000000005e-05 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:09.284292+0000", 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2099000000000001e-05 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154224+0000", 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86993135600000004 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.801 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154329+0000", 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010535300000000001 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.154439+0000", 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011024 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.166586+0000", 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.012147120000000001 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.802 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.166631+0000", 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4297e-05 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176731+0000", 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010100074000000001 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176826+0000", 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5345000000000003e-05 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.803 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:08:10.176908+0000", 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1678e-05 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30436 8.f 8:f8a7c51e:::benchmark_data_smithi003_110242_object30435:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 478.728958492, 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12503477399999999, 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.804 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30436 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854882+0000", 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854883+0000", 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854905+0000", 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1386000000000002e-05 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.806 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854915+0000", 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0138e-05 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.854922+0000", 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2810000000000003e-06 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860219+0000", 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052964709999999996 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.807 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860345+0000", 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012642600000000001 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.860468+0000", 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000122581 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866120+0000", 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0056519700000000001 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.808 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866164+0000", 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4041000000000001e-05 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866321+0000", 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015716800000000001 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.809 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866370+0000", 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8786999999999999e-05 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979917+0000", 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11354760999999999 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.810 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30438 8.f 8:fe80adc5:::benchmark_data_smithi003_110242_object30437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 478.72872448200002, 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12806651599999999, 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30438 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.811 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855116+0000", 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855117+0000", 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8999999999999995e-07 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.812 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855163+0000", 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6081000000000001e-05 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855173+0000", 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6180000000000001e-06 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.855180+0000", 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.81e-06 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.813 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863014+0000", 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078345389999999997 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863134+0000", 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000119196 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.863267+0000", 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000133738 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.814 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.866211+0000", 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002943373 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979935+0000", 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.113724297 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.982232+0000", 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002297112 2024-03-21T14:17:23.815 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.982341+0000", 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010846800000000001 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983183+0000", 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084249399999999995 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.816 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:30445 8.f 8:f69cc62f:::benchmark_data_smithi003_110242_object30444:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e167)", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 478.725573532, 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12520763700000001, 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 30445 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:17:23.817 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858267+0000", 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858269+0000", 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.344e-06 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.818 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858324+0000", 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4917000000000003e-05 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858335+0000", 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1732999999999999e-05 2024-03-21T14:17:23.819 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.858344+0000", 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2150000000000001e-06 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.820 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864522+0000", 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061783519999999998 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864635+0000", 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113094 2024-03-21T14:17:23.821 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.822 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.822 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:17:23.822 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.864765+0000", 2024-03-21T14:17:23.822 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000129839 2024-03-21T14:17:23.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.979940+0000", 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.115175427 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.823 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.980642+0000", 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00070184399999999997 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983236+0000", 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0025941480000000001 2024-03-21T14:17:23.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983433+0000", 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00019648300000000001 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.825 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:09:24.983475+0000", 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2240999999999998e-05 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.826 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:257519 8.f 8:f1508789:::benchmark_data_smithi003_110242_object126602:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-21T14:17:23.827 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:47.474061+0000", 2024-03-21T14:17:23.827 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 276.10977975899999, 2024-03-21T14:17:23.827 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13904603099999999, 2024-03-21T14:17:23.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.827 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.827 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 257519 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.828 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474061+0000", 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474061+0000", 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474063+0000", 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.699e-06 2024-03-21T14:17:23.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474070+0000", 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7689999999999999e-06 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474081+0000", 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1773000000000001e-05 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.830 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474089+0000", 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7300000000000005e-06 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474164+0000", 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4708000000000004e-05 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474296+0000", 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013197299999999999 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.831 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.474416+0000", 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120204 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.479273+0000", 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0048565070000000004 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.479289+0000", 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6345999999999999e-05 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.832 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.612992+0000", 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.133703243 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.613056+0000", 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3341000000000002e-05 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.613107+0000", 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1737999999999998e-05 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.833 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:257520 8.f 8:f44948a3:::benchmark_data_smithi003_110242_object126603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:47.477016+0000", 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 276.10682485400002, 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13916204099999999, 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 257520 2024-03-21T14:17:23.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477016+0000", 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477016+0000", 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477017+0000", 2024-03-21T14:17:23.835 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2279999999999999e-06 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477024+0000", 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9870000000000002e-06 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477035+0000", 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1062e-05 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477042+0000", 2024-03-21T14:17:23.836 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0899999999999999e-06 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477103+0000", 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0429e-05 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477218+0000", 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011553900000000001 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.477343+0000", 2024-03-21T14:17:23.837 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124246 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.480577+0000", 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0032338620000000001 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.581157+0000", 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.10058088699999999 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.838 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.616079+0000", 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.034921761000000003 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.616133+0000", 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3550000000000001e-05 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.616178+0000", 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5399999999999999e-05 2024-03-21T14:17:23.839 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:257524 8.f 8:fcb29f6b:::benchmark_data_smithi003_110242_object126607:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:47.478527+0000", 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 276.105313803, 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13783599299999999, 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:23.840 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 257524 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.478527+0000", 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.478527+0000", 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.841 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.478528+0000", 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6599999999999995e-07 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.478532+0000", 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9609999999999999e-06 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.478540+0000", 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5839999999999995e-06 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.842 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.478546+0000", 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4750000000000001e-06 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.479304+0000", 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075841099999999996 2024-03-21T14:17:23.843 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.479371+0000", 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6717999999999994e-05 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.479435+0000", 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4051000000000001e-05 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.480696+0000", 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012604339999999999 2024-03-21T14:17:23.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.581273+0000", 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.100577203 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.616235+0000", 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.034962015999999999 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.845 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.616322+0000", 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7237000000000005e-05 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:47.616363+0000", 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1137000000000003e-05 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:40338 9.c 9:3e1f7e83:::benchmark_data_smithi003_135814_object40337:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e252)", 2024-03-21T14:17:23.846 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:13:35.153246+0000", 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 228.43059461799999, 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13604575199999999, 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 40338 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.847 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153246+0000", 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153246+0000", 2024-03-21T14:17:23.848 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153247+0000", 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5899999999999995e-07 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153266+0000", 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8876e-05 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.849 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153274+0000", 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7190000000000001e-06 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153279+0000", 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3990000000000003e-06 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153314+0000", 2024-03-21T14:17:23.850 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5228999999999998e-05 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153394+0000", 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9103999999999993e-05 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.153476+0000", 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2640999999999997e-05 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.154657+0000", 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0011810810000000001 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156224+0000", 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001566685 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.852 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289169+0000", 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13294541200000001 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289237+0000", 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8116999999999999e-05 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289292+0000", 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4629999999999997e-05 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.853 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25232.0:40342 9.f e252/244 9:f7766684:::benchmark_data_smithi003_135814_object40341:head v 252'2588, mlcod=252'2586)", 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:13:35.156741+0000", 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 228.42710025599999, 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13561242800000001, 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.854 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156741+0000", 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156741+0000", 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156741+0000", 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7599999999999997e-07 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.855 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156786+0000", 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4634e-05 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156794+0000", 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8329999999999994e-06 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156798+0000", 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5850000000000001e-06 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156852+0000", 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3155999999999997e-05 2024-03-21T14:17:23.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.858 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.858 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.858 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.156866+0000", 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4034e-05 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.292265+0000", 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13539896900000001 2024-03-21T14:17:23.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.292353+0000", 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8541000000000005e-05 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.860 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.25232.0:40338 9.c e252/244 ondisk, result = 0)", 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:13:35.157646+0000", 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 228.42619539500001, 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.131657781, 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.861 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.157646+0000", 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.157646+0000", 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.157646+0000", 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2200000000000004e-07 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.862 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.157650+0000", 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4070000000000002e-06 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.157660+0000", 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0152e-05 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.157665+0000", 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9559999999999996e-06 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.863 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289151+0000", 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13148663299999999 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289161+0000", 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7350000000000008e-06 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289303+0000", 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014227599999999999 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.864 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25232.0:40344 9.f e252/244 9:fd4401f7:::benchmark_data_smithi003_135814_object40343:head v 252'2589, mlcod=252'2586)", 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:13:35.158914+0000", 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 228.42492671299999, 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13390682000000001, 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.158914+0000", 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.865 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.158914+0000", 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.158915+0000", 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3e-07 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.158976+0000", 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1032000000000001e-05 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.866 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.288967+0000", 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12999148299999999 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.288979+0000", 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1571e-05 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289607+0000", 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062821600000000002 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.867 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.289628+0000", 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1291999999999999e-05 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.292747+0000", 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031183080000000002 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:13:35.292821+0000", 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4288000000000004e-05 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.868 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:220686 9.1 9:84ccbe40:::benchmark_data_smithi003_135814_object79952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:15:47.467780+0000", 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 96.116061295999998, 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13541199300000001, 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 220686 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:17:23.869 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467780+0000", 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467780+0000", 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467781+0000", 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.452e-06 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.870 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467786+0000", 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2100000000000001e-06 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467798+0000", 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1807999999999999e-05 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467805+0000", 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9079999999999998e-06 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467855+0000", 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0420999999999997e-05 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467948+0000", 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2130000000000001e-05 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.468019+0000", 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1608999999999998e-05 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.872 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.470349+0000", 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023292899999999999 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.470695+0000", 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000346526 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.603076+0000", 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13238137 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.873 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.603137+0000", 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0977000000000002e-05 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.603192+0000", 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4292000000000001e-05 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:17:23.874 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:17:23.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:23.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:23.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:23.920 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:17:23.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:24.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:24.082 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:17:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:17:24.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:24.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:24.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:23 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1373448637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:17:24.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 15 56970 56955 154.75 110.75 0.00728753 0.00645754 2024-03-21T14:17:24.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1373448637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:17:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:23 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1373448637' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:17:24.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:24.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:24.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:24.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:24.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:24.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.582 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:24.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:24.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.718 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:24.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:24.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:24.853 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:24.955 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:24.981 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:24.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:24.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:25.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:25.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.128 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:25.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:25.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: pgmap v676: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s; 26192/158976 objects degraded (16.475%); 6/158976 objects misplaced (0.004%) 2024-03-21T14:17:25.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:25.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1373448637' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:17:25.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: osdmap e327: 8 total, 7 up, 7 in 2024-03-21T14:17:25.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:17:25.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 26192/158976 objects degraded (16.475%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[29576]: osdmap e328: 8 total, 7 up, 7 in 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: pgmap v676: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s; 26192/158976 objects degraded (16.475%); 6/158976 objects misplaced (0.004%) 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1373448637' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: osdmap e327: 8 total, 7 up, 7 in 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 26192/158976 objects degraded (16.475%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:25.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:24 smithi003 ceph-mon[40510]: osdmap e328: 8 total, 7 up, 7 in 2024-03-21T14:17:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: pgmap v676: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.31k op/s; 26192/158976 objects degraded (16.475%); 6/158976 objects misplaced (0.004%) 2024-03-21T14:17:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1373448637' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:17:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: osdmap e327: 8 total, 7 up, 7 in 2024-03-21T14:17:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:17:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 26192/158976 objects degraded (16.475%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:25.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:24 smithi118 ceph-mon[33862]: osdmap e328: 8 total, 7 up, 7 in 2024-03-21T14:17:25.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 58778 58762 153.008 112.938 0.00552614 0.00653047 2024-03-21T14:17:25.230 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.271 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:25.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:25.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:25.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:25.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:25.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:25.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:25.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.688 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:25.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:25.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:25.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:25.880 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2024-03-21T14:17:23.908653+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3460414163,"num_objects":52992,"num_object_clones":0,"num_object_copies":158976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26192,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":52992,"num_whiteouts":0,"num_read":1991,"num_read_kb":1760,"num_write":53549,"num_write_kb":3379363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":126,"num_bytes_recovered":432208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55076,"ondisk_log_size":55076,"up":406,"acting":413,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":466,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15085764,"kb_used_data":11142732,"kb_used_omap":10,"kb_used_meta":3942965,"kb_avail":641130300,"statfs":{"total":671965249536,"available":656517427200,"internally_reserved":0,"allocated":11410157568,"data_stored":11405810203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":4037596266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,7,8,10],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1347354624,"num_objects":20559,"num_object_clones":0,"num_object_copies":61677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10248,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20559,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":20559,"num_write_kb":1315776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.370966"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":565,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.619192+0000","last_change":"2024-03-21T14:16:18.479867+0000","last_active":"2024-03-21T14:17:09.619192+0000","last_peered":"2024-03-21T14:17:09.619192+0000","last_clean":"2024-03-21T14:17:09.619192+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:17:09.619192+0000","last_undegraded":"2024-03-21T14:17:09.619192+0000","last_fullsized":"2024-03-21T14:17:09.619192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:18.479698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:18.479698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:20.878647+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618614+0000","last_change":"2024-03-21T14:16:13.706591+0000","last_active":"2024-03-21T14:17:09.618614+0000","last_peered":"2024-03-21T14:17:09.618614+0000","last_clean":"2024-03-21T14:17:09.618614+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:17:09.618614+0000","last_undegraded":"2024-03-21T14:17:09.618614+0000","last_fullsized":"2024-03-21T14:17:09.618614+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:13.706475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.706475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.837131+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":388,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.621296+0000","last_change":"2024-03-21T14:16:21.861900+0000","last_active":"2024-03-21T14:17:09.621296+0000","last_peered":"2024-03-21T14:17:09.621296+0000","last_clean":"2024-03-21T14:14:52.709607+0000","last_became_active":"2024-03-21T14:16:21.861900+0000","last_became_peered":"2024-03-21T14:16:21.861900+0000","last_unstale":"2024-03-21T14:17:09.621296+0000","last_undegraded":"2024-03-21T14:17:09.621296+0000","last_fullsized":"2024-03-21T14:16:21.853133+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:07.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:07.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":610,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.618997+0000","last_change":"2024-03-21T14:14:54.733655+0000","last_active":"2024-03-21T14:17:09.618997+0000","last_peered":"2024-03-21T14:17:09.618997+0000","last_clean":"2024-03-21T14:14:52.715690+0000","last_became_active":"2024-03-21T14:14:54.733655+0000","last_became_peered":"2024-03-21T14:14:54.733655+0000","last_unstale":"2024-03-21T14:17:09.618997+0000","last_undegraded":"2024-03-21T14:14:54.724620+0000","last_fullsized":"2024-03-21T14:14:54.724501+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:12:52.533687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:52.533687+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":439,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.622268+0000","last_change":"2024-03-21T14:14:54.740287+0000","last_active":"2024-03-21T14:17:09.622268+0000","last_peered":"2024-03-21T14:17:09.622268+0000","last_clean":"2024-03-21T14:14:52.711070+0000","last_became_active":"2024-03-21T14:14:54.740287+0000","last_became_peered":"2024-03-21T14:14:54.740287+0000","last_unstale":"2024-03-21T14:17:09.622268+0000","last_undegraded":"2024-03-21T14:17:09.622268+0000","last_fullsized":"2024-03-21T14:14:54.732333+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:49.783238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:49.783238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.613881+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":579,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619355+0000","last_change":"2024-03-21T14:14:54.733806+0000","last_active":"2024-03-21T14:17:09.619355+0000","last_peered":"2024-03-21T14:17:09.619355+0000","last_clean":"2024-03-21T14:14:52.715590+0000","last_became_active":"2024-03-21T14:14:54.733806+0000","last_became_peered":"2024-03-21T14:14:54.733806+0000","last_unstale":"2024-03-21T14:17:09.619355+0000","last_undegraded":"2024-03-21T14:17:09.619355+0000","last_fullsized":"2024-03-21T14:14:54.723805+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.781554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.781554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358636+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.623324+0000","last_change":"2024-03-21T14:14:54.731504+0000","last_active":"2024-03-21T14:17:09.623324+0000","last_peered":"2024-03-21T14:17:09.623324+0000","last_clean":"2024-03-21T14:14:52.721005+0000","last_became_active":"2024-03-21T14:14:54.731504+0000","last_became_peered":"2024-03-21T14:14:54.731504+0000","last_unstale":"2024-03-21T14:17:09.623324+0000","last_undegraded":"2024-03-21T14:17:09.623324+0000","last_fullsized":"2024-03-21T14:14:54.723631+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.147407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.147407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853438+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":627,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.625173+0000","last_change":"2024-03-21T14:14:54.733569+0000","last_active":"2024-03-21T14:17:09.625173+0000","last_peered":"2024-03-21T14:17:09.625173+0000","last_clean":"2024-03-21T14:14:52.725358+0000","last_became_active":"2024-03-21T14:14:54.733569+0000","last_became_peered":"2024-03-21T14:14:54.733569+0000","last_unstale":"2024-03-21T14:17:09.625173+0000","last_undegraded":"2024-03-21T14:14:54.724152+0000","last_fullsized":"2024-03-21T14:14:54.724039+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:12:48.244225+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:12:48.244225+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853472+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":434,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.617584+0000","last_change":"2024-03-21T14:14:54.746386+0000","last_active":"2024-03-21T14:17:09.617584+0000","last_peered":"2024-03-21T14:17:09.617584+0000","last_clean":"2024-03-21T14:14:52.715750+0000","last_became_active":"2024-03-21T14:14:54.746386+0000","last_became_peered":"2024-03-21T14:14:54.746386+0000","last_unstale":"2024-03-21T14:17:09.617584+0000","last_undegraded":"2024-03-21T14:17:09.617584+0000","last_fullsized":"2024-03-21T14:14:54.729653+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:39.510830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:39.510830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.626981+0000","last_change":"2024-03-21T14:16:25.570159+0000","last_active":"2024-03-21T14:17:09.626981+0000","last_peered":"2024-03-21T14:17:09.626981+0000","last_clean":"2024-03-21T14:17:09.626981+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:17:09.626981+0000","last_undegraded":"2024-03-21T14:17:09.626981+0000","last_fullsized":"2024-03-21T14:17:09.626981+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:25.570054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:16:25.570054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:44.999312+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":386,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620934+0000","last_change":"2024-03-21T14:16:21.871252+0000","last_active":"2024-03-21T14:17:09.620934+0000","last_peered":"2024-03-21T14:17:09.620934+0000","last_clean":"2024-03-21T14:17:09.620934+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:17:09.620934+0000","last_undegraded":"2024-03-21T14:17:09.620934+0000","last_fullsized":"2024-03-21T14:17:09.620934+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:57.333382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:14:57.333382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":598,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:21.179040+0000","last_change":"2024-03-21T14:17:21.179086+0000","last_active":"2024-03-21T14:17:21.179040+0000","last_peered":"2024-03-21T14:17:21.179040+0000","last_clean":"2024-03-21T14:17:21.179040+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:17:21.179040+0000","last_undegraded":"2024-03-21T14:17:21.179040+0000","last_fullsized":"2024-03-21T14:17:21.179040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:21.178967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:21.178967+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:39.834009+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":523,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619210+0000","last_change":"2024-03-21T14:16:21.867853+0000","last_active":"2024-03-21T14:17:09.619210+0000","last_peered":"2024-03-21T14:17:09.619210+0000","last_clean":"2024-03-21T14:14:52.705556+0000","last_became_active":"2024-03-21T14:16:21.867853+0000","last_became_peered":"2024-03-21T14:16:21.867853+0000","last_unstale":"2024-03-21T14:17:09.619210+0000","last_undegraded":"2024-03-21T14:17:09.619210+0000","last_fullsized":"2024-03-21T14:16:21.860202+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:41.341673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.341673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901322+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":577,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.622340+0000","last_change":"2024-03-21T14:16:22.034648+0000","last_active":"2024-03-21T14:17:09.622340+0000","last_peered":"2024-03-21T14:17:09.622340+0000","last_clean":"2024-03-21T14:17:09.622340+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:17:09.622340+0000","last_undegraded":"2024-03-21T14:17:09.622340+0000","last_fullsized":"2024-03-21T14:17:09.622340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:22.033938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:22.033938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:26.975170+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":60,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.619453+0000","last_change":"2024-03-21T14:17:09.374978+0000","last_active":"2024-03-21T14:17:09.619453+0000","last_peered":"2024-03-21T14:17:09.619453+0000","last_clean":"2024-03-21T14:17:09.619453+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:17:09.619453+0000","last_undegraded":"2024-03-21T14:17:09.619453+0000","last_fullsized":"2024-03-21T14:17:09.619453+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:09.374855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:09.374855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:18.000827+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620728+0000","last_change":"2024-03-21T14:15:38.173272+0000","last_active":"2024-03-21T14:17:09.620728+0000","last_peered":"2024-03-21T14:17:09.620728+0000","last_clean":"2024-03-21T14:17:09.620728+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:17:09.620728+0000","last_undegraded":"2024-03-21T14:17:09.620728+0000","last_fullsized":"2024-03-21T14:17:09.620728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:15:38.173089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.173089+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":573,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623584+0000","last_change":"2024-03-21T14:17:06.425215+0000","last_active":"2024-03-21T14:17:09.623584+0000","last_peered":"2024-03-21T14:17:09.623584+0000","last_clean":"2024-03-21T14:17:09.623584+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:17:09.623584+0000","last_undegraded":"2024-03-21T14:17:09.623584+0000","last_fullsized":"2024-03-21T14:17:09.623584+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:06.425104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:06.425104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:21.128928+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":577,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618400+0000","last_change":"2024-03-21T14:15:29.462169+0000","last_active":"2024-03-21T14:17:09.618400+0000","last_peered":"2024-03-21T14:17:09.618400+0000","last_clean":"2024-03-21T14:17:09.618400+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:17:09.618400+0000","last_undegraded":"2024-03-21T14:17:09.618400+0000","last_fullsized":"2024-03-21T14:17:09.618400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:29.462055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:29.462055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":430,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620509+0000","last_change":"2024-03-21T14:15:37.108767+0000","last_active":"2024-03-21T14:17:09.620509+0000","last_peered":"2024-03-21T14:17:09.620509+0000","last_clean":"2024-03-21T14:17:09.620509+0000","last_became_active":"2024-03-21T14:07:23.353626+0000","last_became_peered":"2024-03-21T14:07:23.353626+0000","last_unstale":"2024-03-21T14:17:09.620509+0000","last_undegraded":"2024-03-21T14:17:09.620509+0000","last_fullsized":"2024-03-21T14:17:09.620509+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.108651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'27","reported_seq":553,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.621090+0000","last_change":"2024-03-21T14:14:54.733568+0000","last_active":"2024-03-21T14:17:09.621090+0000","last_peered":"2024-03-21T14:17:09.621090+0000","last_clean":"2024-03-21T14:12:42.334846+0000","last_became_active":"2024-03-21T14:14:54.733568+0000","last_became_peered":"2024-03-21T14:14:54.733568+0000","last_unstale":"2024-03-21T14:17:09.621090+0000","last_undegraded":"2024-03-21T14:14:54.727590+0000","last_fullsized":"2024-03-21T14:14:54.727474+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:14:48.508532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:48.508532+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":589,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.624676+0000","last_change":"2024-03-21T14:17:05.017847+0000","last_active":"2024-03-21T14:17:09.624676+0000","last_peered":"2024-03-21T14:17:09.624676+0000","last_clean":"2024-03-21T14:17:09.624676+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:17:09.624676+0000","last_undegraded":"2024-03-21T14:17:09.624676+0000","last_fullsized":"2024-03-21T14:17:09.624676+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.017731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.017731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:13.574877+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.620016+0000","last_change":"2024-03-21T14:14:54.732684+0000","last_active":"2024-03-21T14:17:09.620016+0000","last_peered":"2024-03-21T14:17:09.620016+0000","last_clean":"2024-03-21T14:14:52.716513+0000","last_became_active":"2024-03-21T14:14:54.732684+0000","last_became_peered":"2024-03-21T14:14:54.732684+0000","last_unstale":"2024-03-21T14:17:09.620016+0000","last_undegraded":"2024-03-21T14:17:09.620016+0000","last_fullsized":"2024-03-21T14:14:54.724942+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.679529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.679529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358623+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":572,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619456+0000","last_change":"2024-03-21T14:14:54.738998+0000","last_active":"2024-03-21T14:17:09.619456+0000","last_peered":"2024-03-21T14:17:09.619456+0000","last_clean":"2024-03-21T14:14:52.716365+0000","last_became_active":"2024-03-21T14:14:54.738998+0000","last_became_peered":"2024-03-21T14:14:54.738998+0000","last_unstale":"2024-03-21T14:17:09.619456+0000","last_undegraded":"2024-03-21T14:17:09.619456+0000","last_fullsized":"2024-03-21T14:14:54.726991+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:10:49.866989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:10:49.866989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358520+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":613,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.624134+0000","last_change":"2024-03-21T14:15:26.792016+0000","last_active":"2024-03-21T14:17:09.624134+0000","last_peered":"2024-03-21T14:17:09.624134+0000","last_clean":"2024-03-21T14:17:09.624134+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:17:09.624134+0000","last_undegraded":"2024-03-21T14:17:09.624134+0000","last_fullsized":"2024-03-21T14:17:09.624134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:15:26.791862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:26.791862+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":578,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.624001+0000","last_change":"2024-03-21T14:14:54.743535+0000","last_active":"2024-03-21T14:17:09.624001+0000","last_peered":"2024-03-21T14:17:09.624001+0000","last_clean":"2024-03-21T14:14:52.711655+0000","last_became_active":"2024-03-21T14:14:54.743535+0000","last_became_peered":"2024-03-21T14:14:54.743535+0000","last_unstale":"2024-03-21T14:17:09.624001+0000","last_undegraded":"2024-03-21T14:14:54.725946+0000","last_fullsized":"2024-03-21T14:14:54.725819+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.547775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.613978+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"298'10","reported_seq":126,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:13.664025+0000","last_change":"2024-03-21T14:16:59.343397+0000","last_active":"2024-03-21T14:17:13.664025+0000","last_peered":"2024-03-21T14:17:13.664025+0000","last_clean":"2024-03-21T14:17:13.664025+0000","last_became_active":"2024-03-21T14:16:00.605304+0000","last_became_peered":"2024-03-21T14:16:00.605304+0000","last_unstale":"2024-03-21T14:17:13.664025+0000","last_undegraded":"2024-03-21T14:17:13.664025+0000","last_fullsized":"2024-03-21T14:17:13.664025+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'10","last_scrub_stamp":"2024-03-21T14:16:59.343213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:59.343213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:05.718685+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":603,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.895002+0000","last_change":"2024-03-21T14:17:09.895047+0000","last_active":"2024-03-21T14:17:09.895002+0000","last_peered":"2024-03-21T14:17:09.895002+0000","last_clean":"2024-03-21T14:17:09.895002+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:17:09.895002+0000","last_undegraded":"2024-03-21T14:17:09.895002+0000","last_fullsized":"2024-03-21T14:17:09.895002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:17:09.894908+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:17:09.894908+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:18.502723+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":275,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.624569+0000","last_change":"2024-03-21T14:14:54.738663+0000","last_active":"2024-03-21T14:17:09.624569+0000","last_peered":"2024-03-21T14:17:09.624569+0000","last_clean":"2024-03-21T14:14:52.711713+0000","last_became_active":"2024-03-21T14:14:54.738663+0000","last_became_peered":"2024-03-21T14:14:54.738663+0000","last_unstale":"2024-03-21T14:17:09.624569+0000","last_undegraded":"2024-03-21T14:14:54.726825+0000","last_fullsized":"2024-03-21T14:14:54.726705+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:13:41.601807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:41.601807+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.613932+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":573,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618018+0000","last_change":"2024-03-21T14:15:27.529054+0000","last_active":"2024-03-21T14:17:09.618018+0000","last_peered":"2024-03-21T14:17:09.618018+0000","last_clean":"2024-03-21T14:17:09.618018+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:17:09.618018+0000","last_undegraded":"2024-03-21T14:17:09.618018+0000","last_fullsized":"2024-03-21T14:17:09.618018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:27.528922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:27.528922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":227,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.623532+0000","last_change":"2024-03-21T14:14:54.739116+0000","last_active":"2024-03-21T14:17:09.623532+0000","last_peered":"2024-03-21T14:17:09.623532+0000","last_clean":"2024-03-21T14:14:52.711613+0000","last_became_active":"2024-03-21T14:14:54.739116+0000","last_became_peered":"2024-03-21T14:14:54.739116+0000","last_unstale":"2024-03-21T14:17:09.623532+0000","last_undegraded":"2024-03-21T14:17:09.623532+0000","last_fullsized":"2024-03-21T14:14:54.732137+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:59.819350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:12:59.819350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.613864+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":595,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.624564+0000","last_change":"2024-03-21T14:14:54.733221+0000","last_active":"2024-03-21T14:17:09.624564+0000","last_peered":"2024-03-21T14:17:09.624564+0000","last_clean":"2024-03-21T14:14:52.721722+0000","last_became_active":"2024-03-21T14:14:54.733221+0000","last_became_peered":"2024-03-21T14:14:54.733221+0000","last_unstale":"2024-03-21T14:17:09.624564+0000","last_undegraded":"2024-03-21T14:17:09.624564+0000","last_fullsized":"2024-03-21T14:14:54.723133+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:59.621530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.621530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853516+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":317,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:16.961611+0000","last_change":"2024-03-21T14:17:16.961647+0000","last_active":"2024-03-21T14:17:16.961611+0000","last_peered":"2024-03-21T14:17:16.961611+0000","last_clean":"2024-03-21T14:17:16.961611+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:17:16.961611+0000","last_undegraded":"2024-03-21T14:17:16.961611+0000","last_fullsized":"2024-03-21T14:17:16.961611+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:17:16.961547+0000","last_deep_scrub":"127'16","last_deep_scrub_stamp":"2024-03-21T14:07:18.966215+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.961547+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:44.605226+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":514,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619018+0000","last_change":"2024-03-21T14:16:34.611932+0000","last_active":"2024-03-21T14:17:09.619018+0000","last_peered":"2024-03-21T14:17:09.619018+0000","last_clean":"2024-03-21T14:14:52.711440+0000","last_became_active":"2024-03-21T14:16:34.611932+0000","last_became_peered":"2024-03-21T14:16:34.611932+0000","last_unstale":"2024-03-21T14:17:09.619018+0000","last_undegraded":"2024-03-21T14:17:09.619018+0000","last_fullsized":"2024-03-21T14:16:34.605066+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:49.859204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:49.859204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:11.358498+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":260,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.619472+0000","last_change":"2024-03-21T14:16:19.440863+0000","last_active":"2024-03-21T14:17:09.619472+0000","last_peered":"2024-03-21T14:17:09.619472+0000","last_clean":"2024-03-21T14:17:09.619472+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:17:09.619472+0000","last_undegraded":"2024-03-21T14:17:09.619472+0000","last_fullsized":"2024-03-21T14:17:09.619472+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:19.440733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:19.440733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:21.533807+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.625008+0000","last_change":"2024-03-21T14:16:56.847010+0000","last_active":"2024-03-21T14:17:09.625008+0000","last_peered":"2024-03-21T14:17:09.625008+0000","last_clean":"2024-03-21T14:17:09.625008+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:17:09.625008+0000","last_undegraded":"2024-03-21T14:17:09.625008+0000","last_fullsized":"2024-03-21T14:17:09.625008+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:56.846916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:56.846916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:04.434680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"275'10","reported_seq":616,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.625382+0000","last_change":"2024-03-21T14:16:19.562633+0000","last_active":"2024-03-21T14:17:09.625382+0000","last_peered":"2024-03-21T14:17:09.625382+0000","last_clean":"2024-03-21T14:17:09.625382+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:17:09.625382+0000","last_undegraded":"2024-03-21T14:17:09.625382+0000","last_fullsized":"2024-03-21T14:17:09.625382+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'10","last_scrub_stamp":"2024-03-21T14:16:19.562507+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:16:19.562507+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:25.547387+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":600,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.625485+0000","last_change":"2024-03-21T14:17:07.939309+0000","last_active":"2024-03-21T14:17:09.625485+0000","last_peered":"2024-03-21T14:17:09.625485+0000","last_clean":"2024-03-21T14:17:09.625485+0000","last_became_active":"2024-03-21T14:12:00.536387+0000","last_became_peered":"2024-03-21T14:12:00.536387+0000","last_unstale":"2024-03-21T14:17:09.625485+0000","last_undegraded":"2024-03-21T14:17:09.625485+0000","last_fullsized":"2024-03-21T14:17:09.625485+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:07.939188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:17:07.939188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:34.480084+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":600,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:20.623310+0000","last_change":"2024-03-21T14:17:20.623349+0000","last_active":"2024-03-21T14:17:20.623310+0000","last_peered":"2024-03-21T14:17:20.623310+0000","last_clean":"2024-03-21T14:17:20.623310+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:17:20.623310+0000","last_undegraded":"2024-03-21T14:17:20.623310+0000","last_fullsized":"2024-03-21T14:17:20.623310+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:20.623230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:20.623230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:23.216703+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":56,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618028+0000","last_change":"2024-03-21T14:17:04.906653+0000","last_active":"2024-03-21T14:17:09.618028+0000","last_peered":"2024-03-21T14:17:09.618028+0000","last_clean":"2024-03-21T14:17:09.618028+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:17:09.618028+0000","last_undegraded":"2024-03-21T14:17:09.618028+0000","last_fullsized":"2024-03-21T14:17:09.618028+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:17:04.906566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:04.906566+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:07.800918+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":644,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.624923+0000","last_change":"2024-03-21T14:15:38.609735+0000","last_active":"2024-03-21T14:17:09.624923+0000","last_peered":"2024-03-21T14:17:09.624923+0000","last_clean":"2024-03-21T14:17:09.624923+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:17:09.624923+0000","last_undegraded":"2024-03-21T14:17:09.624923+0000","last_fullsized":"2024-03-21T14:17:09.624923+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:15:38.609594+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.609594+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":62,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:10.885273+0000","last_change":"2024-03-21T14:17:10.885316+0000","last_active":"2024-03-21T14:17:10.885273+0000","last_peered":"2024-03-21T14:17:10.885273+0000","last_clean":"2024-03-21T14:17:10.885273+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:17:10.885273+0000","last_undegraded":"2024-03-21T14:17:10.885273+0000","last_fullsized":"2024-03-21T14:17:10.885273+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:10.885070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:10.885070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:13.087418+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:19.983585+0000","last_change":"2024-03-21T14:17:19.983621+0000","last_active":"2024-03-21T14:17:19.983585+0000","last_peered":"2024-03-21T14:17:19.983585+0000","last_clean":"2024-03-21T14:17:19.983585+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:17:19.983585+0000","last_undegraded":"2024-03-21T14:17:19.983585+0000","last_fullsized":"2024-03-21T14:17:19.983585+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:19.983497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:17:19.983497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:35.792093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623057+0000","last_change":"2024-03-21T14:17:05.617992+0000","last_active":"2024-03-21T14:17:09.623057+0000","last_peered":"2024-03-21T14:17:09.623057+0000","last_clean":"2024-03-21T14:17:09.623057+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:17:09.623057+0000","last_undegraded":"2024-03-21T14:17:09.623057+0000","last_fullsized":"2024-03-21T14:17:09.623057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.617865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.617865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:09.660691+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":626,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.623860+0000","last_change":"2024-03-21T14:14:54.733024+0000","last_active":"2024-03-21T14:17:09.623860+0000","last_peered":"2024-03-21T14:17:09.623860+0000","last_clean":"2024-03-21T14:14:52.723132+0000","last_became_active":"2024-03-21T14:14:54.733024+0000","last_became_peered":"2024-03-21T14:14:54.733024+0000","last_unstale":"2024-03-21T14:17:09.623860+0000","last_undegraded":"2024-03-21T14:14:54.728446+0000","last_fullsized":"2024-03-21T14:14:54.727848+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.492830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.492830+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901288+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":264,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.621025+0000","last_change":"2024-03-21T14:16:21.862369+0000","last_active":"2024-03-21T14:17:09.621025+0000","last_peered":"2024-03-21T14:17:09.621025+0000","last_clean":"2024-03-21T14:14:52.705631+0000","last_became_active":"2024-03-21T14:16:21.862369+0000","last_became_peered":"2024-03-21T14:16:21.862369+0000","last_unstale":"2024-03-21T14:17:09.621025+0000","last_undegraded":"2024-03-21T14:17:09.621025+0000","last_fullsized":"2024-03-21T14:16:21.852836+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:01.256711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:01.256711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.020083+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":592,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623663+0000","last_change":"2024-03-21T14:15:36.366086+0000","last_active":"2024-03-21T14:17:09.623663+0000","last_peered":"2024-03-21T14:17:09.623663+0000","last_clean":"2024-03-21T14:17:09.623663+0000","last_became_active":"2024-03-21T14:07:23.359571+0000","last_became_peered":"2024-03-21T14:07:23.359571+0000","last_unstale":"2024-03-21T14:17:09.623663+0000","last_undegraded":"2024-03-21T14:17:09.623663+0000","last_fullsized":"2024-03-21T14:17:09.623663+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:36.365963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:36.365963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":595,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619612+0000","last_change":"2024-03-21T14:14:54.734170+0000","last_active":"2024-03-21T14:17:09.619612+0000","last_peered":"2024-03-21T14:17:09.619612+0000","last_clean":"2024-03-21T14:14:52.716386+0000","last_became_active":"2024-03-21T14:14:54.734170+0000","last_became_peered":"2024-03-21T14:14:54.734170+0000","last_unstale":"2024-03-21T14:17:09.619612+0000","last_undegraded":"2024-03-21T14:17:09.619612+0000","last_fullsized":"2024-03-21T14:14:54.727989+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:47.020235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:47.020235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":648,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:09.620630+0000","last_change":"2024-03-21T14:16:13.416665+0000","last_active":"2024-03-21T14:17:09.620630+0000","last_peered":"2024-03-21T14:17:09.620630+0000","last_clean":"2024-03-21T14:17:09.620630+0000","last_became_active":"2024-03-21T14:16:00.609375+0000","last_became_peered":"2024-03-21T14:16:00.609375+0000","last_unstale":"2024-03-21T14:17:09.620630+0000","last_undegraded":"2024-03-21T14:17:09.620630+0000","last_fullsized":"2024-03-21T14:17:09.620630+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:16:13.416403+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.416403+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:31.227020+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[{"shards":"4,5,6","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":439,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.624467+0000","last_change":"2024-03-21T14:16:59.705337+0000","last_active":"2024-03-21T14:17:09.624467+0000","last_peered":"2024-03-21T14:17:09.624467+0000","last_clean":"2024-03-21T14:17:09.624467+0000","last_became_active":"2024-03-21T14:07:23.357222+0000","last_became_peered":"2024-03-21T14:07:23.357222+0000","last_unstale":"2024-03-21T14:17:09.624467+0000","last_undegraded":"2024-03-21T14:17:09.624467+0000","last_fullsized":"2024-03-21T14:17:09.624467+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:59.705219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:59.705219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:08.017501+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":737,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.663791+0000","last_change":"2024-03-21T14:16:21.870976+0000","last_active":"2024-03-21T14:17:13.663791+0000","last_peered":"2024-03-21T14:17:13.663791+0000","last_clean":"2024-03-21T14:17:13.663791+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:17:13.663791+0000","last_undegraded":"2024-03-21T14:17:13.663791+0000","last_fullsized":"2024-03-21T14:17:13.663791+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'9","last_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.540505+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":590,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.622442+0000","last_change":"2024-03-21T14:14:54.736534+0000","last_active":"2024-03-21T14:17:09.622442+0000","last_peered":"2024-03-21T14:17:09.622442+0000","last_clean":"2024-03-21T14:14:52.711154+0000","last_became_active":"2024-03-21T14:14:54.736534+0000","last_became_peered":"2024-03-21T14:14:54.736534+0000","last_unstale":"2024-03-21T14:17:09.622442+0000","last_undegraded":"2024-03-21T14:17:09.622442+0000","last_fullsized":"2024-03-21T14:14:54.732339+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:34.724192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:13:34.724192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.613947+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":617,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623581+0000","last_change":"2024-03-21T14:16:12.543045+0000","last_active":"2024-03-21T14:17:09.623581+0000","last_peered":"2024-03-21T14:17:09.623581+0000","last_clean":"2024-03-21T14:17:09.623581+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:17:09.623581+0000","last_undegraded":"2024-03-21T14:17:09.623581+0000","last_fullsized":"2024-03-21T14:17:09.623581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:16:12.542909+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:16:12.542909+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:36.540422+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":561,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.621580+0000","last_change":"2024-03-21T14:14:54.741413+0000","last_active":"2024-03-21T14:17:09.621580+0000","last_peered":"2024-03-21T14:17:09.621580+0000","last_clean":"2024-03-21T14:14:52.716802+0000","last_became_active":"2024-03-21T14:14:54.741413+0000","last_became_peered":"2024-03-21T14:14:54.741413+0000","last_unstale":"2024-03-21T14:17:09.621580+0000","last_undegraded":"2024-03-21T14:17:09.621580+0000","last_fullsized":"2024-03-21T14:14:54.729363+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.642926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358553+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"312'11","reported_seq":73,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:13.663623+0000","last_change":"2024-03-21T14:16:34.613524+0000","last_active":"2024-03-21T14:17:13.663623+0000","last_peered":"2024-03-21T14:17:13.663623+0000","last_clean":"2024-03-21T14:02:40.635115+0000","last_became_active":"2024-03-21T14:16:34.613524+0000","last_became_peered":"2024-03-21T14:16:34.613524+0000","last_unstale":"2024-03-21T14:17:13.663623+0000","last_undegraded":"2024-03-21T14:16:34.608081+0000","last_fullsized":"2024-03-21T14:16:34.607957+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'9","last_scrub_stamp":"2024-03-21T14:13:03.638324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:03.638324+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:21.953464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":198,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.617783+0000","last_change":"2024-03-21T14:16:34.620542+0000","last_active":"2024-03-21T14:17:09.617783+0000","last_peered":"2024-03-21T14:17:09.617783+0000","last_clean":"2024-03-21T14:17:09.617783+0000","last_became_active":"2024-03-21T14:16:34.620151+0000","last_became_peered":"2024-03-21T14:16:34.620151+0000","last_unstale":"2024-03-21T14:17:09.617783+0000","last_undegraded":"2024-03-21T14:17:09.617783+0000","last_fullsized":"2024-03-21T14:17:09.617783+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:38.968899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:38.968899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.15","version":"236'15","reported_seq":458,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.624100+0000","last_change":"2024-03-21T14:16:21.889160+0000","last_active":"2024-03-21T14:17:09.624100+0000","last_peered":"2024-03-21T14:17:09.624100+0000","last_clean":"2024-03-21T14:17:09.624100+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:17:09.624100+0000","last_undegraded":"2024-03-21T14:17:09.624100+0000","last_fullsized":"2024-03-21T14:17:09.624100+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:16:08.564278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:16:08.564278+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:23.996276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":104,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.620777+0000","last_change":"2024-03-21T14:14:54.738499+0000","last_active":"2024-03-21T14:17:09.620777+0000","last_peered":"2024-03-21T14:17:09.620777+0000","last_clean":"2024-03-21T14:02:38.441204+0000","last_became_active":"2024-03-21T14:14:54.738499+0000","last_became_peered":"2024-03-21T14:14:54.738499+0000","last_unstale":"2024-03-21T14:17:09.620777+0000","last_undegraded":"2024-03-21T14:17:09.620777+0000","last_fullsized":"2024-03-21T14:14:54.731067+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:43.502435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:43.502435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":439,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620579+0000","last_change":"2024-03-21T14:16:57.987097+0000","last_active":"2024-03-21T14:17:09.620579+0000","last_peered":"2024-03-21T14:17:09.620579+0000","last_clean":"2024-03-21T14:17:09.620579+0000","last_became_active":"2024-03-21T14:07:23.372444+0000","last_became_peered":"2024-03-21T14:07:23.372444+0000","last_unstale":"2024-03-21T14:17:09.620579+0000","last_undegraded":"2024-03-21T14:17:09.620579+0000","last_fullsized":"2024-03-21T14:17:09.620579+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.986983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.986983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:10.230755+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":573,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620136+0000","last_change":"2024-03-21T14:17:05.374072+0000","last_active":"2024-03-21T14:17:09.620136+0000","last_peered":"2024-03-21T14:17:09.620136+0000","last_clean":"2024-03-21T14:17:09.620136+0000","last_became_active":"2024-03-21T14:16:34.616172+0000","last_became_peered":"2024-03-21T14:16:34.616172+0000","last_unstale":"2024-03-21T14:17:09.620136+0000","last_undegraded":"2024-03-21T14:17:09.620136+0000","last_fullsized":"2024-03-21T14:17:09.620136+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.373895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.373895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:35.036037+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":283,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:16.885801+0000","last_change":"2024-03-21T14:17:16.885851+0000","last_active":"2024-03-21T14:17:16.885801+0000","last_peered":"2024-03-21T14:17:16.885801+0000","last_clean":"2024-03-21T14:17:16.885801+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:17:16.885801+0000","last_undegraded":"2024-03-21T14:17:16.885801+0000","last_fullsized":"2024-03-21T14:17:16.885801+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:16.885709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.885709+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:46.661386+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.585453+0000","last_change":"2024-03-21T14:17:13.585502+0000","last_active":"2024-03-21T14:17:13.585453+0000","last_peered":"2024-03-21T14:17:13.585453+0000","last_clean":"2024-03-21T14:17:13.585453+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:17:13.585453+0000","last_undegraded":"2024-03-21T14:17:13.585453+0000","last_fullsized":"2024-03-21T14:17:13.585453+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:13.585214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:13.585214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.246734+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":60,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623451+0000","last_change":"2024-03-21T14:17:03.781272+0000","last_active":"2024-03-21T14:17:09.623451+0000","last_peered":"2024-03-21T14:17:09.623451+0000","last_clean":"2024-03-21T14:17:09.623451+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:17:09.623451+0000","last_undegraded":"2024-03-21T14:17:09.623451+0000","last_fullsized":"2024-03-21T14:17:09.623451+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:03.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.517464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":216,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:09.623052+0000","last_change":"2024-03-21T14:16:22.893949+0000","last_active":"2024-03-21T14:17:09.623052+0000","last_peered":"2024-03-21T14:17:09.623052+0000","last_clean":"2024-03-21T14:17:09.623052+0000","last_became_active":"2024-03-21T14:16:22.893331+0000","last_became_peered":"2024-03-21T14:16:22.893331+0000","last_unstale":"2024-03-21T14:17:09.623052+0000","last_undegraded":"2024-03-21T14:17:09.623052+0000","last_fullsized":"2024-03-21T14:17:09.623052+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:28.983112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:28.983112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,6],"avail_no_missing":["3","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":322,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.620748+0000","last_change":"2024-03-21T14:16:34.615269+0000","last_active":"2024-03-21T14:17:09.620748+0000","last_peered":"2024-03-21T14:17:09.620748+0000","last_clean":"2024-03-21T14:12:42.522855+0000","last_became_active":"2024-03-21T14:16:34.615269+0000","last_became_peered":"2024-03-21T14:16:34.615269+0000","last_unstale":"2024-03-21T14:17:09.620748+0000","last_undegraded":"2024-03-21T14:16:34.611042+0000","last_fullsized":"2024-03-21T14:16:34.610902+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:13:59.567870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:59.567870+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901354+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"326'3328","reported_seq":3356,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:21.498897+0000","last_change":"2024-03-21T14:16:51.057507+0000","last_active":"2024-03-21T14:17:21.498897+0000","last_peered":"2024-03-21T14:17:21.498897+0000","last_clean":"2024-03-21T14:17:21.498897+0000","last_became_active":"2024-03-21T14:16:51.057016+0000","last_became_peered":"2024-03-21T14:16:51.057016+0000","last_unstale":"2024-03-21T14:17:21.498897+0000","last_undegraded":"2024-03-21T14:17:21.498897+0000","last_fullsized":"2024-03-21T14:17:21.498897+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3328,"log_dups_size":0,"ondisk_log_size":3328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:18.774227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3328,"num_write_kb":212992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":459,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.617615+0000","last_change":"2024-03-21T14:16:21.869140+0000","last_active":"2024-03-21T14:17:09.617615+0000","last_peered":"2024-03-21T14:17:09.617615+0000","last_clean":"2024-03-21T14:17:09.617615+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:17:09.617615+0000","last_undegraded":"2024-03-21T14:17:09.617615+0000","last_fullsized":"2024-03-21T14:17:09.617615+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:14.598756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:16:14.598756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:26.510077+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"65'2","reported_seq":618,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623814+0000","last_change":"2024-03-21T14:17:08.612847+0000","last_active":"2024-03-21T14:17:09.623814+0000","last_peered":"2024-03-21T14:17:09.623814+0000","last_clean":"2024-03-21T14:17:09.623814+0000","last_became_active":"2024-03-21T14:07:23.357606+0000","last_became_peered":"2024-03-21T14:07:23.357606+0000","last_unstale":"2024-03-21T14:17:09.623814+0000","last_undegraded":"2024-03-21T14:17:09.623814+0000","last_fullsized":"2024-03-21T14:17:09.623814+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'2","last_scrub_stamp":"2024-03-21T14:17:08.612734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:08.612734+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:34.333021+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":506,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.622828+0000","last_change":"2024-03-21T14:17:02.712012+0000","last_active":"2024-03-21T14:17:09.622828+0000","last_peered":"2024-03-21T14:17:09.622828+0000","last_clean":"2024-03-21T14:17:09.622828+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:17:09.622828+0000","last_undegraded":"2024-03-21T14:17:09.622828+0000","last_fullsized":"2024-03-21T14:17:09.622828+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:17:02.711916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:02.711916+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:19.694933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"214'9","reported_seq":774,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.664102+0000","last_change":"2024-03-21T14:16:14.902618+0000","last_active":"2024-03-21T14:17:13.664102+0000","last_peered":"2024-03-21T14:17:13.664102+0000","last_clean":"2024-03-21T14:17:13.664102+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:17:13.664102+0000","last_undegraded":"2024-03-21T14:17:13.664102+0000","last_fullsized":"2024-03-21T14:17:13.664102+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"214'9","last_scrub_stamp":"2024-03-21T14:16:14.902143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:14.902143+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:39.064216+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"326'3298","reported_seq":3323,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:21.493552+0000","last_change":"2024-03-21T14:17:01.220613+0000","last_active":"2024-03-21T14:17:21.493552+0000","last_peered":"2024-03-21T14:17:21.493552+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.055518+0000","last_became_peered":"2024-03-21T14:16:51.055518+0000","last_unstale":"2024-03-21T14:17:21.493552+0000","last_undegraded":"2024-03-21T14:17:01.207316+0000","last_fullsized":"2024-03-21T14:16:51.047099+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3298,"log_dups_size":0,"ondisk_log_size":3298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:05.018647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216137728,"num_objects":3298,"num_object_clones":0,"num_object_copies":9894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3298,"num_write_kb":211072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":3298}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":316,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.625116+0000","last_change":"2024-03-21T14:15:40.753578+0000","last_active":"2024-03-21T14:17:09.625116+0000","last_peered":"2024-03-21T14:17:09.625116+0000","last_clean":"2024-03-21T14:17:09.625116+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:17:09.625116+0000","last_undegraded":"2024-03-21T14:17:09.625116+0000","last_fullsized":"2024-03-21T14:17:09.625116+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:40.753276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:40.753276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":602,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:14.552380+0000","last_change":"2024-03-21T14:17:14.552416+0000","last_active":"2024-03-21T14:17:14.552380+0000","last_peered":"2024-03-21T14:17:14.552380+0000","last_clean":"2024-03-21T14:17:14.552380+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:17:14.552380+0000","last_undegraded":"2024-03-21T14:17:14.552380+0000","last_fullsized":"2024-03-21T14:17:14.552380+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:14.552299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:14.552299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:25.439994+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":642,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620477+0000","last_change":"2024-03-21T14:15:43.183428+0000","last_active":"2024-03-21T14:17:09.620477+0000","last_peered":"2024-03-21T14:17:09.620477+0000","last_clean":"2024-03-21T14:17:09.620477+0000","last_became_active":"2024-03-21T14:12:00.540768+0000","last_became_peered":"2024-03-21T14:12:00.540768+0000","last_unstale":"2024-03-21T14:17:09.620477+0000","last_undegraded":"2024-03-21T14:17:09.620477+0000","last_fullsized":"2024-03-21T14:17:09.620477+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:15:43.183132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:43.183132+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":104,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.617786+0000","last_change":"2024-03-21T14:14:54.745296+0000","last_active":"2024-03-21T14:17:09.617786+0000","last_peered":"2024-03-21T14:17:09.617786+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:14:54.745296+0000","last_became_peered":"2024-03-21T14:14:54.745296+0000","last_unstale":"2024-03-21T14:17:09.617786+0000","last_undegraded":"2024-03-21T14:17:09.617786+0000","last_fullsized":"2024-03-21T14:14:54.730905+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:39.425147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:14:39.425147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.4","version":"326'3228","reported_seq":3256,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:21.473427+0000","last_change":"2024-03-21T14:16:51.060563+0000","last_active":"2024-03-21T14:17:21.473427+0000","last_peered":"2024-03-21T14:17:21.473427+0000","last_clean":"2024-03-21T14:17:21.473427+0000","last_became_active":"2024-03-21T14:16:51.060300+0000","last_became_peered":"2024-03-21T14:16:51.060300+0000","last_unstale":"2024-03-21T14:17:21.473427+0000","last_undegraded":"2024-03-21T14:17:21.473427+0000","last_fullsized":"2024-03-21T14:17:21.473427+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3228,"log_dups_size":0,"ondisk_log_size":3228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:55.863045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211550208,"num_objects":3228,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":206592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":505,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.623316+0000","last_change":"2024-03-21T14:16:34.615853+0000","last_active":"2024-03-21T14:17:09.623316+0000","last_peered":"2024-03-21T14:17:09.623316+0000","last_clean":"2024-03-21T14:14:52.709870+0000","last_became_active":"2024-03-21T14:16:34.615853+0000","last_became_peered":"2024-03-21T14:16:34.615853+0000","last_unstale":"2024-03-21T14:17:09.623316+0000","last_undegraded":"2024-03-21T14:17:09.623316+0000","last_fullsized":"2024-03-21T14:16:34.611199+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:54.779479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:12:54.779479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":51,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.622720+0000","last_change":"2024-03-21T14:16:21.862254+0000","last_active":"2024-03-21T14:17:09.622720+0000","last_peered":"2024-03-21T14:17:09.622720+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:16:21.862254+0000","last_became_peered":"2024-03-21T14:16:21.862254+0000","last_unstale":"2024-03-21T14:17:09.622720+0000","last_undegraded":"2024-03-21T14:17:09.622720+0000","last_fullsized":"2024-03-21T14:16:21.852757+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:43.154506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.154506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.020097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":564,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.624353+0000","last_change":"2024-03-21T14:16:34.619006+0000","last_active":"2024-03-21T14:17:09.624353+0000","last_peered":"2024-03-21T14:17:09.624353+0000","last_clean":"2024-03-21T14:17:09.624353+0000","last_became_active":"2024-03-21T14:16:34.618652+0000","last_became_peered":"2024-03-21T14:16:34.618652+0000","last_unstale":"2024-03-21T14:17:09.624353+0000","last_undegraded":"2024-03-21T14:17:09.624353+0000","last_fullsized":"2024-03-21T14:17:09.624353+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:25.963838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:25.963838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":535,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618075+0000","last_change":"2024-03-21T14:15:34.404267+0000","last_active":"2024-03-21T14:17:09.618075+0000","last_peered":"2024-03-21T14:17:09.618075+0000","last_clean":"2024-03-21T14:17:09.618075+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:17:09.618075+0000","last_undegraded":"2024-03-21T14:17:09.618075+0000","last_fullsized":"2024-03-21T14:17:09.618075+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:15:34.404114+0000","last_deep_scrub":"151'28","last_deep_scrub_stamp":"2024-03-21T14:11:10.962498+0000","last_clean_scrub_stamp":"2024-03-21T14:15:34.404114+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"326'3412","reported_seq":3437,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:22.178227+0000","last_change":"2024-03-21T14:17:01.213763+0000","last_active":"2024-03-21T14:17:22.178227+0000","last_peered":"2024-03-21T14:17:22.178227+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.048501+0000","last_became_peered":"2024-03-21T14:16:51.048501+0000","last_unstale":"2024-03-21T14:17:22.178227+0000","last_undegraded":"2024-03-21T14:17:01.209817+0000","last_fullsized":"2024-03-21T14:16:51.043684+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3412,"log_dups_size":0,"ondisk_log_size":3412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:10.283522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223608832,"num_objects":3412,"num_object_clones":0,"num_object_copies":10236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3412,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3412,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3412,"num_write_kb":218368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":3412}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":370,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.623337+0000","last_change":"2024-03-21T14:16:34.617543+0000","last_active":"2024-03-21T14:17:09.623337+0000","last_peered":"2024-03-21T14:17:09.623337+0000","last_clean":"2024-03-21T14:14:52.711889+0000","last_became_active":"2024-03-21T14:16:34.617543+0000","last_became_peered":"2024-03-21T14:16:34.617543+0000","last_unstale":"2024-03-21T14:17:09.623337+0000","last_undegraded":"2024-03-21T14:17:09.623337+0000","last_fullsized":"2024-03-21T14:16:34.611916+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.865488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.865488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853545+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":104,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619758+0000","last_change":"2024-03-21T14:14:54.731341+0000","last_active":"2024-03-21T14:17:09.619758+0000","last_peered":"2024-03-21T14:17:09.619758+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:14:54.731341+0000","last_became_peered":"2024-03-21T14:14:54.731341+0000","last_unstale":"2024-03-21T14:17:09.619758+0000","last_undegraded":"2024-03-21T14:17:09.619758+0000","last_fullsized":"2024-03-21T14:14:54.724706+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.532653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.532653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":386,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623498+0000","last_change":"2024-03-21T14:16:21.865674+0000","last_active":"2024-03-21T14:17:09.623498+0000","last_peered":"2024-03-21T14:17:09.623498+0000","last_clean":"2024-03-21T14:17:09.623498+0000","last_became_active":"2024-03-21T14:16:21.865349+0000","last_became_peered":"2024-03-21T14:16:21.865349+0000","last_unstale":"2024-03-21T14:17:09.623498+0000","last_undegraded":"2024-03-21T14:17:09.623498+0000","last_fullsized":"2024-03-21T14:17:09.623498+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:34.477257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:34.477257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":185,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.620032+0000","last_change":"2024-03-21T14:14:54.734284+0000","last_active":"2024-03-21T14:17:09.620032+0000","last_peered":"2024-03-21T14:17:09.620032+0000","last_clean":"2024-03-21T14:12:42.578861+0000","last_became_active":"2024-03-21T14:14:54.734284+0000","last_became_peered":"2024-03-21T14:14:54.734284+0000","last_unstale":"2024-03-21T14:17:09.620032+0000","last_undegraded":"2024-03-21T14:14:54.726445+0000","last_fullsized":"2024-03-21T14:14:54.726319+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.159386+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.a","version":"326'3146","reported_seq":3171,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:21.794695+0000","last_change":"2024-03-21T14:17:01.214830+0000","last_active":"2024-03-21T14:17:21.794695+0000","last_peered":"2024-03-21T14:17:21.794695+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.052354+0000","last_became_peered":"2024-03-21T14:16:51.052354+0000","last_unstale":"2024-03-21T14:17:21.794695+0000","last_undegraded":"2024-03-21T14:17:01.206891+0000","last_fullsized":"2024-03-21T14:16:51.046025+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3146,"log_dups_size":0,"ondisk_log_size":3146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:51.001045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206176256,"num_objects":3146,"num_object_clones":0,"num_object_copies":9438,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3146,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3146,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3146,"num_write_kb":201344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":3146}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":644,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.626312+0000","last_change":"2024-03-21T14:14:54.737456+0000","last_active":"2024-03-21T14:17:09.626312+0000","last_peered":"2024-03-21T14:17:09.626312+0000","last_clean":"2024-03-21T14:14:52.705212+0000","last_became_active":"2024-03-21T14:14:54.737456+0000","last_became_peered":"2024-03-21T14:14:54.737456+0000","last_unstale":"2024-03-21T14:17:09.626312+0000","last_undegraded":"2024-03-21T14:17:09.626312+0000","last_fullsized":"2024-03-21T14:14:54.720914+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:38.263320+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:38.263320+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":["5","0"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623653+0000","last_change":"2024-03-21T14:16:10.542371+0000","last_active":"2024-03-21T14:17:09.623653+0000","last_peered":"2024-03-21T14:17:09.623653+0000","last_clean":"2024-03-21T14:17:09.623653+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:17:09.623653+0000","last_undegraded":"2024-03-21T14:17:09.623653+0000","last_fullsized":"2024-03-21T14:17:09.623653+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:10.542193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:10.542193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:32.264925+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":86,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.617887+0000","last_change":"2024-03-21T14:16:23.920822+0000","last_active":"2024-03-21T14:17:09.617887+0000","last_peered":"2024-03-21T14:17:09.617887+0000","last_clean":"2024-03-21T14:17:09.617887+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:17:09.617887+0000","last_undegraded":"2024-03-21T14:17:09.617887+0000","last_fullsized":"2024-03-21T14:17:09.617887+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:16:23.920695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:16:23.920695+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:33.452138+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":585,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620456+0000","last_change":"2024-03-21T14:16:57.382377+0000","last_active":"2024-03-21T14:17:09.620456+0000","last_peered":"2024-03-21T14:17:09.620456+0000","last_clean":"2024-03-21T14:17:09.620456+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:17:09.620456+0000","last_undegraded":"2024-03-21T14:17:09.620456+0000","last_fullsized":"2024-03-21T14:17:09.620456+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.382291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.382291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:25.063945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"307'53","reported_seq":594,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:13.663652+0000","last_change":"2024-03-21T14:16:21.860558+0000","last_active":"2024-03-21T14:17:13.663652+0000","last_peered":"2024-03-21T14:17:13.663652+0000","last_clean":"2024-03-21T14:12:42.663015+0000","last_became_active":"2024-03-21T14:16:21.860558+0000","last_became_peered":"2024-03-21T14:16:21.860558+0000","last_unstale":"2024-03-21T14:17:13.663652+0000","last_undegraded":"2024-03-21T14:16:21.858369+0000","last_fullsized":"2024-03-21T14:16:21.858165+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'51","last_scrub_stamp":"2024-03-21T14:12:51.290920+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:12:51.290920+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:21.878888+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":10}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":446,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618504+0000","last_change":"2024-03-21T14:15:31.430590+0000","last_active":"2024-03-21T14:17:09.618504+0000","last_peered":"2024-03-21T14:17:09.618504+0000","last_clean":"2024-03-21T14:17:09.618504+0000","last_became_active":"2024-03-21T14:12:00.526762+0000","last_became_peered":"2024-03-21T14:12:00.526762+0000","last_unstale":"2024-03-21T14:17:09.618504+0000","last_undegraded":"2024-03-21T14:17:09.618504+0000","last_fullsized":"2024-03-21T14:17:09.618504+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:31.430514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:31.430514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.9","version":"326'3321","reported_seq":3346,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:21.804165+0000","last_change":"2024-03-21T14:17:01.216005+0000","last_active":"2024-03-21T14:17:21.804165+0000","last_peered":"2024-03-21T14:17:21.804165+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.048295+0000","last_became_peered":"2024-03-21T14:16:51.048295+0000","last_unstale":"2024-03-21T14:17:21.804165+0000","last_undegraded":"2024-03-21T14:17:01.206551+0000","last_fullsized":"2024-03-21T14:16:51.045802+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3321,"log_dups_size":0,"ondisk_log_size":3321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:58.330943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217645056,"num_objects":3321,"num_object_clones":0,"num_object_copies":9963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3321,"num_write_kb":212544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":3321}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":809,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618883+0000","last_change":"2024-03-21T14:16:58.345839+0000","last_active":"2024-03-21T14:17:09.618883+0000","last_peered":"2024-03-21T14:17:09.618883+0000","last_clean":"2024-03-21T14:17:09.618883+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:17:09.618883+0000","last_undegraded":"2024-03-21T14:17:09.618883+0000","last_fullsized":"2024-03-21T14:17:09.618883+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:58.345739+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:58.345739+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.806247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.624245+0000","last_change":"2024-03-21T14:14:54.732344+0000","last_active":"2024-03-21T14:17:09.624245+0000","last_peered":"2024-03-21T14:17:09.624245+0000","last_clean":"2024-03-21T14:14:52.711965+0000","last_became_active":"2024-03-21T14:14:54.732344+0000","last_became_peered":"2024-03-21T14:14:54.732344+0000","last_unstale":"2024-03-21T14:17:09.624245+0000","last_undegraded":"2024-03-21T14:17:09.624245+0000","last_fullsized":"2024-03-21T14:14:54.726496+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:37.711376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:37.711376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:18.525972+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":588,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.618237+0000","last_change":"2024-03-21T14:14:54.746014+0000","last_active":"2024-03-21T14:17:09.618237+0000","last_peered":"2024-03-21T14:17:09.618237+0000","last_clean":"2024-03-21T14:14:52.716087+0000","last_became_active":"2024-03-21T14:14:54.746014+0000","last_became_peered":"2024-03-21T14:14:54.746014+0000","last_unstale":"2024-03-21T14:17:09.618237+0000","last_undegraded":"2024-03-21T14:17:09.618237+0000","last_fullsized":"2024-03-21T14:14:54.731030+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:58.667132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:12:58.667132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":103,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.624590+0000","last_change":"2024-03-21T14:14:54.730388+0000","last_active":"2024-03-21T14:17:09.624590+0000","last_peered":"2024-03-21T14:17:09.624590+0000","last_clean":"2024-03-21T14:02:31.423682+0000","last_became_active":"2024-03-21T14:14:54.730388+0000","last_became_peered":"2024-03-21T14:14:54.730388+0000","last_unstale":"2024-03-21T14:17:09.624590+0000","last_undegraded":"2024-03-21T14:17:09.624590+0000","last_fullsized":"2024-03-21T14:14:54.723417+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:06.212763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:14:06.212763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":654,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618173+0000","last_change":"2024-03-21T14:16:10.652519+0000","last_active":"2024-03-21T14:17:09.618173+0000","last_peered":"2024-03-21T14:17:09.618173+0000","last_clean":"2024-03-21T14:17:09.618173+0000","last_became_active":"2024-03-21T14:07:23.361364+0000","last_became_peered":"2024-03-21T14:07:23.361364+0000","last_unstale":"2024-03-21T14:17:09.618173+0000","last_undegraded":"2024-03-21T14:17:09.618173+0000","last_fullsized":"2024-03-21T14:17:09.618173+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:16:10.652373+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:16:10.652373+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:13.400287+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":50,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.625890+0000","last_change":"2024-03-21T14:16:21.863083+0000","last_active":"2024-03-21T14:17:09.625890+0000","last_peered":"2024-03-21T14:17:09.625890+0000","last_clean":"2024-03-21T14:02:36.433777+0000","last_became_active":"2024-03-21T14:16:21.863083+0000","last_became_peered":"2024-03-21T14:16:21.863083+0000","last_unstale":"2024-03-21T14:17:09.625890+0000","last_undegraded":"2024-03-21T14:17:09.625890+0000","last_fullsized":"2024-03-21T14:16:21.855618+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:07.244640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.244640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.613992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.b","version":"326'3382","reported_seq":3408,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:22.393971+0000","last_change":"2024-03-21T14:16:51.054881+0000","last_active":"2024-03-21T14:17:22.393971+0000","last_peered":"2024-03-21T14:17:22.393971+0000","last_clean":"2024-03-21T14:17:22.393971+0000","last_became_active":"2024-03-21T14:16:51.054471+0000","last_became_peered":"2024-03-21T14:16:51.054471+0000","last_unstale":"2024-03-21T14:17:22.393971+0000","last_undegraded":"2024-03-21T14:17:22.393971+0000","last_fullsized":"2024-03-21T14:17:22.393971+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3382,"log_dups_size":0,"ondisk_log_size":3382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:52.373755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221642752,"num_objects":3382,"num_object_clones":0,"num_object_copies":10146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3382,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3382,"num_write_kb":216448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":59,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.625651+0000","last_change":"2024-03-21T14:16:55.947536+0000","last_active":"2024-03-21T14:17:09.625651+0000","last_peered":"2024-03-21T14:17:09.625651+0000","last_clean":"2024-03-21T14:17:09.625651+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:17:09.625651+0000","last_undegraded":"2024-03-21T14:17:09.625651+0000","last_fullsized":"2024-03-21T14:17:09.625651+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:55.947438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:55.947438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:11.457883+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":885,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.622628+0000","last_change":"2024-03-21T14:14:54.746367+0000","last_active":"2024-03-21T14:17:09.622628+0000","last_peered":"2024-03-21T14:17:09.622628+0000","last_clean":"2024-03-21T14:14:52.710400+0000","last_became_active":"2024-03-21T14:14:54.746367+0000","last_became_peered":"2024-03-21T14:14:54.746367+0000","last_unstale":"2024-03-21T14:17:09.622628+0000","last_undegraded":"2024-03-21T14:17:09.622628+0000","last_fullsized":"2024-03-21T14:14:54.729618+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:37.747359+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:37.747359+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:13.605129+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":601,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.622869+0000","last_change":"2024-03-21T14:16:20.564440+0000","last_active":"2024-03-21T14:17:09.622869+0000","last_peered":"2024-03-21T14:17:09.622869+0000","last_clean":"2024-03-21T14:17:09.622869+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:17:09.622869+0000","last_undegraded":"2024-03-21T14:17:09.622869+0000","last_fullsized":"2024-03-21T14:17:09.622869+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:16:20.564321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:28.547951+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":395,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.664278+0000","last_change":"2024-03-21T14:14:57.048261+0000","last_active":"2024-03-21T14:17:13.664278+0000","last_peered":"2024-03-21T14:17:13.664278+0000","last_clean":"2024-03-21T14:17:13.664278+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:17:13.664278+0000","last_undegraded":"2024-03-21T14:17:13.664278+0000","last_fullsized":"2024-03-21T14:17:13.664278+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'29","last_scrub_stamp":"2024-03-21T14:14:57.048131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:57.048131+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":574,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.624287+0000","last_change":"2024-03-21T14:14:54.743075+0000","last_active":"2024-03-21T14:17:09.624287+0000","last_peered":"2024-03-21T14:17:09.624287+0000","last_clean":"2024-03-21T14:14:52.711749+0000","last_became_active":"2024-03-21T14:14:54.743075+0000","last_became_peered":"2024-03-21T14:14:54.743075+0000","last_unstale":"2024-03-21T14:17:09.624287+0000","last_undegraded":"2024-03-21T14:17:09.624287+0000","last_fullsized":"2024-03-21T14:14:54.725771+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.820408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.820408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:18.525964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"326'3241","reported_seq":3266,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:21.775605+0000","last_change":"2024-03-21T14:17:01.206813+0000","last_active":"2024-03-21T14:17:21.775605+0000","last_peered":"2024-03-21T14:17:21.775605+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.048170+0000","last_became_peered":"2024-03-21T14:16:51.048170+0000","last_unstale":"2024-03-21T14:17:21.775605+0000","last_undegraded":"2024-03-21T14:17:01.206430+0000","last_fullsized":"2024-03-21T14:16:51.045507+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3241,"log_dups_size":0,"ondisk_log_size":3241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:15.453539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212402176,"num_objects":3241,"num_object_clones":0,"num_object_copies":9723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3241,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3241,"num_write_kb":207424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":["7","0"],"object_location_counts":[{"shards":"0,7","objects":3241}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":656,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.621752+0000","last_change":"2024-03-21T14:16:02.693105+0000","last_active":"2024-03-21T14:17:09.621752+0000","last_peered":"2024-03-21T14:17:09.621752+0000","last_clean":"2024-03-21T14:17:09.621752+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:17:09.621752+0000","last_undegraded":"2024-03-21T14:17:09.621752+0000","last_fullsized":"2024-03-21T14:17:09.621752+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:02.692989+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:02.692989+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.572551+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":601,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:20.028344+0000","last_change":"2024-03-21T14:17:20.028344+0000","last_active":"2024-03-21T14:17:20.028344+0000","last_peered":"2024-03-21T14:17:20.028344+0000","last_clean":"2024-03-21T14:17:20.028344+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:17:20.028344+0000","last_undegraded":"2024-03-21T14:17:20.028344+0000","last_fullsized":"2024-03-21T14:17:20.028344+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:20.028230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:17:20.028230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:45.475340+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":380,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620036+0000","last_change":"2024-03-21T14:15:01.061413+0000","last_active":"2024-03-21T14:17:09.620036+0000","last_peered":"2024-03-21T14:17:09.620036+0000","last_clean":"2024-03-21T14:17:09.620036+0000","last_became_active":"2024-03-21T14:12:20.559683+0000","last_became_peered":"2024-03-21T14:12:20.559683+0000","last_unstale":"2024-03-21T14:17:09.620036+0000","last_undegraded":"2024-03-21T14:17:09.620036+0000","last_fullsized":"2024-03-21T14:17:09.620036+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:01.061102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:15:01.061102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1386,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:20.818527+0000","last_change":"2024-03-21T14:16:34.619088+0000","last_active":"2024-03-21T14:17:20.818527+0000","last_peered":"2024-03-21T14:17:20.818527+0000","last_clean":"2024-03-21T14:17:20.818527+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:17:20.818527+0000","last_undegraded":"2024-03-21T14:17:20.818527+0000","last_fullsized":"2024-03-21T14:17:20.818527+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.009777+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":823,"num_read_kb":818,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":503,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.620216+0000","last_change":"2024-03-21T14:14:54.741139+0000","last_active":"2024-03-21T14:17:09.620216+0000","last_peered":"2024-03-21T14:17:09.620216+0000","last_clean":"2024-03-21T14:14:52.718881+0000","last_became_active":"2024-03-21T14:14:54.741139+0000","last_became_peered":"2024-03-21T14:14:54.741139+0000","last_unstale":"2024-03-21T14:17:09.620216+0000","last_undegraded":"2024-03-21T14:14:54.731232+0000","last_fullsized":"2024-03-21T14:14:54.731117+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:13:43.544671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:43.544671+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:09.901305+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":53,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620938+0000","last_change":"2024-03-21T14:16:21.868921+0000","last_active":"2024-03-21T14:17:09.620938+0000","last_peered":"2024-03-21T14:17:09.620938+0000","last_clean":"2024-03-21T14:17:09.620938+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:17:09.620938+0000","last_undegraded":"2024-03-21T14:17:09.620938+0000","last_fullsized":"2024-03-21T14:17:09.620938+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:40.570508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:15:40.570508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"326'3190","reported_seq":3218,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:21.512227+0000","last_change":"2024-03-21T14:16:51.055172+0000","last_active":"2024-03-21T14:17:21.512227+0000","last_peered":"2024-03-21T14:17:21.512227+0000","last_clean":"2024-03-21T14:17:21.512227+0000","last_became_active":"2024-03-21T14:16:51.054796+0000","last_became_peered":"2024-03-21T14:16:51.054796+0000","last_unstale":"2024-03-21T14:17:21.512227+0000","last_undegraded":"2024-03-21T14:17:21.512227+0000","last_fullsized":"2024-03-21T14:17:21.512227+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3190,"log_dups_size":0,"ondisk_log_size":3190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.729411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209059840,"num_objects":3190,"num_object_clones":0,"num_object_copies":9570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3190,"num_write_kb":204160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":650,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623562+0000","last_change":"2024-03-21T14:17:01.305286+0000","last_active":"2024-03-21T14:17:09.623562+0000","last_peered":"2024-03-21T14:17:09.623562+0000","last_clean":"2024-03-21T14:17:09.623562+0000","last_became_active":"2024-03-21T14:14:35.622383+0000","last_became_peered":"2024-03-21T14:14:35.622383+0000","last_unstale":"2024-03-21T14:17:09.623562+0000","last_undegraded":"2024-03-21T14:17:09.623562+0000","last_fullsized":"2024-03-21T14:17:09.623562+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:17:01.305155+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:17:01.305155+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:03.446459+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":584,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619496+0000","last_change":"2024-03-21T14:14:54.741160+0000","last_active":"2024-03-21T14:17:09.619496+0000","last_peered":"2024-03-21T14:17:09.619496+0000","last_clean":"2024-03-21T14:14:52.714204+0000","last_became_active":"2024-03-21T14:14:54.741160+0000","last_became_peered":"2024-03-21T14:14:54.741160+0000","last_unstale":"2024-03-21T14:17:09.619496+0000","last_undegraded":"2024-03-21T14:17:09.619496+0000","last_fullsized":"2024-03-21T14:14:54.728056+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:43.550498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:14:43.550498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":590,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.621393+0000","last_change":"2024-03-21T14:14:54.741947+0000","last_active":"2024-03-21T14:17:09.621393+0000","last_peered":"2024-03-21T14:17:09.621393+0000","last_clean":"2024-03-21T14:14:52.716388+0000","last_became_active":"2024-03-21T14:14:54.741947+0000","last_became_peered":"2024-03-21T14:14:54.741947+0000","last_unstale":"2024-03-21T14:17:09.621393+0000","last_undegraded":"2024-03-21T14:17:09.621393+0000","last_fullsized":"2024-03-21T14:14:54.727020+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:55.416068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:13:55.416068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:15.855079+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":309,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.627014+0000","last_change":"2024-03-21T14:16:58.126142+0000","last_active":"2024-03-21T14:17:09.627014+0000","last_peered":"2024-03-21T14:17:09.627014+0000","last_clean":"2024-03-21T14:17:09.627014+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:17:09.627014+0000","last_undegraded":"2024-03-21T14:17:09.627014+0000","last_fullsized":"2024-03-21T14:17:09.627014+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:16:58.125989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:16:58.125989+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:09.891261+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":177,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.621706+0000","last_change":"2024-03-21T14:16:21.862283+0000","last_active":"2024-03-21T14:17:09.621706+0000","last_peered":"2024-03-21T14:17:09.621706+0000","last_clean":"2024-03-21T14:14:52.705626+0000","last_became_active":"2024-03-21T14:16:21.862283+0000","last_became_peered":"2024-03-21T14:16:21.862283+0000","last_unstale":"2024-03-21T14:17:09.621706+0000","last_undegraded":"2024-03-21T14:17:09.621706+0000","last_fullsized":"2024-03-21T14:16:21.852923+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:55.222353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:12:55.222353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.020066+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"326'3355","reported_seq":3383,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:21.789478+0000","last_change":"2024-03-21T14:16:51.057360+0000","last_active":"2024-03-21T14:17:21.789478+0000","last_peered":"2024-03-21T14:17:21.789478+0000","last_clean":"2024-03-21T14:17:21.789478+0000","last_became_active":"2024-03-21T14:16:51.056996+0000","last_became_peered":"2024-03-21T14:16:51.056996+0000","last_unstale":"2024-03-21T14:17:21.789478+0000","last_undegraded":"2024-03-21T14:17:21.789478+0000","last_fullsized":"2024-03-21T14:17:21.789478+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3355,"log_dups_size":0,"ondisk_log_size":3355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.215097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219873280,"num_objects":3355,"num_object_clones":0,"num_object_copies":10065,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3355,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3355,"num_write_kb":214720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":893,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.623506+0000","last_change":"2024-03-21T14:14:54.743500+0000","last_active":"2024-03-21T14:17:09.623506+0000","last_peered":"2024-03-21T14:17:09.623506+0000","last_clean":"2024-03-21T14:14:52.711337+0000","last_became_active":"2024-03-21T14:14:54.743500+0000","last_became_peered":"2024-03-21T14:14:54.743500+0000","last_unstale":"2024-03-21T14:17:09.623506+0000","last_undegraded":"2024-03-21T14:17:09.623506+0000","last_fullsized":"2024-03-21T14:14:54.728617+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:36.741153+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:36.741153+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.614039+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":75,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:09.623098+0000","last_change":"2024-03-21T14:16:09.598659+0000","last_active":"2024-03-21T14:17:09.623098+0000","last_peered":"2024-03-21T14:17:09.623098+0000","last_clean":"2024-03-21T14:17:09.623098+0000","last_became_active":"2024-03-21T14:16:00.605705+0000","last_became_peered":"2024-03-21T14:16:00.605705+0000","last_unstale":"2024-03-21T14:17:09.623098+0000","last_undegraded":"2024-03-21T14:17:09.623098+0000","last_fullsized":"2024-03-21T14:17:09.623098+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:09.598439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:09.598439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.377213+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,6],"avail_no_missing":["3","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":520,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.623708+0000","last_change":"2024-03-21T14:16:21.865096+0000","last_active":"2024-03-21T14:17:09.623708+0000","last_peered":"2024-03-21T14:17:09.623708+0000","last_clean":"2024-03-21T14:14:52.706578+0000","last_became_active":"2024-03-21T14:16:21.865096+0000","last_became_peered":"2024-03-21T14:16:21.865096+0000","last_unstale":"2024-03-21T14:17:09.623708+0000","last_undegraded":"2024-03-21T14:17:09.623708+0000","last_fullsized":"2024-03-21T14:16:21.859801+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:12:45.295537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:12:45.295537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853403+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":626,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623802+0000","last_change":"2024-03-21T14:15:39.434842+0000","last_active":"2024-03-21T14:17:09.623802+0000","last_peered":"2024-03-21T14:17:09.623802+0000","last_clean":"2024-03-21T14:17:09.623802+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:17:09.623802+0000","last_undegraded":"2024-03-21T14:17:09.623802+0000","last_fullsized":"2024-03-21T14:17:09.623802+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:15:39.434689+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:15:39.434689+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":521,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.663757+0000","last_change":"2024-03-21T14:16:21.864198+0000","last_active":"2024-03-21T14:17:13.663757+0000","last_peered":"2024-03-21T14:17:13.663757+0000","last_clean":"2024-03-21T14:17:13.663757+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:17:13.663757+0000","last_undegraded":"2024-03-21T14:17:13.663757+0000","last_fullsized":"2024-03-21T14:17:13.663757+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"226'10","last_scrub_stamp":"2024-03-21T14:16:07.563840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:07.563840+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"326'3294","reported_seq":3319,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:22.170021+0000","last_change":"2024-03-21T14:17:01.207904+0000","last_active":"2024-03-21T14:17:22.170021+0000","last_peered":"2024-03-21T14:17:22.170021+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.052312+0000","last_became_peered":"2024-03-21T14:16:51.052312+0000","last_unstale":"2024-03-21T14:17:22.170021+0000","last_undegraded":"2024-03-21T14:17:01.207684+0000","last_fullsized":"2024-03-21T14:16:51.043676+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3294,"log_dups_size":0,"ondisk_log_size":3294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.036790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215875584,"num_objects":3294,"num_object_clones":0,"num_object_copies":9882,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3294,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3294,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3294,"num_write_kb":210816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":3294}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":648,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623310+0000","last_change":"2024-03-21T14:16:13.521253+0000","last_active":"2024-03-21T14:17:09.623310+0000","last_peered":"2024-03-21T14:17:09.623310+0000","last_clean":"2024-03-21T14:17:09.623310+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:17:09.623310+0000","last_undegraded":"2024-03-21T14:17:09.623310+0000","last_fullsized":"2024-03-21T14:17:09.623310+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:13.521115+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.521115+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:33.283687+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":576,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.622795+0000","last_change":"2024-03-21T14:14:54.743852+0000","last_active":"2024-03-21T14:17:09.622795+0000","last_peered":"2024-03-21T14:17:09.622795+0000","last_clean":"2024-03-21T14:14:52.711546+0000","last_became_active":"2024-03-21T14:14:54.743852+0000","last_became_peered":"2024-03-21T14:14:54.743852+0000","last_unstale":"2024-03-21T14:17:09.622795+0000","last_undegraded":"2024-03-21T14:17:09.622795+0000","last_fullsized":"2024-03-21T14:14:54.732762+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:51.612367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:02.083143+0000","last_clean_scrub_stamp":"2024-03-21T14:13:51.612367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:18.525998+0000","scrub_duration":79,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":444,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618138+0000","last_change":"2024-03-21T14:15:02.631337+0000","last_active":"2024-03-21T14:17:09.618138+0000","last_peered":"2024-03-21T14:17:09.618138+0000","last_clean":"2024-03-21T14:17:09.618138+0000","last_became_active":"2024-03-21T14:07:23.359883+0000","last_became_peered":"2024-03-21T14:07:23.359883+0000","last_unstale":"2024-03-21T14:17:09.618138+0000","last_undegraded":"2024-03-21T14:17:09.618138+0000","last_fullsized":"2024-03-21T14:17:09.618138+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:15:02.631076+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:15:02.631076+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":639,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.619978+0000","last_change":"2024-03-21T14:15:59.624035+0000","last_active":"2024-03-21T14:17:09.619978+0000","last_peered":"2024-03-21T14:17:09.619978+0000","last_clean":"2024-03-21T14:17:09.619978+0000","last_became_active":"2024-03-21T14:15:59.607826+0000","last_became_peered":"2024-03-21T14:15:59.607826+0000","last_unstale":"2024-03-21T14:17:09.619978+0000","last_undegraded":"2024-03-21T14:17:09.619978+0000","last_fullsized":"2024-03-21T14:17:09.619978+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'28","last_scrub_stamp":"2024-03-21T14:14:58.038582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:58.038582+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":765,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.663903+0000","last_change":"2024-03-21T14:15:31.699369+0000","last_active":"2024-03-21T14:17:13.663903+0000","last_peered":"2024-03-21T14:17:13.663903+0000","last_clean":"2024-03-21T14:17:13.663903+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:17:13.663903+0000","last_undegraded":"2024-03-21T14:17:13.663903+0000","last_fullsized":"2024-03-21T14:17:13.663903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:15:31.699239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:15:31.699239+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"326'3184","reported_seq":3209,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:21.489822+0000","last_change":"2024-03-21T14:17:01.210297+0000","last_active":"2024-03-21T14:17:21.489822+0000","last_peered":"2024-03-21T14:17:21.489822+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.053795+0000","last_became_peered":"2024-03-21T14:16:51.053795+0000","last_unstale":"2024-03-21T14:17:21.489822+0000","last_undegraded":"2024-03-21T14:17:01.206110+0000","last_fullsized":"2024-03-21T14:16:51.048364+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3184,"log_dups_size":0,"ondisk_log_size":3184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:07.322836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208666624,"num_objects":3184,"num_object_clones":0,"num_object_copies":9552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3184,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3184,"num_write_kb":203776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":3184}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":645,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.620138+0000","last_change":"2024-03-21T14:14:54.739215+0000","last_active":"2024-03-21T14:17:09.620138+0000","last_peered":"2024-03-21T14:17:09.620138+0000","last_clean":"2024-03-21T14:14:52.716527+0000","last_became_active":"2024-03-21T14:14:54.739215+0000","last_became_peered":"2024-03-21T14:14:54.739215+0000","last_unstale":"2024-03-21T14:17:09.620138+0000","last_undegraded":"2024-03-21T14:17:09.620138+0000","last_fullsized":"2024-03-21T14:14:54.724221+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.954782+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.954782+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:11.358483+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":593,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:11.405951+0000","last_change":"2024-03-21T14:17:11.405951+0000","last_active":"2024-03-21T14:17:11.405951+0000","last_peered":"2024-03-21T14:17:11.405951+0000","last_clean":"2024-03-21T14:17:11.405951+0000","last_became_active":"2024-03-21T14:13:18.543885+0000","last_became_peered":"2024-03-21T14:13:18.543885+0000","last_unstale":"2024-03-21T14:17:11.405951+0000","last_undegraded":"2024-03-21T14:17:11.405951+0000","last_fullsized":"2024-03-21T14:17:11.405951+0000","mapping_epoch":249,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":250,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:11.405823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:17:11.405823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:28.319568+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":594,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.625101+0000","last_change":"2024-03-21T14:17:02.767201+0000","last_active":"2024-03-21T14:17:09.625101+0000","last_peered":"2024-03-21T14:17:09.625101+0000","last_clean":"2024-03-21T14:17:09.625101+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:17:09.625101+0000","last_undegraded":"2024-03-21T14:17:09.625101+0000","last_fullsized":"2024-03-21T14:17:09.625101+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:02.767079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:17:02.767079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:31.502000+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":574,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:19.581066+0000","last_change":"2024-03-21T14:17:19.581121+0000","last_active":"2024-03-21T14:17:19.581066+0000","last_peered":"2024-03-21T14:17:19.581066+0000","last_clean":"2024-03-21T14:17:19.581066+0000","last_became_active":"2024-03-21T14:08:17.134104+0000","last_became_peered":"2024-03-21T14:08:17.134104+0000","last_unstale":"2024-03-21T14:17:19.581066+0000","last_undegraded":"2024-03-21T14:17:19.581066+0000","last_fullsized":"2024-03-21T14:17:19.581066+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:17:19.580993+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:17:19.580993+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:28.912952+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":571,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:14.254451+0000","last_change":"2024-03-21T14:17:14.254451+0000","last_active":"2024-03-21T14:17:14.254451+0000","last_peered":"2024-03-21T14:17:14.254451+0000","last_clean":"2024-03-21T14:17:14.254451+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:17:14.254451+0000","last_undegraded":"2024-03-21T14:17:14.254451+0000","last_fullsized":"2024-03-21T14:17:14.254451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:14.254334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:14.254334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:35.685465+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"326'3265","reported_seq":3293,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:21.510009+0000","last_change":"2024-03-21T14:16:51.052270+0000","last_active":"2024-03-21T14:17:21.510009+0000","last_peered":"2024-03-21T14:17:21.510009+0000","last_clean":"2024-03-21T14:17:21.510009+0000","last_became_active":"2024-03-21T14:16:51.051942+0000","last_became_peered":"2024-03-21T14:16:51.051942+0000","last_unstale":"2024-03-21T14:17:21.510009+0000","last_undegraded":"2024-03-21T14:17:21.510009+0000","last_fullsized":"2024-03-21T14:17:21.510009+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3265,"log_dups_size":0,"ondisk_log_size":3265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:02.736930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213975040,"num_objects":3265,"num_object_clones":0,"num_object_copies":9795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3265,"num_write_kb":208960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":565,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619052+0000","last_change":"2024-03-21T14:16:21.860397+0000","last_active":"2024-03-21T14:17:09.619052+0000","last_peered":"2024-03-21T14:17:09.619052+0000","last_clean":"2024-03-21T14:09:26.509792+0000","last_became_active":"2024-03-21T14:16:21.860397+0000","last_became_peered":"2024-03-21T14:16:21.860397+0000","last_unstale":"2024-03-21T14:17:09.619052+0000","last_undegraded":"2024-03-21T14:17:09.619052+0000","last_fullsized":"2024-03-21T14:16:21.852037+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:37.210371+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:37.210371+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:11.358465+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":583,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:09.617817+0000","last_change":"2024-03-21T14:17:00.985782+0000","last_active":"2024-03-21T14:17:09.617817+0000","last_peered":"2024-03-21T14:17:09.617817+0000","last_clean":"2024-03-21T14:17:09.617817+0000","last_became_active":"2024-03-21T14:16:22.893432+0000","last_became_peered":"2024-03-21T14:16:22.893432+0000","last_unstale":"2024-03-21T14:17:09.617817+0000","last_undegraded":"2024-03-21T14:17:09.617817+0000","last_fullsized":"2024-03-21T14:17:09.617817+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:00.985637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:17:00.985637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:09.581194+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,6],"avail_no_missing":["1","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":764,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:13.664393+0000","last_change":"2024-03-21T14:17:03.353514+0000","last_active":"2024-03-21T14:17:13.664393+0000","last_peered":"2024-03-21T14:17:13.664393+0000","last_clean":"2024-03-21T14:17:13.664393+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:17:13.664393+0000","last_undegraded":"2024-03-21T14:17:13.664393+0000","last_fullsized":"2024-03-21T14:17:13.664393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:17:03.353394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.353394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:30.866059+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":432,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.618617+0000","last_change":"2024-03-21T14:14:53.682726+0000","last_active":"2024-03-21T14:17:09.618617+0000","last_peered":"2024-03-21T14:17:09.618617+0000","last_clean":"2024-03-21T14:17:09.618617+0000","last_became_active":"2024-03-21T14:07:23.361561+0000","last_became_peered":"2024-03-21T14:07:23.361561+0000","last_unstale":"2024-03-21T14:17:09.618617+0000","last_undegraded":"2024-03-21T14:17:09.618617+0000","last_fullsized":"2024-03-21T14:17:09.618617+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:53.682640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:53.682640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":633,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.620485+0000","last_change":"2024-03-21T14:17:08.943872+0000","last_active":"2024-03-21T14:17:09.620485+0000","last_peered":"2024-03-21T14:17:09.620485+0000","last_clean":"2024-03-21T14:17:09.620485+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:17:09.620485+0000","last_undegraded":"2024-03-21T14:17:09.620485+0000","last_fullsized":"2024-03-21T14:17:09.620485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:17:08.943688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:08.943688+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:19.467204+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"326'3461","reported_seq":3488,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:22.386101+0000","last_change":"2024-03-21T14:16:51.054830+0000","last_active":"2024-03-21T14:17:22.386101+0000","last_peered":"2024-03-21T14:17:22.386101+0000","last_clean":"2024-03-21T14:17:22.386101+0000","last_became_active":"2024-03-21T14:16:51.054526+0000","last_became_peered":"2024-03-21T14:16:51.054526+0000","last_unstale":"2024-03-21T14:17:22.386101+0000","last_undegraded":"2024-03-21T14:17:22.386101+0000","last_fullsized":"2024-03-21T14:17:22.386101+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3461,"log_dups_size":0,"ondisk_log_size":3461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:12.076219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226820096,"num_objects":3461,"num_object_clones":0,"num_object_copies":10383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3461,"num_write_kb":221504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":580,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623412+0000","last_change":"2024-03-21T14:15:37.646778+0000","last_active":"2024-03-21T14:17:09.623412+0000","last_peered":"2024-03-21T14:17:09.623412+0000","last_clean":"2024-03-21T14:17:09.623412+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:17:09.623412+0000","last_undegraded":"2024-03-21T14:17:09.623412+0000","last_fullsized":"2024-03-21T14:17:09.623412+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:37.646668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:37.646668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":559,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:09.621386+0000","last_change":"2024-03-21T14:16:22.893583+0000","last_active":"2024-03-21T14:17:09.621386+0000","last_peered":"2024-03-21T14:17:09.621386+0000","last_clean":"2024-03-21T14:17:09.621386+0000","last_became_active":"2024-03-21T14:16:22.893178+0000","last_became_peered":"2024-03-21T14:16:22.893178+0000","last_unstale":"2024-03-21T14:17:09.621386+0000","last_undegraded":"2024-03-21T14:17:09.621386+0000","last_fullsized":"2024-03-21T14:17:09.621386+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:16.566202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:16.566202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:29.193550+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,6],"avail_no_missing":["4","5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":581,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.619661+0000","last_change":"2024-03-21T14:14:54.739307+0000","last_active":"2024-03-21T14:17:09.619661+0000","last_peered":"2024-03-21T14:17:09.619661+0000","last_clean":"2024-03-21T14:14:52.715781+0000","last_became_active":"2024-03-21T14:14:54.739307+0000","last_became_peered":"2024-03-21T14:14:54.739307+0000","last_unstale":"2024-03-21T14:17:09.619661+0000","last_undegraded":"2024-03-21T14:17:09.619661+0000","last_fullsized":"2024-03-21T14:14:54.726477+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:13:45.602994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:13:45.602994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358594+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":623,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.623494+0000","last_change":"2024-03-21T14:15:32.019352+0000","last_active":"2024-03-21T14:17:09.623494+0000","last_peered":"2024-03-21T14:17:09.623494+0000","last_clean":"2024-03-21T14:17:09.623494+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:17:09.623494+0000","last_undegraded":"2024-03-21T14:17:09.623494+0000","last_fullsized":"2024-03-21T14:17:09.623494+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:15:32.019116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:15:32.019116+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"326'3234","reported_seq":3259,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:21.495751+0000","last_change":"2024-03-21T14:17:01.207830+0000","last_active":"2024-03-21T14:17:21.495751+0000","last_peered":"2024-03-21T14:17:21.495751+0000","last_clean":"2024-03-21T14:16:50.034688+0000","last_became_active":"2024-03-21T14:16:51.052954+0000","last_became_peered":"2024-03-21T14:16:51.052954+0000","last_unstale":"2024-03-21T14:17:21.495751+0000","last_undegraded":"2024-03-21T14:17:01.207619+0000","last_fullsized":"2024-03-21T14:16:51.046136+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3234,"log_dups_size":0,"ondisk_log_size":3234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:14.656424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211943424,"num_objects":3234,"num_object_clones":0,"num_object_copies":9702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3234,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3234,"num_write_kb":206976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":3234}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":564,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.622812+0000","last_change":"2024-03-21T14:16:21.860630+0000","last_active":"2024-03-21T14:17:09.622812+0000","last_peered":"2024-03-21T14:17:09.622812+0000","last_clean":"2024-03-21T14:09:26.509792+0000","last_became_active":"2024-03-21T14:16:21.860630+0000","last_became_peered":"2024-03-21T14:16:21.860630+0000","last_unstale":"2024-03-21T14:17:09.622812+0000","last_undegraded":"2024-03-21T14:17:09.622812+0000","last_fullsized":"2024-03-21T14:16:21.852422+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:36.244697+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:36.244697+0000","objects_scrubbed":2,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.849543+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.f","version":"72'1","reported_seq":114,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.624891+0000","last_change":"2024-03-21T14:14:54.726685+0000","last_active":"2024-03-21T14:17:09.624891+0000","last_peered":"2024-03-21T14:17:09.624891+0000","last_clean":"2024-03-21T14:02:40.461300+0000","last_became_active":"2024-03-21T14:14:54.726685+0000","last_became_peered":"2024-03-21T14:14:54.726685+0000","last_unstale":"2024-03-21T14:17:09.624891+0000","last_undegraded":"2024-03-21T14:14:54.724018+0000","last_fullsized":"2024-03-21T14:14:54.723895+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'1","last_scrub_stamp":"2024-03-21T14:13:36.334881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:13:36.334881+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:12.853487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":586,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.621544+0000","last_change":"2024-03-21T14:15:30.432822+0000","last_active":"2024-03-21T14:17:09.621544+0000","last_peered":"2024-03-21T14:17:09.621544+0000","last_clean":"2024-03-21T14:17:09.621544+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:17:09.621544+0000","last_undegraded":"2024-03-21T14:17:09.621544+0000","last_fullsized":"2024-03-21T14:17:09.621544+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:30.432721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_clean_scrub_stamp":"2024-03-21T14:15:30.432721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":260,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:16.247219+0000","last_change":"2024-03-21T14:17:16.248535+0000","last_active":"2024-03-21T14:17:16.247219+0000","last_peered":"2024-03-21T14:17:16.247219+0000","last_clean":"2024-03-21T14:17:16.247219+0000","last_became_active":"2024-03-21T14:12:00.530704+0000","last_became_peered":"2024-03-21T14:12:00.530704+0000","last_unstale":"2024-03-21T14:17:16.247219+0000","last_undegraded":"2024-03-21T14:17:16.247219+0000","last_fullsized":"2024-03-21T14:17:16.247219+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:16.247145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.247145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:26.479116+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":618,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:09.620949+0000","last_change":"2024-03-21T14:14:54.732979+0000","last_active":"2024-03-21T14:17:09.620949+0000","last_peered":"2024-03-21T14:17:09.620949+0000","last_clean":"2024-03-21T14:14:52.716824+0000","last_became_active":"2024-03-21T14:14:54.732979+0000","last_became_peered":"2024-03-21T14:14:54.732979+0000","last_unstale":"2024-03-21T14:17:09.620949+0000","last_undegraded":"2024-03-21T14:14:54.728543+0000","last_fullsized":"2024-03-21T14:14:54.728421+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:12:53.529362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:12:53.529362+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:14.358581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"326'3456","reported_seq":3484,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:22.387820+0000","last_change":"2024-03-21T14:16:51.056904+0000","last_active":"2024-03-21T14:17:22.387820+0000","last_peered":"2024-03-21T14:17:22.387820+0000","last_clean":"2024-03-21T14:17:22.387820+0000","last_became_active":"2024-03-21T14:16:51.056577+0000","last_became_peered":"2024-03-21T14:16:51.056577+0000","last_unstale":"2024-03-21T14:17:22.387820+0000","last_undegraded":"2024-03-21T14:17:22.387820+0000","last_fullsized":"2024-03-21T14:17:22.387820+0000","mapping_epoch":317,"log_start":"0'0","ondisk_log_start":"0'0","created":317,"last_epoch_clean":318,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:50.034688+0000","last_clean_scrub_stamp":"2024-03-21T14:16:50.034688+0000","objects_scrubbed":0,"log_size":3456,"log_dups_size":0,"ondisk_log_size":3456,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:02.310663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226492416,"num_objects":3456,"num_object_clones":0,"num_object_copies":10368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3456,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3456,"num_write_kb":221184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":893,"reported_epoch":326,"state":"active+undersized","last_fresh":"2024-03-21T14:17:09.621910+0000","last_change":"2024-03-21T14:14:54.737618+0000","last_active":"2024-03-21T14:17:09.621910+0000","last_peered":"2024-03-21T14:17:09.621910+0000","last_clean":"2024-03-21T14:14:52.710614+0000","last_became_active":"2024-03-21T14:14:54.737618+0000","last_became_peered":"2024-03-21T14:14:54.737618+0000","last_unstale":"2024-03-21T14:17:09.621910+0000","last_undegraded":"2024-03-21T14:17:09.621910+0000","last_fullsized":"2024-03-21T14:14:54.727680+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.696891+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.696891+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:17:10.614024+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":577,"reported_epoch":326,"state":"active+clean+remapped","last_fresh":"2024-03-21T14:17:09.623773+0000","last_change":"2024-03-21T14:16:00.603039+0000","last_active":"2024-03-21T14:17:09.623773+0000","last_peered":"2024-03-21T14:17:09.623773+0000","last_clean":"2024-03-21T14:17:09.623773+0000","last_became_active":"2024-03-21T14:16:00.602424+0000","last_became_peered":"2024-03-21T14:16:00.602424+0000","last_unstale":"2024-03-21T14:17:09.623773+0000","last_undegraded":"2024-03-21T14:17:09.623773+0000","last_fullsized":"2024-03-21T14:17:09.623773+0000","mapping_epoch":297,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":298,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:15:33.112232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:15:33.112232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,6],"avail_no_missing":["4","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"279'10","reported_seq":270,"reported_epoch":326,"state":"active+undersized+degraded","last_fresh":"2024-03-21T14:17:13.663775+0000","last_change":"2024-03-21T14:14:54.743410+0000","last_active":"2024-03-21T14:17:13.663775+0000","last_peered":"2024-03-21T14:17:13.663775+0000","last_clean":"2024-03-21T14:05:32.338633+0000","last_became_active":"2024-03-21T14:14:54.743410+0000","last_became_peered":"2024-03-21T14:14:54.743410+0000","last_unstale":"2024-03-21T14:17:13.663775+0000","last_undegraded":"2024-03-21T14:14:54.727338+0000","last_fullsized":"2024-03-21T14:14:54.727093+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'9","last_scrub_stamp":"2024-03-21T14:14:05.166173+0000","last_deep_scrub":"104'9","last_deep_scrub_stamp":"2024-03-21T14:05:32.346553+0000","last_clean_scrub_stamp":"2024-03-21T14:14:05.166173+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":600,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:17.922036+0000","last_change":"2024-03-21T14:17:17.922071+0000","last_active":"2024-03-21T14:17:17.922036+0000","last_peered":"2024-03-21T14:17:17.922036+0000","last_clean":"2024-03-21T14:17:17.922036+0000","last_became_active":"2024-03-21T14:07:23.353181+0000","last_became_peered":"2024-03-21T14:07:23.353181+0000","last_unstale":"2024-03-21T14:17:17.922036+0000","last_undegraded":"2024-03-21T14:17:17.922036+0000","last_fullsized":"2024-03-21T14:17:17.922036+0000","mapping_epoch":129,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":130,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:17.921956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:17.921956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:18:47.408802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":627,"reported_epoch":326,"state":"active+clean","last_fresh":"2024-03-21T14:17:09.619915+0000","last_change":"2024-03-21T14:15:40.120088+0000","last_active":"2024-03-21T14:17:09.619915+0000","last_peered":"2024-03-21T14:17:09.619915+0000","last_clean":"2024-03-21T14:17:09.619915+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:17:09.619915+0000","last_undegraded":"2024-03-21T14:17:09.619915+0000","last_fullsized":"2024-03-21T14:17:09.619915+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:15:40.119580+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:15:40.119580+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":3459973120,"num_objects":52795,"num_object_clones":0,"num_object_copies":158385,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":26130,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52795,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":52795,"num_write_kb":3378880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8687976448,"data_stored":8687976448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52795,"ondisk_log_size":52795,"up":40,"acting":40,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":823,"num_read_kb":818,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":435019,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":38,"num_read_kb":55,"num_write":32,"num_write_kb":438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":429358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":26,"acting":26,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":85,"acting":86,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":58,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":2712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":754,"ondisk_log_size":754,"up":85,"acting":86,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":83,"acting":85,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":79,"acting":82,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103350,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1881644,"kb_used_data":1259044,"kb_used_omap":1,"kb_used_meta":622590,"kb_avail":91863508,"statfs":{"total":95995035648,"available":94068232192,"internally_reserved":0,"allocated":1289261056,"data_stored":1288601386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":637532618},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":206158430395,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1533436,"kb_used_data":862260,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92211716,"statfs":{"total":95995035648,"available":94424797184,"internally_reserved":0,"allocated":882954240,"data_stored":882278493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2241884,"kb_used_data":1719252,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":91503268,"statfs":{"total":95995035648,"available":93699346432,"internally_reserved":0,"allocated":1760514048,"data_stored":1759869362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3212408,"kb_used_data":2753648,"kb_used_omap":1,"kb_used_meta":458750,"kb_avail":90532744,"statfs":{"total":95995035648,"available":92705529856,"internally_reserved":0,"allocated":2819735552,"data_stored":2819353006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":469760453},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182602,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1382504,"kb_used_data":845984,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":92362648,"statfs":{"total":95995035648,"available":94579351552,"internally_reserved":0,"allocated":866287616,"data_stored":865631257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":549386695},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019078,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3018348,"kb_used_data":2481828,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90726804,"statfs":{"total":95995035648,"available":92904247296,"internally_reserved":0,"allocated":2541391872,"data_stored":2540731027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,4,5],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692033,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1815540,"kb_used_data":1220716,"kb_used_omap":1,"kb_used_meta":594814,"kb_avail":91929612,"statfs":{"total":95995035648,"available":94135922688,"internally_reserved":0,"allocated":1250013184,"data_stored":1249345672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":609089989},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1763180544,"data_stored":1763180544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":893452288,"data_stored":893452288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":869859328,"data_stored":869859328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2574385152,"data_stored":2574385152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1290862592,"data_stored":1290862592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1296236544,"data_stored":1296236544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:17:25.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg force-recovery 4.19 4.12 4.17 10.7 4.4 10.9 10.8 10.f 10.3 2024-03-21T14:17:25.924 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:25.959 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:25.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:25.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:26.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:26.096 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:26.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:26.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 60532 60516 151.272 109.625 0.00546924 0.00660617 2024-03-21T14:17:26.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:26.336 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:26.372 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:26.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:26.510 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:26.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:26.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:26.700 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:17:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:17:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:17:26.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:26.783 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:26.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:26.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:26.920 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:26.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:26 smithi003 ceph-mon[29576]: pgmap v679: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s; 26192/158976 objects degraded (16.475%); 6/158976 objects misplaced (0.004%) 2024-03-21T14:17:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:26 smithi003 ceph-mon[40510]: pgmap v679: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s; 26192/158976 objects degraded (16.475%); 6/158976 objects misplaced (0.004%) 2024-03-21T14:17:27.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:27.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:27.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:27.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:26 smithi118 ceph-mon[33862]: pgmap v679: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.2 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s; 26192/158976 objects degraded (16.475%); 6/158976 objects misplaced (0.004%) 2024-03-21T14:17:27.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:17:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:17:27.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:27.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 62278 62262 149.651 109.125 0.00361298 0.0066776 2024-03-21T14:17:27.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:27.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:27.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:17:27.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:27.465 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:27.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:27.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:27.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:27.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:27.790 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:27.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:27.887 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:27.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:27.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:28.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:28.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:28.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.168 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:28.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:28.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 63948 63932 147.974 104.375 0.00981211 0.00675404 2024-03-21T14:17:28.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:28.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:28.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:28.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:17:28.443 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:28.545 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:28.589 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:28.691 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:28.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:28.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:28.854 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:17:28.887 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:28.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:28.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[29576]: pgmap v680: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:29.109 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[40510]: pgmap v680: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:29.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:17:29.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:17:29.110 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:28 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:17:29.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:28 smithi118 ceph-mon[33862]: pgmap v680: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:28 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:17:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:28 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:17:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:28 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:17:29.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:29.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 65459 65443 146.061 94.4375 0.00586956 0.00684163 2024-03-21T14:17:29.213 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:17:29.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:17:29.283 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:17:29.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:29.327 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:29.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:29.332 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:17:29.370 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:17:29.407 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:17:29.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:29.444 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:17:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:17:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:17:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:17:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:17:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:17:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:17:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:17:29.445 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:17:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:29.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:29.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.637 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:29.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:29.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:29.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.780 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:29.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:29.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:29.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:30.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.046 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:30.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:30.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:30.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:30.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:29 smithi118 ceph-mon[33862]: osdmap e329: 8 total, 7 up, 7 in 2024-03-21T14:17:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2674361568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:29 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 30629/185496 objects degraded (16.512%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:29 smithi118 ceph-mon[33862]: osdmap e330: 8 total, 7 up, 7 in 2024-03-21T14:17:30.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 66827 66811 143.973 85.5 0.016812 0.00694069 2024-03-21T14:17:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[29576]: osdmap e329: 8 total, 7 up, 7 in 2024-03-21T14:17:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2674361568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 30629/185496 objects degraded (16.512%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[29576]: osdmap e330: 8 total, 7 up, 7 in 2024-03-21T14:17:30.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[40510]: osdmap e329: 8 total, 7 up, 7 in 2024-03-21T14:17:30.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2674361568' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:30.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 30629/185496 objects degraded (16.512%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:30.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:29 smithi003 ceph-mon[40510]: osdmap e330: 8 total, 7 up, 7 in 2024-03-21T14:17:30.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:30.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:30.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:30.444 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:30.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:30.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:30.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:30.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:30.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.723 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:30.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:30.824 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:30.873 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:30.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:30.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:31.017 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:31.119 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:31.156 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:30 smithi118 ceph-mon[33862]: pgmap v682: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:31.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 68405 68389 142.461 98.625 0.0067163 0.00701491 2024-03-21T14:17:31.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:30 smithi003 ceph-mon[29576]: pgmap v682: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:30 smithi003 ceph-mon[40510]: pgmap v682: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 92 MiB/s wr, 1.47k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:31.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:31.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:31.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:31.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:31.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:31.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:31.562 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:31.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:31.687 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:31.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:31.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:31.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:31.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:31.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:31.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:32.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:32.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.094 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:32.196 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:32.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 69812 69796 140.702 87.9375 0.00821045 0.00710297 2024-03-21T14:17:32.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.235 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:32.337 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:32.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.378 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:32.479 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:32.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:32.618 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:32.654 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:32.756 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:32.779 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:32.881 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:32.906 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:32.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:33.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:33.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:33.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.177 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:33.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:32 smithi118 ceph-mon[33862]: pgmap v684: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 738 B/s rd, 80 MiB/s wr, 1.28k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:32 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:33.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 71243 71227 139.1 89.4375 0.00612887 0.00718488 2024-03-21T14:17:33.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:32 smithi003 ceph-mon[29576]: pgmap v684: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 738 B/s rd, 80 MiB/s wr, 1.28k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:33.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:32 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:33.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:33.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:32 smithi003 ceph-mon[40510]: pgmap v684: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 3.8 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 738 B/s rd, 80 MiB/s wr, 1.28k op/s; 30629/185496 objects degraded (16.512%); 6/185496 objects misplaced (0.003%) 2024-03-21T14:17:33.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:32 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:33.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:33.254 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:33.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:17:33.300 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:17:33.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.308 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:33.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:33.409 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:33.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:33.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:33.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:33.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:33.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:33.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:33.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:33.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:33.862 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:33.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:33.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:34.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:34.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:34.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.145 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:34.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:34.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 72730 72714 137.701 92.9375 0.00620116 0.00725752 2024-03-21T14:17:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:instructing pg(s) [4.12,4.17] on osd.1 to force-recovery; instructing pg(s) [4.4,4.19,10.f] on osd.3 to force-recovery; instructing pg(s) [10.3,10.7] on osd.4 to force-recovery; instructing pg(s) [10.8,10.9] on osd.7 to force-recovery; 2024-03-21T14:17:34.233 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-21T14:17:34.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:34.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:34.382 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:34.415 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:34.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:34.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:34.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:34.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:34.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:34.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:34.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:34.920 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:34.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:34.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:35.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:35.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:35.096 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:35.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:34 smithi118 ceph-mon[33862]: pgmap v685: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 34456/208701 objects degraded (16.510%); 6/208701 objects misplaced (0.003%) 2024-03-21T14:17:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:34 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 34456/208701 objects degraded (16.510%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:35.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:35.204 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 15 74189 74174 136.334 91.25 0.0144359 0.00733019 2024-03-21T14:17:35.237 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:35.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:35.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:34 smithi003 ceph-mon[29576]: pgmap v685: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 34456/208701 objects degraded (16.510%); 6/208701 objects misplaced (0.003%) 2024-03-21T14:17:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:34 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 34456/208701 objects degraded (16.510%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:35.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:34 smithi003 ceph-mon[40510]: pgmap v685: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 129 MiB/s wr, 2.07k op/s; 34456/208701 objects degraded (16.510%); 6/208701 objects misplaced (0.003%) 2024-03-21T14:17:35.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:34 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 34456/208701 objects degraded (16.510%), 22 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:35.301 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:17:35.339 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:35.376 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:35.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:35.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:35.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:35.513 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:35.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:35.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:35.651 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:35.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:35.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:35.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:35.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:35.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:36.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:36.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:36.149 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:35 smithi003 ceph-mon[29576]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["4.19", "4.12", "4.17", "10.7", "4.4", "10.9", "10.8", "10.f", "10.3"], "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:17:36.149 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:35 smithi003 ceph-mon[40510]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["4.19", "4.12", "4.17", "10.7", "4.4", "10.9", "10.8", "10.f", "10.3"], "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:17:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:35 smithi118 ceph-mon[33862]: from='client.15699 -' entity='client.admin' cmd=[{"prefix": "pg force-recovery", "pgid": ["4.19", "4.12", "4.17", "10.7", "4.4", "10.9", "10.8", "10.f", "10.3"], "target": ["mon-mgr", ""]}]: dispatch 2024-03-21T14:17:36.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 75351 75335 134.512 72.5625 0.00576171 0.0074288 2024-03-21T14:17:36.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:36.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:36.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:36.333 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:36.357 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:36.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:36.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:36.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:36.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:36.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:36.637 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:36.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:36.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:17:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:17:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:17:36.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.778 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:36.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:36.879 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:36.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:36.915 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:36.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:37.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.048 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:37.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:37.142 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:36 smithi118 ceph-mon[33862]: pgmap v686: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 966 op/s; 34456/208701 objects degraded (16.510%); 6/208701 objects misplaced (0.003%) 2024-03-21T14:17:37.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:37.187 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:37.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 76596 76580 132.937 77.8125 0.00576997 0.00751805 2024-03-21T14:17:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:36 smithi003 ceph-mon[29576]: pgmap v686: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 966 op/s; 34456/208701 objects degraded (16.510%); 6/208701 objects misplaced (0.003%) 2024-03-21T14:17:37.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:36 smithi003 ceph-mon[40510]: pgmap v686: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 60 MiB/s wr, 966 op/s; 34456/208701 objects degraded (16.510%); 6/208701 objects misplaced (0.003%) 2024-03-21T14:17:37.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:37.314 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:37.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:37.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:17:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:17:37.451 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:37.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:37.576 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:37.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:37.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:37.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:37.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:37.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:37.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:37.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:37.931 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:17:37.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:38.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4103151960' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:38.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4103151960' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:38.106 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:38.143 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:38.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4103151960' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:38.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 77923 77907 131.585 82.9375 0.00798303 0.00759564 2024-03-21T14:17:38.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:38.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.277 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:38.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:38.378 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:38.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:38.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:38.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.550 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:38.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:38.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.689 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:38.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:38.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:38.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:38.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:38.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:38.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[29576]: pgmap v687: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4103151960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[29576]: osdmap e331: 8 total, 7 up, 7 in 2024-03-21T14:17:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[40510]: pgmap v687: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4103151960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:38 smithi003 ceph-mon[40510]: osdmap e331: 8 total, 7 up, 7 in 2024-03-21T14:17:39.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:39.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:39.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:38 smithi118 ceph-mon[33862]: pgmap v687: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:38 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:39.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:38 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4103151960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:39.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:38 smithi118 ceph-mon[33862]: osdmap e331: 8 total, 7 up, 7 in 2024-03-21T14:17:39.195 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:39.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 79243 79227 130.294 82.5 0.0206028 0.00767048 2024-03-21T14:17:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.225 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:39.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:39.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:39.361 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:39.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:39.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:39.490 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:39.592 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:39.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:39.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:39.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:39.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:39.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:39.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:39.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:39.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:39.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.034 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:40.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:40.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:17:40.170 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:17:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:39 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 37809/228735 objects degraded (16.530%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:39 smithi118 ceph-mon[33862]: osdmap e332: 8 total, 7 up, 7 in 2024-03-21T14:17:40.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 80505 80489 128.975 78.875 0.00711484 0.00774957 2024-03-21T14:17:40.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:39 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 37809/228735 objects degraded (16.530%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:40.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:39 smithi003 ceph-mon[29576]: osdmap e332: 8 total, 7 up, 7 in 2024-03-21T14:17:40.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:39 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 37809/228735 objects degraded (16.530%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:40.250 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:39 smithi003 ceph-mon[40510]: osdmap e332: 8 total, 7 up, 7 in 2024-03-21T14:17:40.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.307 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:40.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:40.341 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:17:40.388 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:17:40.392 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:17:40.392 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:17:40.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:40.425 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:17:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:40.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:40.498 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:17:40.541 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:17:40.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:40.580 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:17:40.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:40.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:40.615 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:17:40.615 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:17:40.615 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:17:40.615 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:17:40.616 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:17:40.616 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:17:40.616 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:17:40.616 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:17:40.616 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:17:40.692 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:40.725 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:40.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:40.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:40.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:40.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:40.950 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:17:40.950 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:17:40.950 INFO:tasks.thrashosds.thrasher:Killing osd 6, live_osds are [0, 1, 3, 4, 5, 6, 7] 2024-03-21T14:17:40.950 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T14:17:40.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6 2024-03-21T14:17:40.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:41.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:41.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:41.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.150 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:41.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:40 smithi118 ceph-mon[33862]: pgmap v689: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.45k op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:40 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1188500184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:41.182 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:41 smithi118 systemd[1]: Stopping Ceph osd.6 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:17:41.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:17:41.206328+0000 min lat: 0.00112315 max lat: 0.209043 avg lat: 0.00782605 2024-03-21T14:17:41.205 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:17:41.206 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 81760 81744 127.711 78.4375 0.0042599 0.00782605 2024-03-21T14:17:41.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:41.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:40 smithi003 ceph-mon[40510]: pgmap v689: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.45k op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:40 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1188500184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:41.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:40 smithi003 ceph-mon[29576]: pgmap v689: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.45k op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:41.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:40 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1188500184' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:41.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:41.296 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:17:41.296 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:17:41.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:41.425 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:41.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:41.527 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:41.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:41.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.703 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:41.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:41.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:17:41.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:41.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:41.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:41.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:41.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:41.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:42.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:42.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:42.168 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:42.210 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 83207 83191 126.786 90.4375 0.00952686 0.00788298 2024-03-21T14:17:42.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:42.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:42.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:42.409 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:42.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.547 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:42.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:42.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:42.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:42.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:42.820 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:42.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:42.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:42.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:43.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:43.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.092 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:43.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:42 smithi118 ceph-mon[33862]: pgmap v691: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:43.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:43.210 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 15 84566 84551 125.791 85 0.00884744 0.00794524 2024-03-21T14:17:43.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:42 smithi003 ceph-mon[29576]: pgmap v691: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:42 smithi003 ceph-mon[40510]: pgmap v691: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 4.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s; 37809/228735 objects degraded (16.530%); 6/228735 objects misplaced (0.003%) 2024-03-21T14:17:43.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:17:43.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:43.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:43.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:43.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:43.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:43.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:43.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:43.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:43.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:43.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:43.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:43.998 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:44.031 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:44.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:44.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:44.164 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:44.211 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 85804 85788 124.664 77.3125 0.00358271 0.00801756 2024-03-21T14:17:44.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:44.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:44.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:44.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:44.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:44.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:44.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:44.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:44.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:44.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:44.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:44.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:44.843 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:44.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:44.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:44.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:44 smithi003 ceph-mon[29576]: pgmap v692: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 41126/248718 objects degraded (16.535%); 6/248718 objects misplaced (0.002%) 2024-03-21T14:17:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:44 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 41126/248718 objects degraded (16.535%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:44 smithi003 ceph-mon[40510]: pgmap v692: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 41126/248718 objects degraded (16.535%); 6/248718 objects misplaced (0.002%) 2024-03-21T14:17:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:44 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 41126/248718 objects degraded (16.535%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:45.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:45.211 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 87112 87096 123.688 81.75 0.00877912 0.00808026 2024-03-21T14:17:45.233 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:44 smithi118 ceph-mon[33862]: pgmap v692: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 104 MiB/s wr, 1.67k op/s; 41126/248718 objects degraded (16.535%); 6/248718 objects misplaced (0.002%) 2024-03-21T14:17:45.233 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:44 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 41126/248718 objects degraded (16.535%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:45.360 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:45.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:45.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:45.503 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:45.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:45.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:45.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:45.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:45.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:45.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:17:45.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:45.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:45.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:45.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:45.936 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:45.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:46.038 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:46.076 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:46.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:46.177 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:46.211 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 88557 88541 122.946 90.3125 0.00741003 0.00812902 2024-03-21T14:17:46.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:46.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:46.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:46.354 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:46.403 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:46 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:46 smithi118 ceph-mon[33862]: pgmap v693: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s; 41126/248718 objects degraded (16.535%); 6/248718 objects misplaced (0.002%) 2024-03-21T14:17:46.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:46.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:46.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:46.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-mon[29576]: pgmap v693: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s; 41126/248718 objects degraded (16.535%); 6/248718 objects misplaced (0.002%) 2024-03-21T14:17:46.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:46.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:46.477 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-mon[40510]: pgmap v693: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s; 41126/248718 objects degraded (16.535%); 6/248718 objects misplaced (0.002%) 2024-03-21T14:17:46.498 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:46.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:46.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:46.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:17:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:17:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:17:46.776 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:46.878 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:46.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:46.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:46.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[50523]: 2024-03-21T14:17:46.471+0000 7f6c63012700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:17:46.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[50523]: 2024-03-21T14:17:46.471+0000 7f6c63012700 -1 osd.6 332 *** Got signal Terminated *** 2024-03-21T14:17:46.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[50523]: 2024-03-21T14:17:46.471+0000 7f6c63012700 -1 osd.6 332 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:17:47.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:47.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:47.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:47.187 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:47.211 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 90017 90001 122.256 91.25 0.0156088 0.0081754 2024-03-21T14:17:47.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:17:47.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:17:47.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[29576]: osd.6 marked itself down and dead 2024-03-21T14:17:47.286 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:47.286 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:17:47.287 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:17:47.287 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[40510]: osd.6 marked itself down and dead 2024-03-21T14:17:47.287 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:47.288 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:47.326 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:47.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:47.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:17:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:17:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:47 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:17:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:47 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:17:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:47 smithi118 ceph-mon[33862]: osd.6 marked itself down and dead 2024-03-21T14:17:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:17:47.465 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:47.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:47.578 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:17:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.618 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:47.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.764 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:47.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:47.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:47.905 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:47.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:48.006 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:48.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.063 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:48.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:48.130 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:17:48.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:48.211 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 15 91416 91401 121.517 87.5 0.0140607 0.00822488 2024-03-21T14:17:48.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:48.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:48.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:48.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.356 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:48.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:48 smithi118 ceph-mon[33862]: osdmap e333: 8 total, 6 up, 7 in 2024-03-21T14:17:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:48 smithi118 ceph-mon[33862]: pgmap v695: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3491328285' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:17:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:48 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:17:48.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:48.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[29576]: osdmap e333: 8 total, 6 up, 7 in 2024-03-21T14:17:48.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:48.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[29576]: pgmap v695: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:48.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3491328285' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:17:48.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:17:48.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[40510]: osdmap e333: 8 total, 6 up, 7 in 2024-03-21T14:17:48.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:48.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[40510]: pgmap v695: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.69k op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:48.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3491328285' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:17:48.467 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:48 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:17:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:48.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:48.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:48.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:48.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:48.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:48.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:48.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:48.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:48.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:48.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:49.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:49.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:49.116 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:48 smithi118 podman[127319]: 2024-03-21 14:17:48.822481046 +0000 UTC m=+3.011427819 container died 8b5ed5375f440164918d74874c4d1e7e55d823d44974403ceb84bf682e8c20a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5) 2024-03-21T14:17:49.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:49.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:49.211 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 92896 92880 120.911 92.4375 0.0104547 0.0082664 2024-03-21T14:17:49.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:49.342 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:49.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:49.418 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[29576]: osdmap e334: 8 total, 6 up, 7 in 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[40510]: osdmap e334: 8 total, 6 up, 7 in 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:17:49.419 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:49 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:17:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:49 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:17:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:49 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:17:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:49 smithi118 ceph-mon[33862]: osdmap e334: 8 total, 6 up, 7 in 2024-03-21T14:17:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:49 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:17:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:49 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:17:49.431 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:49 smithi118 podman[127319]: 2024-03-21 14:17:49.340144288 +0000 UTC m=+3.529091055 container cleanup 8b5ed5375f440164918d74874c4d1e7e55d823d44974403ceb84bf682e8c20a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:17:49.431 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:49 smithi118 bash[127319]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6 2024-03-21T14:17:49.444 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:49.481 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:49.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:49.501 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:17:49.538 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:17:49.567 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:17:49.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:49.605 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:17:49.621 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:49.643 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:17:49.681 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:17:49.682 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:17:49.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:49.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:49.857 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:49.896 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:49.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:49.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:50.029 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:50.130 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:49 smithi118 podman[127606]: 2024-03-21 14:17:49.94617759 +0000 UTC m=+2.214020480 container remove 8b5ed5375f440164918d74874c4d1e7e55d823d44974403ceb84bf682e8c20a0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T14:17:50.131 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:50.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:50.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 16 94398 94382 120.359 93.875 0.0101602 0.00830449 2024-03-21T14:17:50.262 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:50.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:50.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:50.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:50.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.427 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:50.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: osdmap e335: 8 total, 6 up, 7 in 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1767109640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: pgmap v698: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 44580/269166 objects degraded (16.562%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:50 smithi118 ceph-mon[33862]: osdmap e336: 8 total, 6 up, 7 in 2024-03-21T14:17:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: osdmap e335: 8 total, 6 up, 7 in 2024-03-21T14:17:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:17:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:17:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1767109640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: pgmap v698: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 44580/269166 objects degraded (16.562%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[29576]: osdmap e336: 8 total, 6 up, 7 in 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: osdmap e335: 8 total, 6 up, 7 in 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1767109640' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: pgmap v698: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.14k op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 44580/269166 objects degraded (16.562%), 22 pgs degraded, 53 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:50 smithi003 ceph-mon[40510]: osdmap e336: 8 total, 6 up, 7 in 2024-03-21T14:17:50.529 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:50.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:50.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:50.666 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:50.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:50.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:50.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:50.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:50.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:50.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:17:50.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:17:50.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:50.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:51.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:51.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:51.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:51.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 95960 95944 119.904 97.625 0.0160003 0.00833555 2024-03-21T14:17:51.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:51.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:17:51.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:17:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:51 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:17:51.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:51.396 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:51 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:17:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:51 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:17:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:51 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:17:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:51 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:17:51.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.536 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:51.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:51.612 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:51.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:51.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:51.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:51.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:51.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:52.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:52.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:52.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:52.203 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:52.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:52.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 97419 97403 119.341 91.1875 0.00617126 0.00837414 2024-03-21T14:17:52.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:52.345 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:52 smithi118 ceph-mon[33862]: pgmap v700: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 30 MiB/s wr, 477 op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:52.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:52.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:52.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:52.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:52 smithi003 ceph-mon[29576]: pgmap v700: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 30 MiB/s wr, 477 op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:52.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:52 smithi003 ceph-mon[40510]: pgmap v700: 158 pgs: 4 active+clean+remapped, 3 peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 30 MiB/s wr, 477 op/s; 44580/269166 objects degraded (16.562%); 6/269166 objects misplaced (0.002%) 2024-03-21T14:17:52.583 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:52.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:52.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:52.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:52.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:52.854 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:52.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:52.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:52.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:17:52.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:53.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:53.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:53.140 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:17:53.140 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:17:53.171 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:53.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 98898 98882 118.823 92.4375 0.0089384 0.00841168 2024-03-21T14:17:53.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:53.311 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:53.413 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.442 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:53.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:53.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.590 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:53.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:53.667 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:17:53.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:53.725 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:53.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:53.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:53.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:53.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:53.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:54.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:54.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:54.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:54.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.160 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:54.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:54.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 100356 100340 118.301 91.125 0.00469494 0.00844884 2024-03-21T14:17:54.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:54.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:54.395 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:54.430 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:54.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:54.534 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:17:54.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:54.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:54.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:54.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:17:54.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: osdmap e337: 8 total, 6 up, 7 in 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: pgmap v702: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 940 B/s rd, 86 MiB/s wr, 1.37k op/s; 48236/291510 objects degraded (16.547%) 2024-03-21T14:17:54.784 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/507117970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:54.785 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:54 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:54.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:54.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:54.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:54.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: osdmap e337: 8 total, 6 up, 7 in 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: pgmap v702: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 940 B/s rd, 86 MiB/s wr, 1.37k op/s; 48236/291510 objects degraded (16.547%) 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/507117970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:54.986 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:54.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:17:54.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:17:54.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:17:54.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:17:54.987 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: osdmap e337: 8 total, 6 up, 7 in 2024-03-21T14:17:54.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: pgmap v702: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 940 B/s rd, 86 MiB/s wr, 1.37k op/s; 48236/291510 objects degraded (16.547%) 2024-03-21T14:17:54.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/507117970' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:54.988 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:54 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:17:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.007 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:55.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:55.101 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:54 smithi118 podman[128362]: 2024-03-21 14:17:54.792551908 +0000 UTC m=+0.039711122 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:17:55.101 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:54 smithi118 podman[128362]: 2024-03-21 14:17:54.924491369 +0000 UTC m=+0.171650547 container create 91e09b86dea7ddb52cb4017ba4dbfd00dc16c67badb3a00bc4744538556557da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:17:55.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:55.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.158 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:55.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:55.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 101745 101729 117.718 86.8125 0.00617836 0.0084907 2024-03-21T14:17:55.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:55.299 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:55.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:55.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:55.432 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:55 smithi118 podman[128362]: 2024-03-21 14:17:55.223386017 +0000 UTC m=+0.470545212 container init 91e09b86dea7ddb52cb4017ba4dbfd00dc16c67badb3a00bc4744538556557da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:17:55.432 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:55 smithi118 podman[128362]: 2024-03-21 14:17:55.228637342 +0000 UTC m=+0.475796528 container start 91e09b86dea7ddb52cb4017ba4dbfd00dc16c67badb3a00bc4744538556557da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5) 2024-03-21T14:17:55.432 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:55 smithi118 podman[128362]: 2024-03-21 14:17:55.320047505 +0000 UTC m=+0.567206705 container attach 91e09b86dea7ddb52cb4017ba4dbfd00dc16c67badb3a00bc4744538556557da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:17:55.452 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:55.554 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:55.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:55.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:55.686 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.723 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:55.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:17:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[29576]: osdmap e338: 8 total, 6 up, 7 in 2024-03-21T14:17:55.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 48236/291510 objects degraded (16.547%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:55.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:17:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[40510]: osdmap e338: 8 total, 6 up, 7 in 2024-03-21T14:17:55.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:55 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 48236/291510 objects degraded (16.547%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:17:55.814 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:55 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:17:55.814 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:55 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:17:55.814 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:55 smithi118 ceph-mon[33862]: osdmap e338: 8 total, 6 up, 7 in 2024-03-21T14:17:55.814 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:55 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 48236/291510 objects degraded (16.547%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:17:55.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:55.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:55.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:55.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:55.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:56.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:56.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:56.136 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:56.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 103187 103171 117.216 90.125 0.00419745 0.00852682 2024-03-21T14:17:56.238 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.278 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:56.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:56.379 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:56.416 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:56.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:56.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:56.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.700 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:56.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:17:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:17:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:17:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:17:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: pgmap v704: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s; 48236/291510 objects degraded (16.547%) 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:17:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: pgmap v704: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s; 48236/291510 objects degraded (16.547%) 2024-03-21T14:17:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:56.756 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:56 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:56.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:56.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:56.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:56.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:17:56.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:17:56.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:17:56.929 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:17:56.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: pgmap v704: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.24k op/s; 48236/291510 objects degraded (16.547%) 2024-03-21T14:17:56.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:56.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:56 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:56.930 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:17:56 smithi118 podman[128362]: 2024-03-21 14:17:56.607850918 +0000 UTC m=+1.855010096 container died 91e09b86dea7ddb52cb4017ba4dbfd00dc16c67badb3a00bc4744538556557da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-21T14:17:56.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:57.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:57.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:57.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:57.140 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:57.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 104634 104618 116.738 90.4375 0.0173812 0.00856207 2024-03-21T14:17:57.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:57.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:57.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:57.345 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:17:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:17:57.378 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:57.390 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T14:17:57.415 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:57.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:57.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:17:57.516 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T14:17:57.516 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.6 2024-03-21T14:17:57.517 INFO:tasks.thrashosds.thrasher:remote for osd 6 is ubuntu@smithi118.front.sepia.ceph.com 2024-03-21T14:17:57.517 INFO:tasks.thrashosds.thrasher:remote for exp osd 2 is ubuntu@smithi003.front.sepia.ceph.com 2024-03-21T14:17:57.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-21T14:17:57.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:57.519 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:17:57.558 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:17:57.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:57.599 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:17:57.639 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:17:57.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:57.677 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:17:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:57 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:17:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:57 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:17:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:57 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:17:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:57 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:17:57.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3793921763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:57.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:57.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:57.723 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:17:57.724 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:17:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3793921763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:17:57.725 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:17:57.726 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:17:57.726 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:17:57.726 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3793921763' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:17:57.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:57.851 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:57.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:57.961 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:17:57.990 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:57.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:58.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:58.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:58.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:58.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:58.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 106135 106119 116.335 93.8125 0.012212 0.00859141 2024-03-21T14:17:58.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:58.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:58.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:58.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:58.407 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:58.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:58.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:58.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:58.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:58.578 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:58.578 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:58.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:17:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:17:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:17:58.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:17:58.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: 4.15 deep-scrub starts 2024-03-21T14:17:58.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: 4.15 deep-scrub ok 2024-03-21T14:17:58.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: pgmap v705: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:17:58.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:17:58.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:17:58.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:58 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:58.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:17:58.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:58.822 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:58.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:17:58.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191648 8.18 8:1e671e02:::benchmark_data_smithi003_110242_object60731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 378.14751574600001, 2024-03-21T14:17:58.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99582953799999996, 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191648 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.901 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749048+0000", 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6000000000000003e-07 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749052+0000", 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.544e-06 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749074+0000", 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1254999999999999e-05 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749082+0000", 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5469999999999997e-06 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749377+0000", 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029512199999999998 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749386+0000", 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8370000000000003e-06 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739054+0000", 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98966760600000003 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.905 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739210+0000", 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015671000000000001 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739479+0000", 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026850799999999998 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744724+0000", 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052452610000000002 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744795+0000", 2024-03-21T14:17:58.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0326000000000004e-05 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744804+0000", 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3680000000000008e-06 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744841+0000", 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7134000000000001e-05 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744877+0000", 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5559999999999998e-05 2024-03-21T14:17:58.909 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191685 8.18 8:1d8c9114:::benchmark_data_smithi003_110242_object60768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.779057+0000", 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 378.117505768, 2024-03-21T14:17:58.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96600729399999996, 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191685 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779057+0000", 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779057+0000", 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779058+0000", 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.347e-06 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779064+0000", 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0799999999999996e-06 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779074+0000", 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.004e-05 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779080+0000", 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8920000000000003e-06 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779153+0000", 2024-03-21T14:17:58.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2942999999999996e-05 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.779161+0000", 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5959999999999998e-06 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.741379+0000", 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96221795600000004 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.741493+0000", 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000113732 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.741693+0000", 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020050499999999999 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744899+0000", 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003205449 2024-03-21T14:17:58.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744973+0000", 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3985000000000004e-05 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744995+0000", 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1997999999999998e-05 2024-03-21T14:17:58.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745027+0000", 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.1952999999999997e-05 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: 4.15 deep-scrub starts 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: 4.15 deep-scrub ok 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: pgmap v705: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:17:58.922 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: 4.15 deep-scrub starts 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: 4.15 deep-scrub ok 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: pgmap v705: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:17:58.923 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:58 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.745064+0000", 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7818000000000003e-05 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219191 8.5 8:a034a08e:::benchmark_data_smithi003_110242_object88274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.36814893399998, 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0133928029999999, 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.924 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219191 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:17:58.925 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528415+0000", 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3300000000000001e-07 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528419+0000", 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0219999999999998e-06 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.926 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528427+0000", 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5499999999999995e-06 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528435+0000", 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.199e-06 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528488+0000", 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3267999999999999e-05 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.927 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528497+0000", 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6379999999999999e-06 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536200+0000", 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.007702536 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536291+0000", 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0702e-05 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.928 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536577+0000", 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028596900000000001 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539703+0000", 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003126396 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541643+0000", 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001940083 2024-03-21T14:17:58.929 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541733+0000", 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0246000000000003e-05 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541768+0000", 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4150000000000003e-05 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541807+0000", 2024-03-21T14:17:58.930 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9211000000000001e-05 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219208 8.5 8:a331d64d:::benchmark_data_smithi003_110242_object88291:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.553819+0000", 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.34274423800002, 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98820481199999999, 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.931 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219208 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553819+0000", 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553819+0000", 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.932 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553820+0000", 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1069999999999999e-06 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553824+0000", 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1500000000000001e-06 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553834+0000", 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9469999999999998e-06 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.933 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553839+0000", 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3709999999999999e-06 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553884+0000", 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4468000000000002e-05 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.553889+0000", 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5829999999999997e-06 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.934 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.538272+0000", 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98438244100000005 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.538358+0000", 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6626999999999996e-05 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.538604+0000", 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024539899999999999 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.935 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541850+0000", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003246709 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541865+0000", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4748e-05 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541933+0000", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8108999999999995e-05 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.936 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541986+0000", 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3062999999999997e-05 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542023+0000", 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7089999999999999e-05 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219209 8.5 8:a65be5ed:::benchmark_data_smithi003_110242_object88292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.554777+0000", 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.34178627900002, 2024-03-21T14:17:58.937 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98745134800000001, 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219209 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554777+0000", 2024-03-21T14:17:58.938 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554777+0000", 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554778+0000", 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.187e-06 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.939 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554783+0000", 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3619999999999997e-06 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554807+0000", 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4091999999999999e-05 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554816+0000", 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9590000000000001e-06 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.940 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554874+0000", 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7794000000000003e-05 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.554884+0000", 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0231000000000001e-05 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539749+0000", 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98486504200000002 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.941 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539862+0000", 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000112456 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.540109+0000", 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024730100000000001 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541873+0000", 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001764034 2024-03-21T14:17:58.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542094+0000", 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000220984 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542143+0000", 2024-03-21T14:17:58.943 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9013e-05 2024-03-21T14:17:58.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.944 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.944 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.944 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542191+0000", 2024-03-21T14:17:58.944 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7596000000000003e-05 2024-03-21T14:17:58.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.945 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.945 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.945 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542228+0000", 2024-03-21T14:17:58.945 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7296999999999999e-05 2024-03-21T14:17:58.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.945 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.946 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219252 8.5 8:a50faeae:::benchmark_data_smithi003_110242_object88335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.946 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:17:58.947 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.308398893, 2024-03-21T14:17:58.947 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96401971099999995, 2024-03-21T14:17:58.947 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.947 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.947 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.947 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.948 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.948 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219252 2024-03-21T14:17:58.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.948 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.948 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.948 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.949 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:17:58.949 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.949 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.949 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.949 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:17:58.950 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.950 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.950 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.950 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588164+0000", 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2399999999999998e-07 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588167+0000", 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.756e-06 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.951 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588177+0000", 2024-03-21T14:17:58.952 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9049999999999992e-06 2024-03-21T14:17:58.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.954 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.954 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.954 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588183+0000", 2024-03-21T14:17:58.954 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2430000000000002e-06 2024-03-21T14:17:58.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588301+0000", 2024-03-21T14:17:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011804299999999999 2024-03-21T14:17:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.956 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.956 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.956 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.588309+0000", 2024-03-21T14:17:58.956 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7999999999999999e-06 2024-03-21T14:17:58.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.957 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.957 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542276+0000", 2024-03-21T14:17:58.957 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.95396667899999998 2024-03-21T14:17:58.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.958 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.958 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.958 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.542404+0000", 2024-03-21T14:17:58.958 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128181 2024-03-21T14:17:58.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.959 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.959 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:58.959 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.543001+0000", 2024-03-21T14:17:58.959 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00059724599999999995 2024-03-21T14:17:58.959 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.960 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.960 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.960 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547671+0000", 2024-03-21T14:17:58.960 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0046693409999999996 2024-03-21T14:17:58.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.961 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.961 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547685+0000", 2024-03-21T14:17:58.961 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4728e-05 2024-03-21T14:17:58.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.961 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.962 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552079+0000", 2024-03-21T14:17:58.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043938019999999996 2024-03-21T14:17:58.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.962 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552135+0000", 2024-03-21T14:17:58.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5318000000000001e-05 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552184+0000", 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9144999999999998e-05 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.964 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.965 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.965 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219264 8.5 8:a6e0eab4:::benchmark_data_smithi003_110242_object88347:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.965 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.599939+0000", 2024-03-21T14:17:58.965 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.29662375999999, 2024-03-21T14:17:58.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.96534715100000001, 2024-03-21T14:17:58.966 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.966 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.966 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219264 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599939+0000", 2024-03-21T14:17:58.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599939+0000", 2024-03-21T14:17:58.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599940+0000", 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.279e-06 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599946+0000", 2024-03-21T14:17:58.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6489999999999996e-06 2024-03-21T14:17:58.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599958+0000", 2024-03-21T14:17:58.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2094e-05 2024-03-21T14:17:58.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.599969+0000", 2024-03-21T14:17:58.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1185999999999999e-05 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.600013+0000", 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3992000000000002e-05 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.600019+0000", 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7969999999999997e-06 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.545425+0000", 2024-03-21T14:17:58.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94540629399999998 2024-03-21T14:17:58.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.974 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.974 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.545546+0000", 2024-03-21T14:17:58.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120964 2024-03-21T14:17:58.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:58.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.546053+0000", 2024-03-21T14:17:58.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050637399999999997 2024-03-21T14:17:58.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.551970+0000", 2024-03-21T14:17:58.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005916915 2024-03-21T14:17:58.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.977 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552203+0000", 2024-03-21T14:17:58.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000233699 2024-03-21T14:17:58.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.977 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565192+0000", 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012988474999999999 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565244+0000", 2024-03-21T14:17:58.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2469000000000002e-05 2024-03-21T14:17:58.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565286+0000", 2024-03-21T14:17:58.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1964e-05 2024-03-21T14:17:58.979 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.980 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.980 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.980 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219331 8.5 8:a2b12185:::benchmark_data_smithi003_110242_object88414:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.980 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.659873+0000", 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.23668991599999, 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90553297399999999, 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.981 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219331 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659873+0000", 2024-03-21T14:17:58.982 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659873+0000", 2024-03-21T14:17:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:58.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659874+0000", 2024-03-21T14:17:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.291e-06 2024-03-21T14:17:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.984 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659881+0000", 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3690000000000003e-06 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:58.985 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659892+0000", 2024-03-21T14:17:58.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1008e-05 2024-03-21T14:17:58.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.986 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.986 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:58.986 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659898+0000", 2024-03-21T14:17:58.986 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.528e-06 2024-03-21T14:17:58.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.987 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.987 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.987 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659971+0000", 2024-03-21T14:17:58.987 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3299000000000004e-05 2024-03-21T14:17:58.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.988 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:58.988 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.659981+0000", 2024-03-21T14:17:58.988 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8670000000000006e-06 2024-03-21T14:17:58.988 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547730+0000", 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88774864799999997 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:58.989 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.547857+0000", 2024-03-21T14:17:58.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012742799999999999 2024-03-21T14:17:58.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.990 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.990 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:58.990 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.548159+0000", 2024-03-21T14:17:58.990 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000301685 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552037+0000", 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038776459999999998 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.991 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:58.992 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552211+0000", 2024-03-21T14:17:58.992 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017444100000000001 2024-03-21T14:17:58.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.992 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.992 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:58.993 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565328+0000", 2024-03-21T14:17:58.993 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.013117070999999999 2024-03-21T14:17:58.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.993 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:17:58.996 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:58.996 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565372+0000", 2024-03-21T14:17:58.996 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3955000000000002e-05 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565406+0000", 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3738000000000001e-05 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.997 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:58.998 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:58.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:58.998 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:58.998 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219333 8.5 8:a1fc9cb9:::benchmark_data_smithi003_110242_object88416:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:17:58.998 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.660701+0000", 2024-03-21T14:17:58.998 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 349.23586212200001, 2024-03-21T14:17:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90488013300000003, 2024-03-21T14:17:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:17:58.999 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219333 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660701+0000", 2024-03-21T14:17:59.000 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.001 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.001 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.001 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660701+0000", 2024-03-21T14:17:59.001 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.002 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.002 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660702+0000", 2024-03-21T14:17:59.002 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4900000000000005e-07 2024-03-21T14:17:59.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.002 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.003 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.003 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660706+0000", 2024-03-21T14:17:59.003 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6280000000000004e-06 2024-03-21T14:17:59.003 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.003 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660718+0000", 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2159e-05 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.004 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660727+0000", 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3410000000000003e-06 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660782+0000", 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4920999999999998e-05 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.660790+0000", 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6889999999999993e-06 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.005 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.549933+0000", 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88914224600000002 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.550057+0000", 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012411099999999999 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.550306+0000", 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024982500000000001 2024-03-21T14:17:59.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.552257+0000", 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001950984 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.561349+0000", 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090918709999999996 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565442+0000", 2024-03-21T14:17:59.007 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040930489999999996 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565531+0000", 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8644e-05 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.565581+0000", 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9815999999999999e-05 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.008 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24350.0:2962 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94450053602048 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:58.649678+0000", 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 120.246884353, 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9659854800000001, 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24350", 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2962 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.009 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649678+0000", 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649678+0000", 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649679+0000", 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6400000000000002e-07 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.010 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649685+0000", 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4090000000000002e-06 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649693+0000", 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6959999999999994e-06 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649701+0000", 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4109999999999999e-06 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.011 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649762+0000", 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1153000000000002e-05 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649771+0000", 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6660000000000003e-06 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.596536+0000", 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94676457700000005 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.012 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.596548+0000", 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1653e-05 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606082+0000", 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0095339270000001 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606255+0000", 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017383599999999999 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.013 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.615664+0000", 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094084879999999996 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24350.0:2963 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94450053602048 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:59.599358+0000", 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 119.29720475400001, 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0164483479999999, 2024-03-21T14:17:59.014 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24350", 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2963 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599358+0000", 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.015 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599358+0000", 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599360+0000", 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.575e-06 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599367+0000", 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8530000000000003e-06 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.016 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599378+0000", 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1440000000000001e-05 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599387+0000", 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.636e-06 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599503+0000", 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116882 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.017 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599515+0000", 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1060000000000001e-05 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606611+0000", 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0070960019999999 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606687+0000", 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6530000000000001e-05 2024-03-21T14:17:59.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.615806+0000", 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091193699999999999 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:83392 10.e 10:74c3883b:::benchmark_data_smithi003_160672_object83391:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:42.336714+0000", 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.559849067000002, 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12482601, 2024-03-21T14:17:59.019 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83392 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.336714+0000", 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.020 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.336714+0000", 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.336714+0000", 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5300000000000004e-07 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.336745+0000", 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0970999999999999e-05 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.021 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.336752+0000", 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3899999999999998e-06 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.336757+0000", 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5659999999999997e-06 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.337650+0000", 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00089317200000000002 2024-03-21T14:17:59.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.337770+0000", 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000119793 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5", 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.337902+0000", 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000132278 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342970+0000", 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0050678420000000004 2024-03-21T14:17:59.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.459398+0000", 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.116427339 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.461437+0000", 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020392990000000001 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.461540+0000", 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010280699999999999 2024-03-21T14:17:59.024 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:83397 10.6 10:699a97d9:::benchmark_data_smithi003_160672_object83396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:42.340513+0000", 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.556049883, 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13103346099999999, 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83397 2024-03-21T14:17:59.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340513+0000", 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340513+0000", 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.026 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340514+0000", 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1209999999999999e-06 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340551+0000", 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7101000000000003e-05 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340560+0000", 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.755e-06 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340567+0000", 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.951e-06 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.340760+0000", 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019316799999999999 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.341023+0000", 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026288800000000002 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.341161+0000", 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013781199999999999 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.459418+0000", 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.118256904 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.459459+0000", 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1553e-05 2024-03-21T14:17:59.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471489+0000", 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012029556 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471519+0000", 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0394000000000001e-05 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471546+0000", 2024-03-21T14:17:59.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7257999999999999e-05 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:83400 10.e 10:7a37c008:::benchmark_data_smithi003_160672_object83399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:42.342431+0000", 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.554131562999999, 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.129186936, 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.031 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83400 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342431+0000", 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342431+0000", 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342432+0000", 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7199999999999997e-07 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342464+0000", 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2098999999999999e-05 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342472+0000", 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5480000000000004e-06 2024-03-21T14:17:59.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342478+0000", 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.913e-06 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.343006+0000", 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052863499999999996 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.343142+0000", 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135258 2024-03-21T14:17:59.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5", 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.343286+0000", 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000144559 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.466286+0000", 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.122999945 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471551+0000", 2024-03-21T14:17:59.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052651989999999999 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471575+0000", 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3961e-05 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471618+0000", 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2846999999999998e-05 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:83404 10.e 10:7ccdeddd:::benchmark_data_smithi003_160672_object83403:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:42.342540+0000", 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.554022722999999, 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12922154799999999, 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83404 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342540+0000", 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342540+0000", 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342541+0000", 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2199999999999999e-07 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342568+0000", 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.792e-05 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342574+0000", 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8289999999999996e-06 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.342579+0000", 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7910000000000004e-06 2024-03-21T14:17:59.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.461580+0000", 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.119000825 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.461780+0000", 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020003299999999999 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5", 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.461958+0000", 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017845099999999999 2024-03-21T14:17:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471479+0000", 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0095204739999999993 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471624+0000", 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000144835 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471687+0000", 2024-03-21T14:17:59.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3662000000000001e-05 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471762+0000", 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4306000000000003e-05 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:83407 10.e 10:7def857b:::benchmark_data_smithi003_160672_object83406:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:42.344264+0000", 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.552298376, 2024-03-21T14:17:59.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.135739362, 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 83407 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344264+0000", 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344264+0000", 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344265+0000", 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.009e-06 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344322+0000", 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6937000000000001e-05 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344333+0000", 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0302000000000001e-05 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344339+0000", 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7680000000000003e-06 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.464033+0000", 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.11969312999999999 2024-03-21T14:17:59.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.464160+0000", 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012750399999999999 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5", 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.464321+0000", 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016105199999999999 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.471783+0000", 2024-03-21T14:17:59.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0074623210000000001 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.479840+0000", 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080569419999999992 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.479896+0000", 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5194e-05 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.480004+0000", 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010820299999999999 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15666.0:83406 10.b e332/317 10:d8af4cdc:::benchmark_data_smithi003_160672_object83405:head v 332'5280, mlcod=332'5278)", 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:42.344948+0000", 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 16.551615103, 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.121478526, 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344948+0000", 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344948+0000", 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.344948+0000", 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1200000000000002e-07 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.345005+0000", 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6650000000000001e-05 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.345027+0000", 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2033999999999999e-05 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.345037+0000", 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0152e-05 2024-03-21T14:17:59.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.345089+0000", 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1616000000000002e-05 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.345109+0000", 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.035e-05 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.466381+0000", 2024-03-21T14:17:59.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12127155100000001 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:42.466426+0000", 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5460999999999997e-05 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:87023 10.1 10:8503bf2a:::benchmark_data_smithi003_160672_object87022:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:45.049307+0000", 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.847255593, 2024-03-21T14:17:59.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13462961400000001, 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 87023 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049307+0000", 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049307+0000", 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049308+0000", 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1299999999999999e-07 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049350+0000", 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2478000000000002e-05 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049359+0000", 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3480000000000004e-06 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049366+0000", 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7909999999999999e-06 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049645+0000", 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00027962199999999999 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.049857+0000", 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021160600000000001 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.050017+0000", 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000160416 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.052095+0000", 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020781720000000001 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.052647+0000", 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00055164500000000002 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.183813+0000", 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.13116581399999999 2024-03-21T14:17:59.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.183884+0000", 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0992000000000001e-05 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.183937+0000", 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3016999999999997e-05 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:87040 10.1 10:865d7e48:::benchmark_data_smithi003_160672_object87039:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:45.057926+0000", 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 13.838637164, 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12613774, 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 87040 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.057926+0000", 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.057926+0000", 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.057926+0000", 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1500000000000004e-07 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.057951+0000", 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4258999999999999e-05 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.057957+0000", 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3380000000000001e-06 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.057962+0000", 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8210000000000001e-06 2024-03-21T14:17:59.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.058575+0000", 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00061294300000000002 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.058693+0000", 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000117749 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.058825+0000", 2024-03-21T14:17:59.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013229100000000001 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.060761+0000", 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001936145 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.062543+0000", 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001781728 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.183975+0000", 2024-03-21T14:17:59.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.12143230000000001 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.184028+0000", 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3316999999999998e-05 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:45.184063+0000", 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5133999999999999e-05 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24350.0:3182 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94450053602048 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:47.116552+0000", 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 11.780011197, 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0192424769999999, 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24350", 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3182 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:17:59.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116552+0000", 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116552+0000", 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116553+0000", 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.136e-06 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116558+0000", 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.164e-06 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116572+0000", 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3713000000000001e-05 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116580+0000", 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.123e-06 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.117294+0000", 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000713855 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.117300+0000", 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8260000000000004e-06 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.125149+0000", 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078487069999999999 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.125158+0000", 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9199999999999993e-06 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:48.132480+0000", 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.00732208 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:48.132614+0000", 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134101 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:17:59.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:48.135794+0000", 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031798519999999999 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:17:59.070 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:17:59.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:59.108 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:59.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:17:59.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:59.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 15 107582 107567 115.89 90.5 0.00997793 0.00862471 2024-03-21T14:17:59.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:59.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:59.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:59.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:17:59.478 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:17:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:59.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:17:59.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:17:59.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:59.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:59.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:17:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:17:59.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:17:59.716 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:59.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:59.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:59.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:17:59.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:17:59.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:17:59.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:17:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:59 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:17:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:59 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:17:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:59 smithi118 ceph-mon[33862]: pgmap v706: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.5 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:17:59.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:17:59 smithi118 ceph-mon[33862]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T14:17:59.984 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:17:59.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:17:59.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:17:59.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[29576]: pgmap v706: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.5 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:17:59.985 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[29576]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T14:17:59.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:17:59.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:17:59.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[40510]: pgmap v706: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.5 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:17:59.986 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:17:59 smithi003 ceph-mon[40510]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-21T14:18:00.010 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:00.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:00.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:00.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:18:00.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:00.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:00.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 109000 108984 115.426 88.5625 0.0128592 0.00865932 2024-03-21T14:18:00.256 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:00.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:00.390 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:00.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:00.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:00.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:00.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:00.569 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:00.579 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:18:00.618 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:00.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:00.690 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:00.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:00.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:00.848 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:00 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:18:00.848 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:00 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:18:00.848 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:00 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 51788/313047 objects degraded (16.543%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:00.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:00 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:18:00.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:00 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:18:00.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:00 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 51788/313047 objects degraded (16.543%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:00.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:00 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:18:00.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:00 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:18:00.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:00 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 51788/313047 objects degraded (16.543%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:00.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:00.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:00.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:00.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:01.095 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:01.133 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:01.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:18:01.213730+0000 min lat: 0.00112315 max lat: 0.209043 avg lat: 0.00868867 2024-03-21T14:18:01.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:18:01.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 110472 110456 115.036 92 0.0151334 0.00868867 2024-03-21T14:18:01.234 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.272 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:01.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:01.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:01.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:01.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:01.473 INFO:teuthology.orchestra.run.smithi003.stdout:9.b 2024-03-21T14:18:01.473 INFO:teuthology.orchestra.run.smithi003.stdout:9.7 2024-03-21T14:18:01.473 INFO:teuthology.orchestra.run.smithi003.stdout:9.2 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:9.0 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.9 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.5 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.18 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.17 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.12 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.11 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:6.0 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:5.a 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:3.4 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:5.b 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:5.14 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:3.1a 2024-03-21T14:18:01.474 INFO:teuthology.orchestra.run.smithi003.stdout:5.1c 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:3.13 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:1.1 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:5.8 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:4.19 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:3.1f 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:6.8 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:1.9 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:4.13 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:6.1 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:1.5 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:3.1e 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:2.1 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:1.0 2024-03-21T14:18:01.475 INFO:teuthology.orchestra.run.smithi003.stdout:3.18 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:6.1b 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:1.8 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:1.7 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:4.1e 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:4.d 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:5.6 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:3.0 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:1.a 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:3.5 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:4.12 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:6.f 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:5.2 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:5.d 2024-03-21T14:18:01.476 INFO:teuthology.orchestra.run.smithi003.stdout:3.14 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:4.17 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:6.19 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:6.14 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:5.4 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:4.18 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:4.1c 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:6.1c 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:4.4 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:4.6 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:3.9 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:4.b 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:9.6 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:6.3 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:5.18 2024-03-21T14:18:01.477 INFO:teuthology.orchestra.run.smithi003.stdout:5.1e 2024-03-21T14:18:01.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:01.548 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:01.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:18:01.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:01.679 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:01.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:01 smithi003 ceph-mon[29576]: pgmap v707: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 740 B/s rd, 65 MiB/s wr, 1.04k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:18:01.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:01 smithi003 ceph-mon[40510]: pgmap v707: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 740 B/s rd, 65 MiB/s wr, 1.04k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:18:01.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:01.816 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:01.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:01.918 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:01.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:01 smithi118 ceph-mon[33862]: pgmap v707: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 740 B/s rd, 65 MiB/s wr, 1.04k op/s; 51788/313047 objects degraded (16.543%) 2024-03-21T14:18:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:01.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:01.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:02.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:02.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:02.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:02.212 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 15 111992 111977 114.708 95.0625 0.013902 0.0087135 2024-03-21T14:18:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:02.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:02.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:01 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:18:01.962+0000 7fa17996d700 -1 mgr get_metadata_python Requested missing service osd.6 2024-03-21T14:18:02.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:02.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:02.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:02.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:02.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:02.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:02.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:02.676 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:02 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:18:02.676 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:02 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:18:02.676 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:02 smithi118 ceph-mon[33862]: 5.1a deep-scrub starts 2024-03-21T14:18:02.676 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:02 smithi118 ceph-mon[33862]: 5.1a deep-scrub ok 2024-03-21T14:18:02.676 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:18:02.676 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:02.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:18:02.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:18:02.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[40510]: 5.1a deep-scrub starts 2024-03-21T14:18:02.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[40510]: 5.1a deep-scrub ok 2024-03-21T14:18:02.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:18:02.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:02.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:18:02.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:18:02.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[29576]: 5.1a deep-scrub starts 2024-03-21T14:18:02.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[29576]: 5.1a deep-scrub ok 2024-03-21T14:18:02.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:18:02.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:02.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:02.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.811 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:02.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15666.0:113040 10.e e338/317 10:7483e923:::benchmark_data_smithi003_160672_object113039:head v 338'7021, mlcod=338'7019)", 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:02.884714+0000", 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.002607023, 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026381540000000002, 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:02.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.884714+0000", 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.884714+0000", 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.884715+0000", 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2900000000000003e-07 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.884738+0000", 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2997e-05 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.884751+0000", 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2602999999999999e-05 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.884758+0000", 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885000+0000", 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000242031 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885076+0000", 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6038999999999995e-05 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:02.891 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop(client.15666.0:113035 10.9 e338/317 10:9df194a8:::benchmark_data_smithi003_160672_object113034:head v 338'6997, mlcod=338'6994)", 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:02.885653+0000", 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.001668215, 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001774058, 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885653+0000", 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885653+0000", 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885654+0000", 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4300000000000001e-07 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885680+0000", 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6258000000000002e-05 2024-03-21T14:18:02.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885691+0000", 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0958000000000001e-05 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885699+0000", 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3989999999999998e-06 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:18:02.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885774+0000", 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5618999999999995e-05 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:02.885795+0000", 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0681000000000001e-05 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 2 2024-03-21T14:18:02.895 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:18:02.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:02.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:02.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:02.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:03.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.078 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:03.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:03.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.13 --file /var/log/ceph/exp.4.13.2 2024-03-21T14:18:03.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:03.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 16 113543 113527 114.421 96.875 0.00775131 0.00873578 2024-03-21T14:18:03.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.215 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:03.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:03.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:03.355 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:03.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:03.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:03.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:03.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:03.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.626 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:03.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:03.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:03.740 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:18:03.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:18:03.839 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:03.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:03.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:03.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:04.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.075 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:04.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:04.166 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:04.176 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:04.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:04.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 114992 114976 114.042 90.5625 0.00876972 0.00876477 2024-03-21T14:18:04.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:03 smithi003 ceph-mon[29576]: pgmap v708: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 577 B/s rd, 93 MiB/s wr, 1.49k op/s; 55490/331095 objects degraded (16.760%) 2024-03-21T14:18:04.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:03 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:18:04.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:03 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:18:04.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:03 smithi003 ceph-mon[40510]: pgmap v708: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 577 B/s rd, 93 MiB/s wr, 1.49k op/s; 55490/331095 objects degraded (16.760%) 2024-03-21T14:18:04.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:03 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:18:04.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:03 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:18:04.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:04.310 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:03 smithi118 ceph-mon[33862]: pgmap v708: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.7 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 577 B/s rd, 93 MiB/s wr, 1.49k op/s; 55490/331095 objects degraded (16.760%) 2024-03-21T14:18:04.310 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:03 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:18:04.310 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:03 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:18:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.350 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:04.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:04.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:04.482 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:04.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:04.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:04.713 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:04.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:04.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:04.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:04.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:04.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:04.977 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:18:04.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:05.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:05.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.173 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:05.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/33728643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:04 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:18:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:04 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:18:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:04 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 55490/331095 objects degraded (16.760%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:04 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:18:05.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 116516 116500 113.748 95.25 0.00677069 0.00878693 2024-03-21T14:18:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/33728643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:18:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:18:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 55490/331095 objects degraded (16.760%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:18:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/33728643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:18:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:18:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 55490/331095 objects degraded (16.760%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:04 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:18:05.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:05.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.307 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:05.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:05.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:05.440 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:05.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:05.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:05.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:05.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:05.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:05.722 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:05.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:05.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:05.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:05.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:05.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:05.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:06.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:06.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:06.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:06.144 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:18:06.144 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:18:06.145 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:18:06.145 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:18:06.145 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:18:06.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 118008 117992 113.433 93.25 0.0134845 0.008812 2024-03-21T14:18:06.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:18:06.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:18:06.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[29576]: pgmap v709: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 55490/335187 objects degraded (16.555%) 2024-03-21T14:18:06.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:06.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/33728643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:18:06.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[29576]: osdmap e339: 8 total, 6 up, 7 in 2024-03-21T14:18:06.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:18:06.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:18:06.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[40510]: pgmap v709: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 55490/335187 objects degraded (16.555%) 2024-03-21T14:18:06.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:06.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/33728643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:18:06.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:05 smithi003 ceph-mon[40510]: osdmap e339: 8 total, 6 up, 7 in 2024-03-21T14:18:06.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:06.239 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:18:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.266 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:06.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:06.367 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:06.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:06.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:05 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:05 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:05 smithi118 ceph-mon[33862]: pgmap v709: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s; 55490/335187 objects degraded (16.555%) 2024-03-21T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:05 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:05 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/33728643' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:18:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:05 smithi118 ceph-mon[33862]: osdmap e339: 8 total, 6 up, 7 in 2024-03-21T14:18:06.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:06.528 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:06.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:06.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:18:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:18:06.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:06.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:06.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:06.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:06.948 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:06.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:07.049 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.091 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:07.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:07.192 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:07.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 119655 119639 113.274 102.938 0.0140539 0.00882398 2024-03-21T14:18:07.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.231 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:07.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:07.298 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:18:07.332 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:07.390 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:07.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:18:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:18:07.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:07.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:07.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:07.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:07.718 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:07.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:07.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:07.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:07.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:07.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:07.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:08.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:08.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:08.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:07 smithi118 ceph-mon[33862]: pgmap v711: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 577 B/s rd, 52 MiB/s wr, 832 op/s; 55490/335187 objects degraded (16.555%) 2024-03-21T14:18:08.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:08.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 121056 121040 112.89 87.5625 0.0117357 0.00885385 2024-03-21T14:18:08.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:08.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:08.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:07 smithi003 ceph-mon[29576]: pgmap v711: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 577 B/s rd, 52 MiB/s wr, 832 op/s; 55490/335187 objects degraded (16.555%) 2024-03-21T14:18:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:07 smithi003 ceph-mon[40510]: pgmap v711: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 6.8 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 577 B/s rd, 52 MiB/s wr, 832 op/s; 55490/335187 objects degraded (16.555%) 2024-03-21T14:18:08.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.359 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:08.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:08.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:08.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:08.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.628 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:08.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:08.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:08.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:08.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:08.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:08.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:08.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:08.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:18:09.006 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:09.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:09.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:09.151 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:09.180 INFO:teuthology.orchestra.run.smithi003.stderr:Exporting 4.13 info 4.13( v 236'24 (0'0,236'24] local-lis/les=74/75 n=5 ec=66/66 lis/c=74/74 les/c/f=75/75/0 sis=74) 2024-03-21T14:18:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:09.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.188 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:09.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:09.199 INFO:teuthology.orchestra.run.smithi003.stderr:Read #4:c8333aab:gc::gc.11:head# 2024-03-21T14:18:09.199 INFO:teuthology.orchestra.run.smithi003.stderr:Read #4:cba150de:::obj_delete_at_hint.0000000051:head# 2024-03-21T14:18:09.199 INFO:teuthology.orchestra.run.smithi003.stderr:Read #4:cc909b6b:::obj_delete_at_hint.0000000106:head# 2024-03-21T14:18:09.200 INFO:teuthology.orchestra.run.smithi003.stderr:Read #4:cd178ae0:reshard::reshard.0000000014:head# 2024-03-21T14:18:09.200 INFO:teuthology.orchestra.run.smithi003.stderr:Read #4:cedbb205:::obj_delete_at_hint.0000000017:head# 2024-03-21T14:18:09.200 INFO:teuthology.orchestra.run.smithi003.stderr:Export successful 2024-03-21T14:18:09.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:09.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:09.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 15 122514 122499 112.571 91.1875 0.00685795 0.00887941 2024-03-21T14:18:09.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:09.317 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:18:09.325 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:09.354 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:18:09.393 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:18:09.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:09.434 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:18:09.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.461 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:09.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:09.482 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:09.505 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:18:09.543 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:18:09.544 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:18:09.544 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:18:09.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:09.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:09.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:09.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.743 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:09.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:09.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:09.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:09.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:09.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:09.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:10.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:10.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:10.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:10.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 123952 123936 112.241 89.8125 0.00939935 0.00890516 2024-03-21T14:18:10.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[29576]: pgmap v712: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s; 59368/358026 objects degraded (16.582%) 2024-03-21T14:18:10.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:18:10.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:18:10.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3405900856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 59368/358026 objects degraded (16.582%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[29576]: osdmap e340: 8 total, 6 up, 7 in 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[40510]: pgmap v712: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s; 59368/358026 objects degraded (16.582%) 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3405900856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 59368/358026 objects degraded (16.582%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:10.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:09 smithi003 ceph-mon[40510]: osdmap e340: 8 total, 6 up, 7 in 2024-03-21T14:18:10.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.291 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:10.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:10.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:10.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:09 smithi118 ceph-mon[33862]: pgmap v712: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1023 B/s rd, 94 MiB/s wr, 1.50k op/s; 59368/358026 objects degraded (16.582%) 2024-03-21T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:09 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:09 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:09 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3405900856' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:09 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 59368/358026 objects degraded (16.582%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:10.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:09 smithi118 ceph-mon[33862]: osdmap e340: 8 total, 6 up, 7 in 2024-03-21T14:18:10.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:10.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.556 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:10.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:10.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.692 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:10.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:18:10.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:18:10.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:10.829 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:10.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:10.966 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:10.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:11.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:11.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:11.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:11.206 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:11.213 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 125266 125250 111.81 82.125 0.00907629 0.00893961 2024-03-21T14:18:11.243 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:11.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:11.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:11.477 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:11.517 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:11.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.654 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:11.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:11.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:11.792 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:11.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:11.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:11.928 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:11.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:12.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:12.070 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:12.172 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:12.208 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:12.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 126830 126814 111.612 97.75 0.00643403 0.00895517 2024-03-21T14:18:12.245 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:11 smithi003 ceph-mon[29576]: pgmap v714: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 59368/358026 objects degraded (16.582%) 2024-03-21T14:18:12.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:11 smithi003 ceph-mon[40510]: pgmap v714: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 59368/358026 objects degraded (16.582%) 2024-03-21T14:18:12.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:12.336 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:12.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:11 smithi118 ceph-mon[33862]: pgmap v714: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.3 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 59368/358026 objects degraded (16.582%) 2024-03-21T14:18:12.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:12.475 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:12.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:12.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:12.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:12.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:12.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:18:12.738 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:12.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.13 2024-03-21T14:18:12.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:12.876 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:12.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:12.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:13.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:13.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:13.136 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:13.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 128025 128009 111.099 74.6875 0.00951015 0.00899706 2024-03-21T14:18:13.238 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.273 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:13.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:13.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:13.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:13.517 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.543 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:13.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:13.625 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:18:13.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.681 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:13.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:13.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.810 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:13.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:13.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:13.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:13.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:13.990 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:18:13.994 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:13 smithi118 ceph-mon[33862]: pgmap v715: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.68k op/s; 62934/375609 objects degraded (16.755%) 2024-03-21T14:18:13.994 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4197733720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:14.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:14.091 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:14.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:14.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:14.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 129281 129265 110.653 78.5 0.00676998 0.00903308 2024-03-21T14:18:14.229 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:13 smithi003 ceph-mon[29576]: pgmap v715: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.68k op/s; 62934/375609 objects degraded (16.755%) 2024-03-21T14:18:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4197733720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:13 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:14.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:13 smithi003 ceph-mon[40510]: pgmap v715: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.6 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.68k op/s; 62934/375609 objects degraded (16.755%) 2024-03-21T14:18:14.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4197733720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:14.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:13 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:14.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:14.366 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:14.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:14.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:13 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:14.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:14.505 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:14.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:14.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:14.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:14.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:14.742 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:14.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:14.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:14.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:14.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:14.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:15.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:15.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:15.146 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:15.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:15.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 130581 130565 110.255 81.25 0.0059357 0.00906591 2024-03-21T14:18:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[29576]: osdmap e341: 8 total, 6 up, 7 in 2024-03-21T14:18:15.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 62934/375609 objects degraded (16.755%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:15.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[29576]: osdmap e342: 8 total, 6 up, 7 in 2024-03-21T14:18:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[40510]: osdmap e341: 8 total, 6 up, 7 in 2024-03-21T14:18:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 62934/375609 objects degraded (16.755%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:15.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:14 smithi003 ceph-mon[40510]: osdmap e342: 8 total, 6 up, 7 in 2024-03-21T14:18:15.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:15.313 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:15.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:15.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:14 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:15 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:15 smithi118 ceph-mon[33862]: osdmap e341: 8 total, 6 up, 7 in 2024-03-21T14:18:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:15 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 62934/375609 objects degraded (16.755%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:15 smithi118 ceph-mon[33862]: osdmap e342: 8 total, 6 up, 7 in 2024-03-21T14:18:15.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:15.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.576 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:15.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:15.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:15.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:15.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:15.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:15.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:15.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:15.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:16.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:16.126 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:16.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 131747 131731 109.757 72.875 0.0166376 0.00910721 2024-03-21T14:18:16.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:16.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:15 smithi003 ceph-mon[29576]: pgmap v717: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 62934/379710 objects degraded (16.574%) 2024-03-21T14:18:16.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:15 smithi003 ceph-mon[40510]: pgmap v717: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 62934/379710 objects degraded (16.574%) 2024-03-21T14:18:16.263 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:16.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:16.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:16.400 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:16.407 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:18:16.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:15 smithi118 ceph-mon[33862]: pgmap v717: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 116 MiB/s wr, 1.85k op/s; 62934/379710 objects degraded (16.574%) 2024-03-21T14:18:16.456 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:16.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:16.527 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:16.605 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:18:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:18:16.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:16.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:16.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:16.803 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:16.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:16.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:18:16.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:16.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:16.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:17.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:17.094 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:17.111 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:18:17.010+0000 7fa17996d700 -1 mgr get_metadata_python Requested missing service osd.6 2024-03-21T14:18:17.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:17.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 133226 133210 109.529 92.4375 0.00816361 0.00912608 2024-03-21T14:18:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.236 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:17.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:17.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:18:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:18:17.337 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.370 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:17.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:17.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:17.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:17.548 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:18:17.548 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:18:17.609 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:17.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:17.670 INFO:teuthology.orchestra.run.smithi003.stdout: marking collection for removal 2024-03-21T14:18:17.670 INFO:teuthology.orchestra.run.smithi003.stdout:setting '_remove' omap key 2024-03-21T14:18:17.670 INFO:teuthology.orchestra.run.smithi003.stdout:finish_remove_pgs 4.13_head removing 4.13 2024-03-21T14:18:17.670 INFO:teuthology.orchestra.run.smithi003.stdout:Remove successful 2024-03-21T14:18:17.747 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:17.782 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:17.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:17.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:17.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:18.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:18.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:18.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:18.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:18.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 134696 134680 109.3 91.875 0.0157037 0.00914426 2024-03-21T14:18:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-mon[29576]: pgmap v719: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 752 B/s rd, 66 MiB/s wr, 1.06k op/s; 62934/379710 objects degraded (16.574%) 2024-03-21T14:18:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:18:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:18.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-mon[40510]: pgmap v719: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 752 B/s rd, 66 MiB/s wr, 1.06k op/s; 62934/379710 objects degraded (16.574%) 2024-03-21T14:18:18.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:18:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:18.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:18.298 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:17 smithi118 ceph-mon[33862]: pgmap v719: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 752 B/s rd, 66 MiB/s wr, 1.06k op/s; 62934/379710 objects degraded (16.574%) 2024-03-21T14:18:18.298 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:18:18.298 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.342 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:18.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:18.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:18:18.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:18.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:18.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:18.609 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:18.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:18.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:18.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:18.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:18.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:18.910 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:18.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:19.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:19.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:19.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:19.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:19.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 136261 136245 109.152 97.8125 0.00995148 0.00915767 2024-03-21T14:18:19.227 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:19.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:19.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:19.281 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:19.316 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:19.418 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:19.455 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:19.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:19.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:19.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:19.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:19.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:19.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:19.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:19.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:19.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.6 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-6 --journal-path /var/lib/ceph/osd/ceph-6/journal --op list-pgs 2024-03-21T14:18:19.870 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:19.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:19.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:20.030 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:20.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:20.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:20.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:20.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:20.191 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.6/config 2024-03-21T14:18:20.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 137802 137786 108.99 96.3125 0.00981771 0.00917119 2024-03-21T14:18:20.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:19 smithi003 ceph-mon[29576]: pgmap v720: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 66126/398865 objects degraded (16.579%) 2024-03-21T14:18:20.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:19 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 66126/398865 objects degraded (16.579%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:20.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:19 smithi003 ceph-mon[40510]: pgmap v720: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 66126/398865 objects degraded (16.579%) 2024-03-21T14:18:20.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:19 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 66126/398865 objects degraded (16.579%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:20.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:20.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:20.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:20.350 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:19 smithi118 ceph-mon[33862]: pgmap v720: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 66126/398865 objects degraded (16.579%) 2024-03-21T14:18:20.350 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:19 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 66126/398865 objects degraded (16.579%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:20.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:20.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:20.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:20.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:20.719 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:20.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:20.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:20.852 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:20.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:20.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:20.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:20.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:20.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:21.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:21.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:21.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:18:21.215485+0000 min lat: 0.00112315 max lat: 0.235562 avg lat: 0.00918511 2024-03-21T14:18:21.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:18:21.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 139337 139321 108.826 95.9375 0.00403088 0.00918511 2024-03-21T14:18:21.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:21.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:21.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:21.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:21.395 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:18:21.419 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:21.427 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:18:21.467 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:18:21.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:21.505 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:18:21.526 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:21.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:21.539 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:18:21.573 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:18:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:18:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:18:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:18:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:18:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:18:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:18:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:18:21.574 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:18:21.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:21.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:21.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:21.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.799 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:21.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:21.900 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:21.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:21.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:22.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:22.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:22.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:21 smithi118 ceph-mon[33862]: pgmap v721: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 969 op/s; 66126/398865 objects degraded (16.579%) 2024-03-21T14:18:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:21 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2079083871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:22.206 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:21 smithi003 ceph-mon[29576]: pgmap v721: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 969 op/s; 66126/398865 objects degraded (16.579%) 2024-03-21T14:18:22.206 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2079083871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:22.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:21 smithi003 ceph-mon[40510]: pgmap v721: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.1 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 969 op/s; 66126/398865 objects degraded (16.579%) 2024-03-21T14:18:22.206 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:21 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2079083871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:22.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:22.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:22.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 140735 140719 108.561 87.375 0.0108047 0.00920735 2024-03-21T14:18:22.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:22.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.339 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:22.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:22.440 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:22.475 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:22.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:22.577 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:22.605 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:22.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:22.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:22.847 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:22.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:22.882 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:22.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:22.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:23.030 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:23.038 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:18:23.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:18:23.131 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:23.167 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:23.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:23.214 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 142066 142050 108.252 83.1875 0.00995369 0.00923399 2024-03-21T14:18:23.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:23.311 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:23.413 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.455 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:23.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:23.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:23.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:23.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:23.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:23.731 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:23.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:23.874 INFO:teuthology.orchestra.run.smithi118.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:18:23.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:23.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:23.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:142382 10.c 10:361b5fc3:::benchmark_data_smithi003_160672_object142381:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:23.437503+0000", 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0082303770000000005, 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0082687420000000008, 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 142382 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437503+0000", 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437503+0000", 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437505+0000", 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9700000000000002e-06 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437544+0000", 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8859999999999997e-05 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437558+0000", 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3471e-05 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437568+0000", 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0033e-05 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437752+0000", 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018409599999999999 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.437902+0000", 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014986200000000001 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,4", 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.438108+0000", 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020606199999999999 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_repop_reply(client.15666.0:142382 10.c e343/317)", 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:23.440662+0000", 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0050712470000000001, 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051863389999999999, 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.440662+0000", 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.440662+0000", 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.440663+0000", 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4399999999999998e-07 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.440671+0000", 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1849999999999998e-06 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.440682+0000", 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1717e-05 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.440690+0000", 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9680000000000006e-06 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:142387 10.9 10:9bbf2792:::benchmark_data_smithi003_160672_object142386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:23.443439+0000", 2024-03-21T14:18:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0022948199999999999, 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002462971, 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 142387 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443439+0000", 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443439+0000", 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443441+0000", 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7400000000000001e-06 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443490+0000", 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9635999999999997e-05 2024-03-21T14:18:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443510+0000", 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9619999999999998e-05 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443520+0000", 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0145e-05 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443696+0000", 2024-03-21T14:18:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017614099999999999 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443865+0000", 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016883499999999999 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5", 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.444297+0000", 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00043219899999999997 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:142389 10.c 10.e7ff24ac (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:23.443546+0000", 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0021879289999999999, 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024320399999999999, 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 142389 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443546+0000", 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443546+0000", 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443546+0000", 2024-03-21T14:18:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9699999999999996e-07 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443588+0000", 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1239999999999998e-05 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443599+0000", 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1460999999999999e-05 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443607+0000", 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:142393 10.8 10:10a3b756:::benchmark_data_smithi003_160672_object142392:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:23.443618+0000", 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.0021156970000000002, 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024249549999999999, 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:18:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 142393 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443618+0000", 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443618+0000", 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443618+0000", 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3000000000000001e-07 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443652+0000", 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3065000000000003e-05 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443662+0000", 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0390999999999999e-05 2024-03-21T14:18:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443668+0000", 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3069999999999999e-06 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.443903+0000", 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000234749 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.444023+0000", 2024-03-21T14:18:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120123 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0", 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:23.444360+0000", 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033640999999999999 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 5 2024-03-21T14:18:23.981 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:18:23.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:24.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:24.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:24.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:24.154 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:24 smithi118 ceph-mon[33862]: pgmap v722: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.5 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 605 B/s rd, 93 MiB/s wr, 1.49k op/s; 69964/417486 objects degraded (16.758%) 2024-03-21T14:18:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:24 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:18:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:24 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:18:24.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:24 smithi118 ceph-mon[33862]: osdmap e343: 8 total, 6 up, 7 in 2024-03-21T14:18:24.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 143583 143567 108.09 94.8125 0.02748 0.0092473 2024-03-21T14:18:24.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[29576]: pgmap v722: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.5 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 605 B/s rd, 93 MiB/s wr, 1.49k op/s; 69964/417486 objects degraded (16.758%) 2024-03-21T14:18:24.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:18:24.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:18:24.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[29576]: osdmap e343: 8 total, 6 up, 7 in 2024-03-21T14:18:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[40510]: pgmap v722: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.5 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 605 B/s rd, 93 MiB/s wr, 1.49k op/s; 69964/417486 objects degraded (16.758%) 2024-03-21T14:18:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:18:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:18:24.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:24 smithi003 ceph-mon[40510]: osdmap e343: 8 total, 6 up, 7 in 2024-03-21T14:18:24.257 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:24.292 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:24.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:24.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:24.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:24.567 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:24.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:24.649 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:18:24.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:24.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:24.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:24.766 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:18:24.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:24.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:24.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:24.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:24.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:24.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:25.016 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:25 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2935939513' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:18:25.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:18:25.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:25.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:25.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:25.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 145028 145012 107.878 90.3125 0.00281679 0.00926201 2024-03-21T14:18:25.241 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:25.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:25.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:25.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:25 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 69964/417486 objects degraded (16.758%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:25 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:25 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2935939513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:18:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:25 smithi118 ceph-mon[33862]: osdmap e344: 8 total, 6 up, 7 in 2024-03-21T14:18:25.451 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2935939513' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 69964/417486 objects degraded (16.758%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2935939513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[29576]: osdmap e344: 8 total, 6 up, 7 in 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2935939513' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 69964/417486 objects degraded (16.758%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2935939513' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:18:25.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:25 smithi003 ceph-mon[40510]: osdmap e344: 8 total, 6 up, 7 in 2024-03-21T14:18:25.481 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:25.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:25.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:25.571 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:25.618 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:25.647 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:25.743 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:25.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:25.799 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:25.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:25.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:25.952 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:25.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:26.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:26.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:26 smithi118 ceph-mon[33862]: pgmap v724: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 69964/421620 objects degraded (16.594%) 2024-03-21T14:18:26.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:26 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:18:26.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:26 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:18:26.185 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:26.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 146388 146372 107.609 85 0.00524841 0.00928724 2024-03-21T14:18:26.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.231 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:26.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-mon[29576]: pgmap v724: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 69964/421620 objects degraded (16.594%) 2024-03-21T14:18:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:18:26.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:18:26.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-mon[40510]: pgmap v724: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 69964/421620 objects degraded (16.594%) 2024-03-21T14:18:26.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:18:26.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:18:26.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:26.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:26.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:26.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.505 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:26.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:26.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:26.637 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:26.739 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:26.751 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:18:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:18:26.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.781 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:26.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:26.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:26.914 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:26.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:26.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:27.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:27.061 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:27.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:27 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:18:27.196 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:27.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 147776 147760 107.366 86.75 0.0120518 0.00930976 2024-03-21T14:18:27.298 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:27.381 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:27.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:27.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:27 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:18:27.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:27 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:18:27.452 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:18:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:18:27.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:27.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:18:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:27.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:27.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:27.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:27.748 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:27.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:27.796 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:18:27.796 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:18:27.797 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:18:27.797 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:18:27.797 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:18:27.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:27.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:27.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:27.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:28.035 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:28.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:28 smithi118 ceph-mon[33862]: pgmap v726: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 59 MiB/s wr, 948 op/s; 69964/421620 objects degraded (16.594%) 2024-03-21T14:18:28.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:28 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:18:28.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:28 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:18:28.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:28 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:18:28.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:28 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:18:28.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:28 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:18:28.081 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:28.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:28.098 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:18:28.132 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:18:28.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[29576]: pgmap v726: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 59 MiB/s wr, 948 op/s; 69964/421620 objects degraded (16.594%) 2024-03-21T14:18:28.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:18:28.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:18:28.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:18:28.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:18:28.171 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:18:28.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:18:28.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[40510]: pgmap v726: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.6 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 59 MiB/s wr, 948 op/s; 69964/421620 objects degraded (16.594%) 2024-03-21T14:18:28.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:18:28.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:18:28.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:18:28.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:18:28.172 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:28 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:18:28.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:28.204 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:18:28.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 15 149073 149058 107.065 81.125 0.0072484 0.0093363 2024-03-21T14:18:28.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:28.241 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:18:28.279 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:18:28.279 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:18:28.280 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:18:28.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:28.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:28.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:28.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:28.503 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:28.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:18:28.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:28.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.668 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:28.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:28.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:28.826 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:28.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:28.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:28.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:28.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:29.012 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:29.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:29.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:29.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:18:29.202 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:18:29.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 150404 150388 106.793 83.125 0.0225227 0.00935967 2024-03-21T14:18:29.224 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:29.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:29.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:29 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:29.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1046903129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:29.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:29 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:18:29.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:29 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:18:29.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:29.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1046903129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:29.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:18:29.341 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:18:29.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:29.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1046903129' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:29.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:18:29.342 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:29 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:18:29.373 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:29.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:6.e 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:4.13 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:3.10 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:5.1f 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:5.f 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:6.0 2024-03-21T14:18:29.389 INFO:teuthology.orchestra.run.smithi118.stdout:6.d 2024-03-21T14:18:29.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:29.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:29.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:29.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:29.645 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:29.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:29.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.790 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:29.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:29.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:29.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:30.035 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:30.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:30.169 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:30 smithi118 ceph-mon[33862]: pgmap v727: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.0 GiB data, 29 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 73502/442935 objects degraded (16.594%) 2024-03-21T14:18:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:30 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:18:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:30 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:18:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:30 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 73502/442935 objects degraded (16.594%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:30 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:18:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:30 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:18:30.206 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:30.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 151931 151915 106.665 95.4375 0.0063554 0.00937145 2024-03-21T14:18:30.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[29576]: pgmap v727: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.0 GiB data, 29 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 73502/442935 objects degraded (16.594%) 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 73502/442935 objects degraded (16.594%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[40510]: pgmap v727: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.0 GiB data, 29 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 73502/442935 objects degraded (16.594%) 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:18:30.241 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:18:30.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 73502/442935 objects degraded (16.594%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:30.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:18:30.242 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:30 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:18:30.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:30.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:30.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:30.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:30.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:30.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.638 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:30.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:30.739 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:30.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:30.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:30.911 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:30.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:31.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:31.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:31.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:31.150 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:31.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:31.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:18:31.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.0076 2024-03-21T14:18:31.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 153373 2024-03-21T14:18:31.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-03-21T14:18:31.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-03-21T14:18:31.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 106.5 2024-03-21T14:18:31.215 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 35.9606 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 244.25 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 72.5625 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1704 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 575.37 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 3908 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 1161 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.00938604 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.00825179 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 0.235562 2024-03-21T14:18:31.216 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00112315 2024-03-21T14:18:31.218 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T14:18:31.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:31.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:31.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:31.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:31.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:31 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:18:31.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:31 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:18:31.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:31.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:31.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:31 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:18:31.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:31 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:18:31.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:31 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:18:31.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:31 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:18:31.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:31.583 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:31.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell -n osd.2 --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.13.2 2024-03-21T14:18:31.632 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:31.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:31.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:31.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:31.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:31.917 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:31.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:32.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:32.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:32.078 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:18:32.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:32.178 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:18:32.178 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:18:32.178 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:32 smithi118 ceph-mon[33862]: pgmap v728: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.0 GiB data, 29 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 73502/442935 objects degraded (16.594%) 2024-03-21T14:18:32.179 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1577803952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:32.179 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:32 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:32.179 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:32.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:32.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:32.254 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:18:32.028+0000 7fa17996d700 -1 mgr get_metadata_python Requested missing service osd.6 2024-03-21T14:18:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[29576]: pgmap v728: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.0 GiB data, 29 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 73502/442935 objects degraded (16.594%) 2024-03-21T14:18:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1577803952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:32.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[40510]: pgmap v728: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.0 GiB data, 29 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 73502/442935 objects degraded (16.594%) 2024-03-21T14:18:32.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1577803952' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:32.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:32.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:32.306 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:32.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:32.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:32.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:32.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:32.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:32.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:32.632 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/osd.2/config 2024-03-21T14:18:32.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:32.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.761 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:32.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:32.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:18:32.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:32.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:32.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:33.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:33.034 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:33.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:33.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:33 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:18:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:33 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:33 smithi003 ceph-mon[40510]: osdmap e345: 8 total, 6 up, 7 in 2024-03-21T14:18:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:33 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:18:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:33 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:33 smithi003 ceph-mon[29576]: osdmap e345: 8 total, 6 up, 7 in 2024-03-21T14:18:33.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:33.297 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:33.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:33 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:18:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:33 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:33.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:33 smithi118 ceph-mon[33862]: osdmap e345: 8 total, 6 up, 7 in 2024-03-21T14:18:33.438 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:33.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.573 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:33.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:33.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:33.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:33.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:33.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:33.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:33.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:33.970 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:34.005 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:34.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:34.139 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:34.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.277 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:34.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:34.378 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:34.415 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:34.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: pgmap v730: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 767 B/s rd, 94 MiB/s wr, 1.61k op/s; 75867/455103 objects degraded (16.670%) 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: osdmap e346: 8 total, 6 up, 7 in 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:18:34.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:34 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: pgmap v730: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 767 B/s rd, 94 MiB/s wr, 1.61k op/s; 75867/455103 objects degraded (16.670%) 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: osdmap e346: 8 total, 6 up, 7 in 2024-03-21T14:18:34.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: pgmap v730: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 767 B/s rd, 94 MiB/s wr, 1.61k op/s; 75867/455103 objects degraded (16.670%) 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: osdmap e346: 8 total, 6 up, 7 in 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:18:34.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:34 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:18:34.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:34.551 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:34.653 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:34.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:34.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:34.822 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:34.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:34.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:34.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:34.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:35.062 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:35.095 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:35.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:35.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:35.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:35.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:35.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:35.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:18:35.347 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[29576]: pgmap v732: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.4 KiB/s rd, 102 MiB/s wr, 1.79k op/s; 75867/456936 objects degraded (16.603%) 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 75867/455103 objects degraded (16.670%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[40510]: pgmap v732: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.4 KiB/s rd, 102 MiB/s wr, 1.79k op/s; 75867/456936 objects degraded (16.603%) 2024-03-21T14:18:35.348 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:35 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 75867/455103 objects degraded (16.670%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:35.370 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:35.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:35 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:18:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:35 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:18:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:35 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:18:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:35 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:18:35.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:35 smithi118 ceph-mon[33862]: pgmap v732: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.4 KiB/s rd, 102 MiB/s wr, 1.79k op/s; 75867/456936 objects degraded (16.603%) 2024-03-21T14:18:35.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:35 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 75867/455103 objects degraded (16.670%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:35.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:35.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:35.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:35.611 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:35.644 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:35.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:35.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:35.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:35.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:18:35.878 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:35.916 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:35.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:36.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:36.034 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:36.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:36.145 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:18:36.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:18:36.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:18:36.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:18:36.146 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:18:36.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:18:36.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:18:36.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:18:36.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:18:36.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:36.188 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:18:36.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:36.227 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:18:36.266 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:18:36.291 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:36.307 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:18:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:36.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:36.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:18:36.347 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:18:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:36 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:18:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:36 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:18:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:36 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:18:36.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:36 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:18:36.435 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:36.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:36.568 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:36.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:36.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:36.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:18:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:18:36.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:36.879 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:36.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:36.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:36.981 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.016 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:37.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:37.117 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:37.152 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:37.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:37.254 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:37.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:37.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:37.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1406731337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:37.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:37 smithi003 ceph-mon[29576]: pgmap v733: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 767 B/s rd, 46 MiB/s wr, 898 op/s; 75867/456936 objects degraded (16.603%) 2024-03-21T14:18:37.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:37 smithi003 ceph-mon[29576]: 10.c scrub starts 2024-03-21T14:18:37.412 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1406731337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:37.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:37 smithi003 ceph-mon[40510]: pgmap v733: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 767 B/s rd, 46 MiB/s wr, 898 op/s; 75867/456936 objects degraded (16.603%) 2024-03-21T14:18:37.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:37 smithi003 ceph-mon[40510]: 10.c scrub starts 2024-03-21T14:18:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.425 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:37.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:37.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1406731337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:37.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:37 smithi118 ceph-mon[33862]: pgmap v733: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 9.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 767 B/s rd, 46 MiB/s wr, 898 op/s; 75867/456936 objects degraded (16.603%) 2024-03-21T14:18:37.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:37 smithi118 ceph-mon[33862]: 10.c scrub starts 2024-03-21T14:18:37.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:18:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:18:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:18:37.458 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:18:37.527 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:37.563 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:37.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:37.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:37.701 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:37.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:37.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:37.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:37.920 INFO:teuthology.orchestra.run.smithi003.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-21T14:18:37.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:37.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:37.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:38.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:38.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:38.221 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:38.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:38.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:38.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:38.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:38.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:38 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:18:38.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:38 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:18:38.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:38.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:38 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:18:38.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:38 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:18:38.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:38 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:18:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:38 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:18:38.481 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:38.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.536 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:38.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:38.637 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:38.674 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:38.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:38.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:38.913 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:38.947 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:38.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:38.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:39.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:39.085 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:39.167 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:39.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:39.221 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:39.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:39.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:39.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:39.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:18:39.377 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 8.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 46 MiB/s wr, 1.96k op/s; 71721/431556 objects degraded (16.619%) 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 8.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 46 MiB/s wr, 1.96k op/s; 71721/431556 objects degraded (16.619%) 2024-03-21T14:18:39.378 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:18:39.379 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:39 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:18:39.412 INFO:teuthology.orchestra.run.smithi003.stdout:get_pg_num_history pg_num_history pg_num_history(e277 pg_nums {1={19=1,268=11},2={63=3},3={64=32},4={66=32},5={68=32},6={70=32},8={195=26},9={244=16}} deleted_pools 243,8) 2024-03-21T14:18:39.412 INFO:teuthology.orchestra.run.smithi003.stdout:Importing pgid 4.13 2024-03-21T14:18:39.412 INFO:teuthology.orchestra.run.smithi003.stdout:Write #4:c8333aab:gc::gc.11:head# 2024-03-21T14:18:39.412 INFO:teuthology.orchestra.run.smithi003.stdout:snapset 0=[]:{} 2024-03-21T14:18:39.413 INFO:teuthology.orchestra.run.smithi003.stdout:Write #4:cba150de:::obj_delete_at_hint.0000000051:head# 2024-03-21T14:18:39.413 INFO:teuthology.orchestra.run.smithi003.stdout:snapset 0=[]:{} 2024-03-21T14:18:39.413 INFO:teuthology.orchestra.run.smithi003.stdout:Write #4:cc909b6b:::obj_delete_at_hint.0000000106:head# 2024-03-21T14:18:39.413 INFO:teuthology.orchestra.run.smithi003.stdout:snapset 0=[]:{} 2024-03-21T14:18:39.413 INFO:teuthology.orchestra.run.smithi003.stdout:Write #4:cd178ae0:reshard::reshard.0000000014:head# 2024-03-21T14:18:39.413 INFO:teuthology.orchestra.run.smithi003.stdout:snapset 0=[]:{} 2024-03-21T14:18:39.414 INFO:teuthology.orchestra.run.smithi003.stdout:Write #4:cedbb205:::obj_delete_at_hint.0000000017:head# 2024-03-21T14:18:39.414 INFO:teuthology.orchestra.run.smithi003.stdout:snapset 0=[]:{} 2024-03-21T14:18:39.414 INFO:teuthology.orchestra.run.smithi003.stdout:write_pg epoch 277 info 4.13( v 236'24 (0'0,236'24] local-lis/les=74/75 n=5 ec=66/66 lis/c=74/74 les/c/f=75/75/0 sis=74) 2024-03-21T14:18:39.414 INFO:teuthology.orchestra.run.smithi003.stdout:Import successful 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 8.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 46 MiB/s wr, 1.96k op/s; 71721/431556 objects degraded (16.619%) 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:18:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:39 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:18:39.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:18:39.463 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:39.471 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:18:39.471 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:18:39.471 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-03-21T14:18:39.471 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:18:39.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:39.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:39.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:39.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:39.795 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:39.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:39.927 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:39.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:40.029 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:40.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:40.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:40.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[29576]: 10.c scrub ok 2024-03-21T14:18:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 71721/431556 objects degraded (16.619%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:18:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:18:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[40510]: 10.c scrub ok 2024-03-21T14:18:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 71721/431556 objects degraded (16.619%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:18:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:40 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:18:40.300 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:40.334 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:40.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:40 smithi118 ceph-mon[33862]: 10.c scrub ok 2024-03-21T14:18:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:40 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 71721/431556 objects degraded (16.619%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:40 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:18:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:40 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:18:40.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.477 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:40.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:40.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:40.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:40.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:40.746 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:40.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:40.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:40.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:40.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:41.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:41.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:41.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:41.151 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:41.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:41.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:41.286 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:41.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:41.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:41 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:18:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:41 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:18:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:41 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:18:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:41 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:18:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:41 smithi118 ceph-mon[33862]: pgmap v735: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 8.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 1.29k op/s; 71721/431556 objects degraded (16.619%) 2024-03-21T14:18:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:18:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:18:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:18:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:18:41.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[29576]: pgmap v735: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 8.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 1.29k op/s; 71721/431556 objects degraded (16.619%) 2024-03-21T14:18:41.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:18:41.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:18:41.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:18:41.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:18:41.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:41 smithi003 ceph-mon[40510]: pgmap v735: 158 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 8.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 13 MiB/s wr, 1.29k op/s; 71721/431556 objects degraded (16.619%) 2024-03-21T14:18:41.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:41.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:41.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:41.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:41.703 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:41.805 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:41.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:41.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:41.982 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:41.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:41.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:42.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:42.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:42.221 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:42.256 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:42.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:42.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:42 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:18:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:42 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:18:42.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:42 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:18:42.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:42 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:18:42.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:42 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:18:42.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:42 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:18:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.525 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:42.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:42.662 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:42.764 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:42.801 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:42.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:42.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:18:42.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:42.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:43.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:43.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:43.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:43.216 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:43.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:43.352 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:43.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:43.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/exp.4.13.2 2024-03-21T14:18:43.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:18:43.413 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: pgmap v736: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 6.3 MiB/s wr, 1.66k op/s; 67607/411474 objects degraded (16.430%) 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: 5.11 deep-scrub starts 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: 5.11 deep-scrub ok 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: pgmap v736: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 6.3 MiB/s wr, 1.66k op/s; 67607/411474 objects degraded (16.430%) 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: 5.11 deep-scrub starts 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: 5.11 deep-scrub ok 2024-03-21T14:18:43.414 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:18:43.415 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:43 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:18:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:18:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:18:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: pgmap v736: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.4 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 6.3 MiB/s wr, 1.66k op/s; 67607/411474 objects degraded (16.430%) 2024-03-21T14:18:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: 5.11 deep-scrub starts 2024-03-21T14:18:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: 5.11 deep-scrub ok 2024-03-21T14:18:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:18:43.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:43 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:18:43.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:43.487 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:43.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:43.621 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:43.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:43.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:43.758 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:43.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:43.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:43.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:43.899 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:43.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:43.953 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:43.957 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:44.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.055 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:44.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:44.156 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:44.193 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:44.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:44.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:44 smithi118 ceph-mon[33862]: 1.3 scrub starts 2024-03-21T14:18:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:44 smithi118 ceph-mon[33862]: 1.3 scrub ok 2024-03-21T14:18:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:44 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:18:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:44 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:18:44.440 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:44.474 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[29576]: 1.3 scrub starts 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[29576]: 1.3 scrub ok 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[40510]: 1.3 scrub starts 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[40510]: 1.3 scrub ok 2024-03-21T14:18:44.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:18:44.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:44 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:18:44.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:44.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:44.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:44.753 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:44.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:44.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:44.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:44.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:45.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:45.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:45.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:45.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:45.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:45.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:45.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:45.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:45.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:45 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:18:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:45 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:18:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:45 smithi118 ceph-mon[33862]: pgmap v737: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.3 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 900 B/s rd, 0 B/s wr, 1.47k op/s; 67607/406827 objects degraded (16.618%) 2024-03-21T14:18:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:45 smithi118 ceph-mon[33862]: 10.4 deep-scrub starts 2024-03-21T14:18:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:45 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 67607/411474 objects degraded (16.430%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.433 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:45.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:18:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:18:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[29576]: pgmap v737: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.3 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 900 B/s rd, 0 B/s wr, 1.47k op/s; 67607/406827 objects degraded (16.618%) 2024-03-21T14:18:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[29576]: 10.4 deep-scrub starts 2024-03-21T14:18:45.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 67607/411474 objects degraded (16.430%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:45.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:18:45.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:18:45.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[40510]: pgmap v737: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.3 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 900 B/s rd, 0 B/s wr, 1.47k op/s; 67607/406827 objects degraded (16.618%) 2024-03-21T14:18:45.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[40510]: 10.4 deep-scrub starts 2024-03-21T14:18:45.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:45 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 67607/411474 objects degraded (16.430%), 24 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:45.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:45.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:45.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:45.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:45.812 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:45.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:45.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:45.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:45.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:46.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:46.115 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:46.180 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:18:46.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:46.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:46.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:46.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:46.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:46 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:18:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:46 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:18:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:46 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1859894657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:46 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:46.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:18:46.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:18:46.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1859894657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:46.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:46.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:18:46.495 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:18:46.495 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1859894657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:46.495 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:18:46.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:46.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:46.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:46.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:18:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:18:46.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:46.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:46.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:46.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:46.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:47.035 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:47.066 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:47.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:47.160 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:18:47.051+0000 7fa17996d700 -1 mgr get_metadata_python Requested missing service osd.6 2024-03-21T14:18:47.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:47.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:47.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:47.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:47.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:47 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:47 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:18:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:47 smithi118 ceph-mon[33862]: osdmap e347: 8 total, 6 up, 7 in 2024-03-21T14:18:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:47 smithi118 ceph-mon[33862]: pgmap v739: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.3 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.67k op/s; 67607/406827 objects degraded (16.618%) 2024-03-21T14:18:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:47.432 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:18:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:18:47.449 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:47.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:47.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:47.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:47.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:18:47.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[29576]: osdmap e347: 8 total, 6 up, 7 in 2024-03-21T14:18:47.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[29576]: pgmap v739: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.3 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.67k op/s; 67607/406827 objects degraded (16.618%) 2024-03-21T14:18:47.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:47.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:47.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:18:47.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[40510]: osdmap e347: 8 total, 6 up, 7 in 2024-03-21T14:18:47.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[40510]: pgmap v739: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 8.3 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.67k op/s; 67607/406827 objects degraded (16.618%) 2024-03-21T14:18:47.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:18:47.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:47.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:47.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:47.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:47.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:47.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:47.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:47.997 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:48.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:48.164 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:48.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:48.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:48.303 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:48.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:48.429 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [2, 6] live_osds: [0, 1, 3, 4, 5, 7] 2024-03-21T14:18:48.429 INFO:tasks.thrashosds.thrasher:Reviving osd 6 2024-03-21T14:18:48.429 INFO:tasks.cephadm.osd.6:Restarting osd.6 (starting--it wasn't running)... 2024-03-21T14:18:48.429 DEBUG:teuthology.orchestra.run.smithi118:osd.6> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6.service 2024-03-21T14:18:48.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl start ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6 2024-03-21T14:18:48.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:48.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:48.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:48.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:48.540 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:48.567 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:48.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:48.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:48.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:48.801 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:48.829 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:48.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:48.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:48.972 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:48.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:49.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:49.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:49.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:49.211 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:49.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:49.345 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:49.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:49.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:49 smithi118 ceph-mon[33862]: pgmap v740: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.63k op/s; 63617/382704 objects degraded (16.623%) 2024-03-21T14:18:49.431 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:49 smithi118 podman[133438]: 2024-03-21 14:18:48.951174951 +0000 UTC m=+0.035164980 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:18:49.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:49.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:49 smithi003 ceph-mon[29576]: pgmap v740: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.63k op/s; 63617/382704 objects degraded (16.623%) 2024-03-21T14:18:49.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:49 smithi003 ceph-mon[40510]: pgmap v740: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.63k op/s; 63617/382704 objects degraded (16.623%) 2024-03-21T14:18:49.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.510 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:49.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:49.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:49.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:18:49.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:49.778 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:49.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:49.900 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:18:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:18:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:18:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:18:49.901 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:18:49.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:49.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:50.025 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:50.070 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:50.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:50.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.206 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:50.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:50.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:18:50.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:18:50.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:18:50.251 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:18:50.255 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:18:50.287 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:18:50.288 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:18:50.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:50.341 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:50.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:50.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:50 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 63617/382704 objects degraded (16.623%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:50 smithi003 ceph-mon[29576]: osdmap e348: 8 total, 6 up, 7 in 2024-03-21T14:18:50.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3772308629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:50.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:50 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 63617/382704 objects degraded (16.623%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:50.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:50 smithi003 ceph-mon[40510]: osdmap e348: 8 total, 6 up, 7 in 2024-03-21T14:18:50.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3772308629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:50.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:50.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:50 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 63617/382704 objects degraded (16.623%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:50 smithi118 ceph-mon[33862]: osdmap e348: 8 total, 6 up, 7 in 2024-03-21T14:18:50.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3772308629' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:50.721 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:50.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:50.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:50.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:50.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:50.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:50.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:50.998 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:51.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.028 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:51.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:51.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:51.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.167 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:51.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:51.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:50 smithi118 podman[133438]: 2024-03-21 14:18:50.691308386 +0000 UTC m=+1.775298401 container create 2cae55492a7697e3d357430d171b9e70b88aa50677e4ba05460e0e1106cf04d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-21T14:18:51.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:18:51.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:18:51.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:51.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.299 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:51.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:51.400 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:51.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:51.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:51 smithi003 ceph-mon[29576]: pgmap v742: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 63617/382704 objects degraded (16.623%) 2024-03-21T14:18:51.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:51 smithi003 ceph-mon[40510]: pgmap v742: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 63617/382704 objects degraded (16.623%) 2024-03-21T14:18:51.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:51.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:51.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:51.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:51.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:51 smithi118 ceph-mon[33862]: pgmap v742: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.8 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 63617/382704 objects degraded (16.623%) 2024-03-21T14:18:51.705 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:51.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:51.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:51.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:51.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:51.983 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:51.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:52.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:18:52.085 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:52.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:52.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:52.115 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:52.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:52.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:52.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:52.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:52.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:52.513 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:52.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:52.653 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:52.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:52.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:52.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:52.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:52.889 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:52.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:52.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:53.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.044 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:53.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:53.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:53.182 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:52 smithi118 podman[133438]: 2024-03-21 14:18:52.737283351 +0000 UTC m=+3.821273371 container init 2cae55492a7697e3d357430d171b9e70b88aa50677e4ba05460e0e1106cf04d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T14:18:53.182 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:52 smithi118 podman[133438]: 2024-03-21 14:18:52.742047883 +0000 UTC m=+3.826037903 container start 2cae55492a7697e3d357430d171b9e70b88aa50677e4ba05460e0e1106cf04d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-21T14:18:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.184 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:53.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:53.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:18:53.285 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:53.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:53.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:53.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:53.576 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:53.608 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:53.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:53.738 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:53 smithi003 ceph-mon[29576]: pgmap v743: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.67k op/s; 60333/366702 objects degraded (16.453%) 2024-03-21T14:18:53.738 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:53 smithi003 ceph-mon[40510]: pgmap v743: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.67k op/s; 60333/366702 objects degraded (16.453%) 2024-03-21T14:18:53.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:53.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:53.881 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:53.884 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:18:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:53.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:53.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:53.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-mon[33862]: pgmap v743: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.67k op/s; 60333/366702 objects degraded (16.453%) 2024-03-21T14:18:53.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate[133625]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 podman[133438]: 2024-03-21 14:18:53.60970014 +0000 UTC m=+4.693690167 container attach 2cae55492a7697e3d357430d171b9e70b88aa50677e4ba05460e0e1106cf04d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-21T14:18:53.932 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 bash[133438]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-21T14:18:53.933 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:53 smithi118 podman[133438]: 2024-03-21 14:18:53.6106674 +0000 UTC m=+4.694657419 container died 2cae55492a7697e3d357430d171b9e70b88aa50677e4ba05460e0e1106cf04d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-21T14:18:53.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:54.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:54.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:54.127 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:54.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:54.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:54.398 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:54.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:54.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:54.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:54.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:54.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:54.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:54.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:54.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:54.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:54.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:54.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:55.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:55.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:55.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:55.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:55.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.377 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:55.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:55.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:55.503 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:55.541 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 podman[133865]: 2024-03-21 14:18:55.270661759 +0000 UTC m=+1.764074054 container remove 2cae55492a7697e3d357430d171b9e70b88aa50677e4ba05460e0e1106cf04d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-21T14:18:55.541 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:55 smithi118 ceph-mon[33862]: pgmap v744: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 60333/362739 objects degraded (16.633%) 2024-03-21T14:18:55.542 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:55 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 60333/362739 objects degraded (16.633%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:55.542 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:55 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3526276330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:55.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:55.639 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:55.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:55 smithi003 ceph-mon[29576]: pgmap v744: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 60333/362739 objects degraded (16.633%) 2024-03-21T14:18:55.647 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:55 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 60333/362739 objects degraded (16.633%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:55.648 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:55 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3526276330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:55.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:55 smithi003 ceph-mon[40510]: pgmap v744: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 60333/362739 objects degraded (16.633%) 2024-03-21T14:18:55.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:55 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 60333/362739 objects degraded (16.633%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:18:55.648 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:55 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3526276330' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:18:55.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:55.764 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:18:55.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:55.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:55.874 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 podman[134052]: 2024-03-21 14:18:55.445822438 +0000 UTC m=+0.026535928 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:18:55.874 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 podman[134052]: 2024-03-21 14:18:55.587348849 +0000 UTC m=+0.168062342 container create fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-21T14:18:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:55.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:55.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:55.990 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:18:56.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:56.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:56.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:56.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 podman[134052]: 2024-03-21 14:18:55.874544886 +0000 UTC m=+0.455258395 container init fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-21T14:18:56.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 podman[134052]: 2024-03-21 14:18:55.882006025 +0000 UTC m=+0.462719520 container start fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-21T14:18:56.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 bash[134052]: fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 2024-03-21T14:18:56.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:55 smithi118 systemd[1]: Started Ceph osd.6 for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:18:56.215 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:56.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:56.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:56.414 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:18:56.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:56.483 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:56.515 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:56 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:18:56.190+0000 7efc179c97c0 -1 Falling back to public interface 2024-03-21T14:18:56.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:56.616 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:56.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:18:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:18:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3526276330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:56.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-mon[29576]: osdmap e349: 8 total, 6 up, 7 in 2024-03-21T14:18:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3526276330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:56.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:56 smithi003 ceph-mon[40510]: osdmap e349: 8 total, 6 up, 7 in 2024-03-21T14:18:56.850 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:56.886 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:56.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:56 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:18:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3526276330' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:18:56.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:56 smithi118 ceph-mon[33862]: osdmap e349: 8 total, 6 up, 7 in 2024-03-21T14:18:56.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:57.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:57.122 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:57.156 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:57.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:57.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:57.294 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:57.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:57.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:57.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:57.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:57.573 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:57.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:57.656 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:18:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:18:57.676 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:57.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:57.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:18:57.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:18:57.843 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:18:57.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:57 smithi118 ceph-mon[33862]: pgmap v746: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 832 op/s; 60333/362739 objects degraded (16.633%) 2024-03-21T14:18:57.939 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:57 smithi003 ceph-mon[29576]: pgmap v746: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 832 op/s; 60333/362739 objects degraded (16.633%) 2024-03-21T14:18:57.939 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:57 smithi003 ceph-mon[40510]: pgmap v746: 158 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 44 active+undersized, 89 active+clean; 7.4 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 832 op/s; 60333/362739 objects degraded (16.633%) 2024-03-21T14:18:57.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:57.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:57.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:58.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:58.114 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:58.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:58.252 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:58.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:58.389 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:58.431 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:58 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:18:58.232+0000 7efc179c97c0 -1 osd.6 332 log_to_monitors true 2024-03-21T14:18:58.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:58.526 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:58.564 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:18:58.593 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:18:58.621 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:18:58.628 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:58.669 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:18:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.670 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:58.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:58.711 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[29576]: from='osd.6 v2:172.21.15.118:6808/1863637416' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1301673626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[40510]: from='osd.6 v2:172.21.15.118:6808/1863637416' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:18:58.742 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:18:58.743 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1301673626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:58.743 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:18:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:18:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:18:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:18:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:18:58.743 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:18:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:18:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.6 2024-03-21T14:18:58.744 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:18:58.744 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:58 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:18:58.744 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:58 smithi118 ceph-mon[33862]: from='osd.6 v2:172.21.15.118:6808/1863637416' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:18:58.744 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:58 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-21T14:18:58.744 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1301673626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:18:58.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:58.805 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:58.906 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:58.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:58.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:59.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:18:59.069 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:18:59.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:59.222 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:59.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:59.250 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:18:59.250 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:18:59.250 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-03-21T14:18:59.250 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:18:59.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:18:59.373 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:18:59.431 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:18:59.049+0000 7efc04e19700 -1 osd.6 350 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:18:59.475 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:59.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:59.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:18:59.661 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:18:59.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:18:59.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:18:59.800 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:18:59.901 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:18:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:18:59.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:18:59.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: pgmap v747: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.50k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T14:19:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: osdmap e350: 8 total, 6 up, 7 in 2024-03-21T14:19:00.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: from='osd.6 v2:172.21.15.118:6808/1863637416' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: from='osd.6 ' entity='osd.6' 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 57079/343170 objects degraded (16.633%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: pgmap v747: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.50k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: osdmap e350: 8 total, 6 up, 7 in 2024-03-21T14:19:00.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: from='osd.6 v2:172.21.15.118:6808/1863637416' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:19:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:19:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:19:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:19:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: from='osd.6 ' entity='osd.6' 2024-03-21T14:19:00.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:18:59 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 57079/343170 objects degraded (16.633%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:00.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:00.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.086 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:00.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: pgmap v747: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.50k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: osdmap e350: 8 total, 6 up, 7 in 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: from='osd.6 v2:172.21.15.118:6808/1863637416' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: from='osd.6 ' entity='osd.6' 2024-03-21T14:19:00.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:18:59 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 57079/343170 objects degraded (16.633%), 24 pgs degraded, 68 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:00.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:00.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:00.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:00.323 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.347 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:00.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:00.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:00.485 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:00.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:00.587 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:00.625 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:00.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:00.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:00.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:00.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:00.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:00.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:19:00.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:19:00.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:01.019 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:01.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:01.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:01.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:01.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: osd.6 v2:172.21.15.118:6808/1863637416 boot 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: osdmap e351: 8 total, 7 up, 7 in 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: osd.6 v2:172.21.15.118:6808/1863637416 boot 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: osdmap e351: 8 total, 7 up, 7 in 2024-03-21T14:19:01.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:00 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:19:01.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:01.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:01.158 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:19:01.158 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:19:01.158 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:19:01.158 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:19:01.158 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:228274 8.6 8.e78a0f06 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:19.639563+0000", 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 401.51529792999997, 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92258471799999997, 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 228274 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.159 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639563+0000", 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639563+0000", 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639564+0000", 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4780000000000001e-06 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639571+0000", 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4520000000000002e-06 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639585+0000", 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4477999999999999e-05 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639594+0000", 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1519999999999998e-06 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.161 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639673+0000", 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9060000000000005e-05 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:19.639694+0000", 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1234999999999999e-05 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:20.562121+0000", 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.92242685000000002 2024-03-21T14:19:01.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:20.562147+0000", 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6013e-05 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:47249 10.5 e326/317 10:a2742cd9:::benchmark_data_smithi003_160672_object47248:head v 326'3041, mlcod=326'3039)", 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:18.853797+0000", 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 102.30106393200001, 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.069691949000000003, 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.853797+0000", 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.853797+0000", 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.164 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.853797+0000", 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.61e-07 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.853817+0000", 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9607999999999999e-05 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920419+0000", 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.066601885999999999 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.165 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920435+0000", 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.649e-05 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920777+0000", 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034140499999999997 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920801+0000", 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4369999999999999e-05 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.166 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.923440+0000", 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026386000000000001 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.923489+0000", 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8928999999999999e-05 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:47250 10.8 e326/317 10:1cfe061b:::benchmark_data_smithi003_160672_object47249:head v 326'2907, mlcod=326'2905)", 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:18.854223+0000", 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 102.300637634, 2024-03-21T14:19:01.167 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.069507545000000004, 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.854223+0000", 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.854223+0000", 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.168 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.854223+0000", 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2899999999999999e-07 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.854267+0000", 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3596000000000001e-05 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920427+0000", 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.066159678999999999 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.169 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920439+0000", 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1951e-05 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920598+0000", 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015933200000000001 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920625+0000", 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.7165000000000002e-05 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.170 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.923655+0000", 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003029585 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.923730+0000", 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5808000000000003e-05 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:47251 10.d e326/317 10:bebf77f7:::benchmark_data_smithi003_160672_object47250:head v 326'2866, mlcod=326'2864)", 2024-03-21T14:19:01.171 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:18.855664+0000", 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 102.299196174, 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.067387326999999997, 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.855664+0000", 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.855664+0000", 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.855665+0000", 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4799999999999998e-07 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.855773+0000", 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010773 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920433+0000", 2024-03-21T14:19:01.173 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.064660250000000002 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.920443+0000", 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0185e-05 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.921194+0000", 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00075120799999999995 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.174 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.921213+0000", 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9119000000000001e-05 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.922995+0000", 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001781113 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:18.923052+0000", 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7173999999999998e-05 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:52753 10.8 e326/317 10:1c0ff436:::benchmark_data_smithi003_160672_object52752:head v 326'3242, mlcod=326'3240)", 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:21.790122+0000", 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.364738486999997, 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.062588352999999999, 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790122+0000", 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.176 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790122+0000", 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790123+0000", 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1099999999999995e-07 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790183+0000", 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0611e-05 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.177 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790192+0000", 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8119999999999997e-06 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790198+0000", 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3180000000000003e-06 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790246+0000", 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7207999999999999e-05 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.178 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.790265+0000", 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9729e-05 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.852664+0000", 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.062398602999999997 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.852710+0000", 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6461000000000001e-05 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.179 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:52742 10.1 e326/317 10:802e6a1c:::benchmark_data_smithi003_160672_object52741:head v 326'3307, mlcod=326'3302)", 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:21.792014+0000", 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 99.362846986999998, 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.069739762999999996, 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.792014+0000", 2024-03-21T14:19:01.180 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.792014+0000", 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.792014+0000", 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5799999999999998e-07 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.792076+0000", 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1939000000000004e-05 2024-03-21T14:19:01.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.852655+0000", 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.060579168000000003 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.852670+0000", 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4253000000000001e-05 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.852771+0000", 2024-03-21T14:19:01.182 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000101586 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.852795+0000", 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3825999999999999e-05 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.861705+0000", 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089095830000000004 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:21.861753+0000", 2024-03-21T14:19:01.183 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8649999999999997e-05 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:85172 10.8 e332/317 10:1153b16f:::benchmark_data_smithi003_160672_object85171:head v 332'5227, mlcod=332'5222)", 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:43.737234+0000", 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.417626573999996, 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.075701706999999993, 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.184 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737234+0000", 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737234+0000", 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737235+0000", 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6400000000000002e-07 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.185 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737303+0000", 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8164999999999995e-05 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737310+0000", 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0339999999999999e-06 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737315+0000", 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8069999999999999e-06 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.186 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737351+0000", 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.6427999999999997e-05 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.737365+0000", 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3648999999999999e-05 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.812864+0000", 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.075498894999999996 2024-03-21T14:19:01.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.812936+0000", 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2064999999999995e-05 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:85173 10.8 e332/317 10:1549ce96:::benchmark_data_smithi003_160672_object85172:head v 332'5228, mlcod=332'5222)", 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:43.738975+0000", 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.415885062000001, 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.079897790999999996, 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.738975+0000", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.738975+0000", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.738976+0000", 2024-03-21T14:19:01.189 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9999999999999997e-07 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739042+0000", 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5399000000000002e-05 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739050+0000", 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1810000000000001e-06 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.190 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739056+0000", 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0950000000000001e-06 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.810589+0000", 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.071533523000000002 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.810616+0000", 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6568e-05 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.191 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.818831+0000", 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082147929999999997 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.818873+0000", 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2531999999999999e-05 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:85180 10.c e332/317 10:33f0177e:::benchmark_data_smithi003_160672_object85179:head v 332'5352, mlcod=332'5350)", 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:43.739320+0000", 2024-03-21T14:19:01.192 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.415540065000002, 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.077253522000000005, 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739320+0000", 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739320+0000", 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.193 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739321+0000", 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8699999999999995e-07 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739374+0000", 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2462e-05 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809990+0000", 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070616119000000005 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.810000+0000", 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0000000000000001e-05 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.811898+0000", 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001898339 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.811918+0000", 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9791000000000001e-05 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.816356+0000", 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0044378339999999999 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.816574+0000", 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021839 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.196 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:85182 10.5 e332/317 10:a5982d4c:::benchmark_data_smithi003_160672_object85181:head v 332'5371, mlcod=332'5369)", 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:43.739666+0000", 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.415194821, 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.077705758999999999, 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739666+0000", 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.197 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739666+0000", 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739667+0000", 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3200000000000004e-07 2024-03-21T14:19:01.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: osd.6 v2:172.21.15.118:6808/1863637416 boot 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: osdmap e351: 8 total, 7 up, 7 in 2024-03-21T14:19:01.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:00 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.739692+0000", 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.5024e-05 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809738+0000", 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070046466000000002 2024-03-21T14:19:01.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809759+0000", 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1410999999999999e-05 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809866+0000", 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107018 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809892+0000", 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6038999999999999e-05 2024-03-21T14:19:01.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.817327+0000", 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0074341709999999998 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.817371+0000", 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4798e-05 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:85183 10.1 e332/317 10:8b580ad5:::benchmark_data_smithi003_160672_object85182:head v 332'5287, mlcod=332'5285)", 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:43.741441+0000", 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 77.413419474999998, 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073758649999999995, 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.741441+0000", 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.203 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.741441+0000", 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.741442+0000", 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3399999999999997e-07 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.741542+0000", 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9517000000000003e-05 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.204 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809984+0000", 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.068442111 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.809993+0000", 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9290000000000004e-06 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.810062+0000", 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9162e-05 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.205 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.810081+0000", 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9000000000000001e-05 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.815158+0000", 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050772159999999998 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:43.815200+0000", 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1780999999999999e-05 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.206 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108647 10.b e338/317 10:d3b72263:::benchmark_data_smithi003_160672_object108646:head v 338'6882, mlcod=338'6879)", 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.868094+0000", 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.286766815, 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14574438100000001, 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868094+0000", 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868094+0000", 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868095+0000", 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.22e-06 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868123+0000", 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8098000000000001e-05 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868134+0000", 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0606e-05 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868142+0000", 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1729999999999995e-06 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868370+0000", 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000227937 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.868391+0000", 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1152999999999999e-05 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013804+0000", 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14541299699999999 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013838+0000", 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4196999999999998e-05 2024-03-21T14:19:01.210 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108652 10.b e338/317 10:db3b8a6a:::benchmark_data_smithi003_160672_object108651:head v 338'6883, mlcod=338'6880)", 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.869059+0000", 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.28580169, 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14807394700000001, 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869059+0000", 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869059+0000", 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869060+0000", 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7000000000000004e-07 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869082+0000", 2024-03-21T14:19:01.212 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2208000000000002e-05 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869091+0000", 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2660000000000007e-06 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869098+0000", 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6189999999999996e-06 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.213 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.870340+0000", 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012424929999999999 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.870359+0000", 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8845999999999999e-05 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.016957+0000", 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14659840900000001 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.017133+0000", 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017543600000000001 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108654 10.2 e338/317 10:461a763d:::benchmark_data_smithi003_160672_object108653:head v 338'6856, mlcod=338'6854)", 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.869610+0000", 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.285250103000003, 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.144153318, 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.215 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869610+0000", 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869610+0000", 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.216 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869611+0000", 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3699999999999999e-07 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869634+0000", 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2552999999999999e-05 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869644+0000", 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0094e-05 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.217 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869651+0000", 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2840000000000001e-06 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869728+0000", 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6579e-05 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.869748+0000", 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0421e-05 2024-03-21T14:19:01.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871977+0000", 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022290529999999999 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013764+0000", 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14178649700000001 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108649 10.8 e338/317 10:19d00ab1:::benchmark_data_smithi003_160672_object108648:head v 338'6649, mlcod=338'6647)", 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.871453+0000", 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.283407259000001, 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14231327299999999, 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871453+0000", 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.220 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871453+0000", 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871454+0000", 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6500000000000008e-07 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871485+0000", 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0806999999999998e-05 2024-03-21T14:19:01.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871497+0000", 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1807000000000001e-05 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871504+0000", 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6229999999999997e-06 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871583+0000", 2024-03-21T14:19:01.222 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8887999999999994e-05 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871607+0000", 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3788e-05 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013722+0000", 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.142115097 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.223 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013767+0000", 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4298000000000002e-05 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108656 10.d e338/317 10:b1ec3e35:::benchmark_data_smithi003_160672_object108655:head v 338'6680, mlcod=338'6678)", 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.871491+0000", 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.283369405000002, 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15084352000000001, 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.224 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871491+0000", 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871491+0000", 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871492+0000", 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3500000000000005e-07 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.225 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871777+0000", 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000285321 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871787+0000", 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0156999999999999e-05 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871795+0000", 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.289e-06 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.226 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871849+0000", 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4080999999999999e-05 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871870+0000", 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0848000000000001e-05 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.022244+0000", 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.150374705 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.227 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.022335+0000", 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0284000000000005e-05 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108658 10.b e338/317 10:d98aa8ea:::benchmark_data_smithi003_160672_object108657:head v 338'6884, mlcod=338'6880)", 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.871858+0000", 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.283002644, 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14738870500000001, 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.228 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871858+0000", 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871858+0000", 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871859+0000", 2024-03-21T14:19:01.229 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8500000000000001e-07 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871880+0000", 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1829000000000001e-05 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871889+0000", 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3429999999999996e-06 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.871895+0000", 2024-03-21T14:19:01.230 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1260000000000003e-06 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013891+0000", 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14199653000000001 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013912+0000", 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0217e-05 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.019190+0000", 2024-03-21T14:19:01.231 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052786430000000004 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.019247+0000", 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6332000000000002e-05 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108659 10.d e338/317 10:b7a22c18:::benchmark_data_smithi003_160672_object108658:head v 338'6681, mlcod=338'6678)", 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.873956+0000", 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.280904350999997, 2024-03-21T14:19:01.232 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.153322026, 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.873956+0000", 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.873956+0000", 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.233 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.873957+0000", 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0599999999999999e-07 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.874013+0000", 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6097000000000002e-05 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013713+0000", 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13970026399999999 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.234 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013730+0000", 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6093e-05 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.016708+0000", 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029788599999999998 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.016787+0000", 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8990000000000001e-05 2024-03-21T14:19:01.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.027185+0000", 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010397404000000001 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.027278+0000", 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3412000000000006e-05 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.236 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108651 10.8 e338/317 10:1361e5dc:::benchmark_data_smithi003_160672_object108650:head v 338'6650, mlcod=338'6647)", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.874449+0000", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.280411696000002, 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14467717999999999, 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.874449+0000", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.874449+0000", 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.874450+0000", 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0130000000000001e-06 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.874497+0000", 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7157999999999998e-05 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013728+0000", 2024-03-21T14:19:01.238 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13923147599999999 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013737+0000", 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4349999999999997e-06 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013807+0000", 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0413999999999994e-05 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013836+0000", 2024-03-21T14:19:01.239 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8677999999999999e-05 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.018881+0000", 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0050453140000000004 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.019126+0000", 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024469200000000002 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:108669 10.b e338/317 10:d71a1aae:::benchmark_data_smithi003_160672_object108668:head v 338'6885, mlcod=338'6880)", 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.880149+0000", 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 61.274711576999998, 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.14162665399999999, 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.880149+0000", 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.241 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.880149+0000", 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.880150+0000", 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.344e-06 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.242 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.880175+0000", 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.482e-05 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013711+0000", 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13353578499999999 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.013729+0000", 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8363e-05 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.243 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.017176+0000", 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034468400000000001 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.017196+0000", 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9840999999999999e-05 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.021735+0000", 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045391240000000003 2024-03-21T14:19:01.244 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.021776+0000", 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0537000000000002e-05 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:01.245 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:19:01.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.286 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:01.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:01.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:19:01.362 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-21T14:19:01.362 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-03-21T14:19:01.362 DEBUG:teuthology.orchestra.run.smithi003:osd.2> sudo journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2.service 2024-03-21T14:19:01.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:01.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl start ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2 2024-03-21T14:19:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.461 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:01.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:01.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:01.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:01.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:01.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:01.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:01.879 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:01.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:01.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:02.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:02.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:02.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:02.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:19:02.152 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:02.248 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:02.253 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.288 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:02.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: pgmap v750: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: osdmap e352: 8 total, 7 up, 7 in 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:02.369 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: pgmap v750: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:02.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: osdmap e352: 8 total, 7 up, 7 in 2024-03-21T14:19:02.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:19:02.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:19:02.370 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:02.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:02.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.423 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:02.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: pgmap v750: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: osdmap e352: 8 total, 7 up, 7 in 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:19:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:02.525 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:02.559 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:02.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:02.690 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:02 smithi003 podman[174578]: 2024-03-21 14:19:02.269379817 +0000 UTC m=+0.045112492 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:19:02.690 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:02 smithi003 podman[174578]: 2024-03-21 14:19:02.389110969 +0000 UTC m=+0.164843651 container create 5e30d8f70bf7bf60581c4d3cb1e3793e75301c6d2e4f595a9b0a102625dc5c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T14:19:02.690 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:02 smithi003 podman[174578]: 2024-03-21 14:19:02.683005199 +0000 UTC m=+0.458737888 container init 5e30d8f70bf7bf60581c4d3cb1e3793e75301c6d2e4f595a9b0a102625dc5c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T14:19:02.690 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:02 smithi003 podman[174578]: 2024-03-21 14:19:02.689131705 +0000 UTC m=+0.464864379 container start 5e30d8f70bf7bf60581c4d3cb1e3793e75301c6d2e4f595a9b0a102625dc5c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:19:02.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:02.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:02.800 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:02.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:02.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:02.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:19:02.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:02.954 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:02 smithi003 podman[174578]: 2024-03-21 14:19:02.783043329 +0000 UTC m=+0.558776014 container attach 5e30d8f70bf7bf60581c4d3cb1e3793e75301c6d2e4f595a9b0a102625dc5c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-21T14:19:03.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:03.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:03.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:03.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:03.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[40510]: pgmap v752: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:03.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[40510]: pgmap v753: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:03.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:03.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[40510]: osdmap e353: 8 total, 7 up, 7 in 2024-03-21T14:19:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[29576]: pgmap v752: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[29576]: pgmap v753: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-mon[29576]: osdmap e353: 8 total, 7 up, 7 in 2024-03-21T14:19:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.280 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:03.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:03.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:03.418 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:03 smithi118 ceph-mon[33862]: pgmap v752: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.33k op/s; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:03 smithi118 ceph-mon[33862]: pgmap v753: 158 pgs: 24 active+undersized+degraded, 44 active+undersized, 90 active+clean; 7.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 57079/343170 objects degraded (16.633%) 2024-03-21T14:19:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:03 smithi118 ceph-mon[33862]: osdmap e353: 8 total, 7 up, 7 in 2024-03-21T14:19:03.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:03.554 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:03.640 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T14:19:03.640 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T14:19:03.640 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate[174617]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-21T14:19:03.641 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 bash[174578]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-21T14:19:03.656 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:03.692 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:03.793 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:03.830 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:03.932 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:03.951 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:03 smithi003 podman[174578]: 2024-03-21 14:19:03.64002745 +0000 UTC m=+1.415760113 container died 5e30d8f70bf7bf60581c4d3cb1e3793e75301c6d2e4f595a9b0a102625dc5c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T14:19:03.971 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:03.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:04.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:04.108 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:04.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:04.248 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:04.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:04.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:04.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:04.520 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:04.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:04.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:04.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:04.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:04.754 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:04 smithi003 podman[174859]: 2024-03-21 14:19:04.394489781 +0000 UTC m=+0.924446240 container remove 5e30d8f70bf7bf60581c4d3cb1e3793e75301c6d2e4f595a9b0a102625dc5c3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-activate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T14:19:04.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:04.794 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:04.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:04.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:04.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:04.985 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:05.008 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:05.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:05.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:05.170 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:05.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:05 smithi003 ceph-mon[29576]: pgmap v755: 158 pgs: 3 active+clean+remapped, 4 remapped+peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2.23k op/s; 53425/321018 objects degraded (16.642%) 2024-03-21T14:19:05.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:05 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 53425/321018 objects degraded (16.642%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:05.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:05 smithi003 ceph-mon[29576]: osdmap e354: 8 total, 7 up, 7 in 2024-03-21T14:19:05.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:05 smithi003 ceph-mon[40510]: pgmap v755: 158 pgs: 3 active+clean+remapped, 4 remapped+peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2.23k op/s; 53425/321018 objects degraded (16.642%) 2024-03-21T14:19:05.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:05 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 53425/321018 objects degraded (16.642%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:05.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:05 smithi003 ceph-mon[40510]: osdmap e354: 8 total, 7 up, 7 in 2024-03-21T14:19:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:05.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:05.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:05.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:05 smithi118 ceph-mon[33862]: pgmap v755: 158 pgs: 3 active+clean+remapped, 4 remapped+peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2.23k op/s; 53425/321018 objects degraded (16.642%) 2024-03-21T14:19:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:05 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 53425/321018 objects degraded (16.642%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:05 smithi118 ceph-mon[33862]: osdmap e354: 8 total, 7 up, 7 in 2024-03-21T14:19:05.444 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.480 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:05.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:05.581 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:05.611 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:05.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:05.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:05.751 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:05.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:05.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:05.894 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:05.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:05.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.032 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:06.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:06.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:06.160 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:06.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:06.300 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:06.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:06.436 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:06.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:06.538 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:06.576 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:06.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:06.678 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:06.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:06.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[29576]: pgmap v757: 158 pgs: 3 active+clean+remapped, 4 remapped+peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 53425/321018 objects degraded (16.642%) 2024-03-21T14:19:06.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[29576]: osdmap e355: 8 total, 7 up, 7 in 2024-03-21T14:19:06.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:19:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:19:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[40510]: pgmap v757: 158 pgs: 3 active+clean+remapped, 4 remapped+peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 53425/321018 objects degraded (16.642%) 2024-03-21T14:19:06.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:06 smithi003 ceph-mon[40510]: osdmap e355: 8 total, 7 up, 7 in 2024-03-21T14:19:06.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:06.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:06 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:06.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:06 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:06 smithi118 ceph-mon[33862]: pgmap v757: 158 pgs: 3 active+clean+remapped, 4 remapped+peering, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 53425/321018 objects degraded (16.642%) 2024-03-21T14:19:06.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:06 smithi118 ceph-mon[33862]: osdmap e355: 8 total, 7 up, 7 in 2024-03-21T14:19:06.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:06.974 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:06.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:06.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:07.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:07.110 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:07.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:07.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:07.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:07.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:07.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:07.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:07.523 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:07.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:07.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:07.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:07.659 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:19:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:19:07.679 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:19:07.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:07.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:07.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:07.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:07.930 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/234328817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:19:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:07 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:19:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:07.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/234328817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:19:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:07 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:19:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/234328817' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:19:08.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:07 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:19:08.005 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:07 smithi003 podman[174965]: 2024-03-21 14:19:07.499085233 +0000 UTC m=+2.486127788 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:19:08.006 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:07 smithi003 podman[174965]: 2024-03-21 14:19:07.62145667 +0000 UTC m=+2.608499214 container create 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009) 2024-03-21T14:19:08.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:08.065 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:08.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:08.201 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:08.303 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:08.336 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:08.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:08.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:08.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:08.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:08.609 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:08.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:19:08.711 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:08.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:08.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:08.854 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:08.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: pgmap v759: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 2.37k op/s; 49218/300540 objects degraded (16.377%); 6/300540 objects misplaced (0.002%) 2024-03-21T14:19:08.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:19:08.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:19:08.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: osdmap e356: 8 total, 7 up, 7 in 2024-03-21T14:19:08.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:19:08.855 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:19:08.855 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:08 smithi003 podman[174965]: 2024-03-21 14:19:08.518964355 +0000 UTC m=+3.506006903 container init 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:19:08.855 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:08 smithi003 podman[174965]: 2024-03-21 14:19:08.524021967 +0000 UTC m=+3.511064513 container start 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T14:19:08.856 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-osd[175208]: -- 172.21.15.3:0/387804242 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a568f9d680 con 0x55a568f67400 2024-03-21T14:19:08.856 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:08 smithi003 bash[174965]: 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 2024-03-21T14:19:08.856 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:08 smithi003 systemd[1]: Started Ceph osd.2 for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: pgmap v759: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 2.37k op/s; 49218/300540 objects degraded (16.377%); 6/300540 objects misplaced (0.002%) 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: osdmap e356: 8 total, 7 up, 7 in 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:19:08.856 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:08 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:19:08.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:08.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: pgmap v759: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.1 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 2.37k op/s; 49218/300540 objects degraded (16.377%); 6/300540 objects misplaced (0.002%) 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: osdmap e356: 8 total, 7 up, 7 in 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:19:08.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:08 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:19:08.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:09.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:09.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:09.170 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:09.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:09.241 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:09 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[175187]: 2024-03-21T14:19:09.079+0000 7f7e7f8117c0 -1 Falling back to public interface 2024-03-21T14:19:09.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:09.310 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:09.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:09.441 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:09.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:09.574 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:09.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:09.676 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:09.707 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:09.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:09.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:09.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:09.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:09 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 49218/300540 objects degraded (16.377%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:09.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:09.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:09.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:09 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 49218/300540 objects degraded (16.377%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:10.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:09 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 49218/300540 objects degraded (16.377%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:10.080 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:10.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:10.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:10.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.395 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:10.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:10.496 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:10.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.531 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:10.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:10.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:10.666 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:10.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:10.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:10.802 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:10.904 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:10.941 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:10.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:10.942 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:19:10.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:10 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:19:10.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:10 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:19:10.943 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:10 smithi003 ceph-mon[29576]: pgmap v761: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.39k op/s; 49218/295914 objects degraded (16.633%); 6/295914 objects misplaced (0.002%) 2024-03-21T14:19:10.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:10 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:19:10.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:10 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:19:10.943 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:10 smithi003 ceph-mon[40510]: pgmap v761: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.39k op/s; 49218/295914 objects degraded (16.633%); 6/295914 objects misplaced (0.002%) 2024-03-21T14:19:10.977 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:19:11.017 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:19:11.043 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:11.053 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:19:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:11.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:11.088 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:19:11.124 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:19:11.163 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:19:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:19:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:19:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-21T14:19:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:19:11.163 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:19:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:19:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:19:11.164 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:19:11.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:10 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:19:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:10 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:19:11.183 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:10 smithi118 ceph-mon[33862]: pgmap v761: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.39k op/s; 49218/295914 objects degraded (16.633%); 6/295914 objects misplaced (0.002%) 2024-03-21T14:19:11.212 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:11.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:11.316 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:19:11.316 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:19:11.349 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:11.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:11.485 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:11.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:11.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:11.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:11.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:11.763 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:11.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:11.865 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:11.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:11.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:11.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:11.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:11 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1751844835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:11 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1751844835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.033 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:12.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:12.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.172 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:12.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:11 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1751844835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:12.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:12.308 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:12.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:12.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:12.444 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:12.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:12.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:19:12.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:12.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:12.728 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:12.830 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:12.894 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:12.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:12.920 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[175187]: 2024-03-21T14:19:12.624+0000 7f7e7f8117c0 -1 osd.2 277 log_to_monitors true 2024-03-21T14:19:12.920 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[175187]: 2024-03-21T14:19:12.919+0000 7f7e70c69700 -1 osd.2 357 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: pgmap v762: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 763 B/s rd, 0 B/s wr, 1.25k op/s; 49218/295914 objects degraded (16.633%); 6/295914 objects misplaced (0.002%) 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: from='osd.2 v2:172.21.15.3:6809/3011075847' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:19:12.920 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: pgmap v762: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 763 B/s rd, 0 B/s wr, 1.25k op/s; 49218/295914 objects degraded (16.633%); 6/295914 objects misplaced (0.002%) 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: from='osd.2 v2:172.21.15.3:6809/3011075847' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:19:12.921 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:12 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:19:12.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:13.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:13.129 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:13.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:13.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: pgmap v762: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 6.0 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 763 B/s rd, 0 B/s wr, 1.25k op/s; 49218/295914 objects degraded (16.633%); 6/295914 objects misplaced (0.002%) 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: from='osd.2 v2:172.21.15.3:6809/3011075847' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:19:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:12 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-21T14:19:13.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:13.303 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:13.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:13.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:13.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:13.538 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:13.572 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:13.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:13.682 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:19:13.362124792Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-21T14:19:13.683 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:19:13.373164317Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-21T14:19:13.683 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:19:13.384109358Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-21T14:19:13.683 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:19:13.395271973Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-21T14:19:13.683 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=sqlstore.transactions t=2024-03-21T14:19:13.406555281Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=4 code="database is locked" 2024-03-21T14:19:13.711 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:13.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.839 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:13.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:13.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:13.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:13.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:13.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:14.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:14.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-mon[33862]: osdmap e357: 8 total, 7 up, 7 in 2024-03-21T14:19:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-mon[33862]: from='osd.2 v2:172.21.15.3:6809/3011075847' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:19:14.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:13 smithi118 ceph-mon[33862]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:19:14.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:14.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:19:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:19:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T14:19:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[29576]: osdmap e357: 8 total, 7 up, 7 in 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[29576]: from='osd.2 v2:172.21.15.3:6809/3011075847' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[29576]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[40510]: osdmap e357: 8 total, 7 up, 7 in 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[40510]: from='osd.2 v2:172.21.15.3:6809/3011075847' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:19:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:13 smithi003 ceph-mon[40510]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-03-21T14:19:14.318 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.343 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:14.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:14.444 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:14.458 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-21T14:19:14.458 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-21T14:19:14.470 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:14.478 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:14.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:14.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:14.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:14.721 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:14.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:14.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:14.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:14.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:14.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:14.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:14.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:15.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:15.098 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:15.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:15.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:19:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: pgmap v764: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.19k op/s; 44984/270255 objects degraded (16.645%); 6/270255 objects misplaced (0.002%) 2024-03-21T14:19:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:19:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:19:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T14:19:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: osd.2 v2:172.21.15.3:6809/3011075847 boot 2024-03-21T14:19:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: osdmap e358: 8 total, 8 up, 7 in 2024-03-21T14:19:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:19:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 44984/270255 objects degraded (16.645%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:15.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:14 smithi118 ceph-mon[33862]: osdmap e359: 8 total, 8 up, 7 in 2024-03-21T14:19:15.192 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: pgmap v764: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.19k op/s; 44984/270255 objects degraded (16.645%); 6/270255 objects misplaced (0.002%) 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: osd.2 v2:172.21.15.3:6809/3011075847 boot 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: osdmap e358: 8 total, 8 up, 7 in 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 44984/270255 objects degraded (16.645%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[40510]: osdmap e359: 8 total, 8 up, 7 in 2024-03-21T14:19:15.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: pgmap v764: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 2.19k op/s; 44984/270255 objects degraded (16.645%); 6/270255 objects misplaced (0.002%) 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: osd.2 v2:172.21.15.3:6809/3011075847 boot 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: osdmap e358: 8 total, 8 up, 7 in 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 44984/270255 objects degraded (16.645%), 22 pgs degraded, 61 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:15.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:14 smithi003 ceph-mon[29576]: osdmap e359: 8 total, 8 up, 7 in 2024-03-21T14:19:15.200 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:15.223 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:15.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:15.350 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:15.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:15.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:15.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:15.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:15.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:15.729 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:15.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:15.831 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:15.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:15.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:15.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:15.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:16.001 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: 10.4 scrub starts 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: 10.4 scrub ok 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: osdmap e360: 8 total, 8 up, 7 in 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13630 2024-03-21T14:19:16.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[29576]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 357'13470 2024-03-21T14:19:16.005 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:16.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: 10.4 scrub starts 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: 10.4 scrub ok 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: osdmap e360: 8 total, 8 up, 7 in 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13630 2024-03-21T14:19:16.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:15 smithi003 ceph-mon[40510]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 357'13470 2024-03-21T14:19:16.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: 10.4 scrub starts 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: 10.4 scrub ok 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: osdmap e360: 8 total, 8 up, 7 in 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13630 2024-03-21T14:19:16.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:15 smithi118 ceph-mon[33862]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 357'13470 2024-03-21T14:19:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:16.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:16.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:16.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:16.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:16.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:19:16.463 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:16.565 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:16.596 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:16.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:19:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:19:16.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.755 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:16.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:16.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:16.889 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:16.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:16.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:17.024 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: pgmap v767: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.43k op/s; 44984/270255 objects degraded (16.645%); 6/270255 objects misplaced (0.002%) 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 1.7 scrub starts 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13576 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 357'13660 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 1.7 scrub ok 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13910 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 357'13673 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13842 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13556 2024-03-21T14:19:17.102 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:19:17.103 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: pgmap v767: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.43k op/s; 44984/270255 objects degraded (16.645%); 6/270255 objects misplaced (0.002%) 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 1.7 scrub starts 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13576 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 357'13660 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 1.7 scrub ok 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13910 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 357'13673 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13842 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13556 2024-03-21T14:19:17.104 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:16 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:19:17.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:17.169 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: pgmap v767: 158 pgs: 7 active+clean+remapped, 22 active+undersized+degraded, 39 active+undersized, 90 active+clean; 5.5 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.43k op/s; 44984/270255 objects degraded (16.645%); 6/270255 objects misplaced (0.002%) 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 1.7 scrub starts 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13576 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 357'13660 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 1.7 scrub ok 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13910 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 357'13673 2024-03-21T14:19:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13842 2024-03-21T14:19:17.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 357'13556 2024-03-21T14:19:17.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:16 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:19:17.271 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:17.303 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:17.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:17.405 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:17.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:17.455 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:17.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:17.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:17.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:19:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:19:17.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:17.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:17.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:17.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:17.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:17.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:18.015 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:18.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.151 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:18.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:18.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:18.282 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:18.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:18.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: 3.0 scrub starts 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: 3.0 scrub ok 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: pgmap v769: 158 pgs: 117 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 22 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 10 active+undersized+degraded, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 42266/259236 objects degraded (16.304%); 5.7 MiB/s, 91 objects/s recovering 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:19:18.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:18 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: 3.0 scrub starts 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: 3.0 scrub ok 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: pgmap v769: 158 pgs: 117 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 22 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 10 active+undersized+degraded, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 42266/259236 objects degraded (16.304%); 5.7 MiB/s, 91 objects/s recovering 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:19:18.475 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: 3.0 scrub starts 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: 3.0 scrub ok 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: pgmap v769: 158 pgs: 117 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 22 active+undersized, 3 active+undersized+degraded+remapped+backfilling, 10 active+undersized+degraded, 1 active+clean+remapped, 2 active+undersized+degraded+remapped+backfill_toofull; 5.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 0 B/s wr, 791 op/s; 42266/259236 objects degraded (16.304%); 5.7 MiB/s, 91 objects/s recovering 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:19:18.476 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:18 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:19:18.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:18.590 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:18.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:18.723 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:18.825 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:18.832 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:18.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:18.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:223927 10.a e360/359 10:58fc7afc:::benchmark_data_smithi003_160672_object70551:head v 360'13720, mlcod=360'13718)", 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:18.885348+0000", 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0027025059999999999, 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027359490000000001, 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:18.893 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885348+0000", 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885348+0000", 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885349+0000", 2024-03-21T14:19:18.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4300000000000002e-07 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885355+0000", 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7640000000000002e-06 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885364+0000", 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0510000000000002e-06 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885373+0000", 2024-03-21T14:19:18.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8279999999999998e-06 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885580+0000", 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000206708 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.885600+0000", 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9542999999999999e-05 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888072+0000", 2024-03-21T14:19:18.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0024722490000000001 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:223921 10.5 e360/359 10:a9efbbec:::benchmark_data_smithi003_160672_object70545:head v 360'14110, mlcod=360'14107)", 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:18.886425+0000", 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001625221, 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001768031, 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:18.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886425+0000", 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886425+0000", 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886426+0000", 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1900000000000002e-07 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886432+0000", 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8030000000000002e-06 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886444+0000", 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1846e-05 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886450+0000", 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7619999999999998e-06 2024-03-21T14:19:18.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886524+0000", 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3354000000000003e-05 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.886544+0000", 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0579000000000001e-05 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:223929 10.7 e360/359 10:e917e102:::benchmark_data_smithi003_160672_object70553:head v 360'14161, mlcod=360'14159)", 2024-03-21T14:19:18.900 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:18.888045+0000", 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 5.186e-06, 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025054299999999999, 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888045+0000", 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888045+0000", 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888046+0000", 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8599999999999998e-07 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888052+0000", 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7080000000000002e-06 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888060+0000", 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1810000000000001e-06 2024-03-21T14:19:18.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888067+0000", 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7630000000000003e-06 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888184+0000", 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000117198 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888205+0000", 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1520999999999999e-05 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.903 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "MOSDPGPush(10.9 360/359 [PushOp(10:914ffdf7:::benchmark_data_smithi003_160672_object135344:head, version: 342'8330, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(10:914ffdf7:::benchmark_data_smithi003_160672_object135344:head@342'8330, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:18.888108+0000", 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9999428, 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027982000000000001, 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888108+0000", 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888108+0000", 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888108+0000", 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2300000000000001e-07 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888177+0000", 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8959000000000002e-05 2024-03-21T14:19:18.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888194+0000", 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6395000000000001e-05 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:18.888209+0000", 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5920999999999999e-05 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:19:18.906 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 4 2024-03-21T14:19:18.907 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:19:18.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.025 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:19.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:19.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:19.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:19.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.295 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:19.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:19.396 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:19.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:19 smithi003 ceph-mon[29576]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:19:19.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:19 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:19:19.412 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:19 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:19:19.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:19 smithi003 ceph-mon[40510]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:19:19.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:19 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:19:19.413 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:19 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:19:19.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:19 smithi118 ceph-mon[33862]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:19:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:19 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:19:19.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:19 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:19:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:19.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:19.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:19.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:19.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:19.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:19.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:19.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:19.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:19.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:20.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:20.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.138 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:20.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:20.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T14:19:20.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:20.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:20.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:20.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:20 smithi118 ceph-mon[33862]: pgmap v770: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 757 op/s; 42228/256743 objects degraded (16.448%); 4.4 MiB/s, 70 objects/s recovering 2024-03-21T14:19:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:20 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:19:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:20 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 42266/259236 objects degraded (16.304%), 18 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:20.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:20 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:19:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.455 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:20.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[29576]: pgmap v770: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 757 op/s; 42228/256743 objects degraded (16.448%); 4.4 MiB/s, 70 objects/s recovering 2024-03-21T14:19:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:19:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 42266/259236 objects degraded (16.304%), 18 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:19:20.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[40510]: pgmap v770: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 757 op/s; 42228/256743 objects degraded (16.448%); 4.4 MiB/s, 70 objects/s recovering 2024-03-21T14:19:20.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:19:20.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 42266/259236 objects degraded (16.304%), 18 pgs degraded, 32 pgs undersized (PG_DEGRADED) 2024-03-21T14:19:20.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:20 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:19:20.556 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:20.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:20.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:20.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:20.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:20.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:20.845 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:20.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:20.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:21.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:21.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:21.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:21.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:21.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:21.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:21.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:21.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:21.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:21.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:21.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:21.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:21.749 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:21.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:21.772 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:19:21.772 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:19:21.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:21.890 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:21.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:21.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:22.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.028 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:22.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:22.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:22.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:22.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.311 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:22.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:22.412 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:22.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:22.441 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:19:22.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:22.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:22.688 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:22.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[29576]: pgmap v771: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.3 KiB/s rd, 29 KiB/s wr, 595 op/s; 42228/256743 objects degraded (16.448%); 3.5 MiB/s, 55 objects/s recovering 2024-03-21T14:19:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:19:22.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:19:22.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/47064565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:19:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[40510]: pgmap v771: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.3 KiB/s rd, 29 KiB/s wr, 595 op/s; 42228/256743 objects degraded (16.448%); 3.5 MiB/s, 55 objects/s recovering 2024-03-21T14:19:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:19:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:19:22.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:22 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/47064565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:19:22.828 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:22.860 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:22.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:22 smithi118 ceph-mon[33862]: pgmap v771: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 5.2 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 4.3 KiB/s rd, 29 KiB/s wr, 595 op/s; 42228/256743 objects degraded (16.448%); 3.5 MiB/s, 55 objects/s recovering 2024-03-21T14:19:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:22 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:19:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:22 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:19:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:22 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/47064565' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:19:22.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:22.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:23.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:23.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:23.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:23.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:23.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:23.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:23.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:19:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/47064565' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:19:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[29576]: osdmap e361: 8 total, 8 up, 7 in 2024-03-21T14:19:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:19:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:19:23.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:19:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/47064565' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:19:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[40510]: osdmap e361: 8 total, 8 up, 7 in 2024-03-21T14:19:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:19:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:23 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:19:23.535 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:23.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:23.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:23.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:23 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:19:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:23 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/47064565' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:19:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:23 smithi118 ceph-mon[33862]: osdmap e361: 8 total, 8 up, 7 in 2024-03-21T14:19:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:23 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:19:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:23 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:19:23.710 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:23.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:23.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:23.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:23.954 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:23.962 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:19:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:23.990 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:23.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:24.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:24.126 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:24.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:24.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:24.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:24.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.410 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:24.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:24.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:24.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:24.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:24.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:24.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:24 smithi003 ceph-mon[29576]: pgmap v773: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.42k op/s; 37128/236241 objects degraded (15.716%); 17 MiB/s, 275 objects/s recovering 2024-03-21T14:19:24.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:24 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:19:24.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:24 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:19:24.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:24 smithi003 ceph-mon[40510]: pgmap v773: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.42k op/s; 37128/236241 objects degraded (15.716%); 17 MiB/s, 275 objects/s recovering 2024-03-21T14:19:24.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:24 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:19:24.676 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:24 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:19:24.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:24.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:24.792 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.830 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:24.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:24.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:24 smithi118 ceph-mon[33862]: pgmap v773: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 1.42k op/s; 37128/236241 objects degraded (15.716%); 17 MiB/s, 275 objects/s recovering 2024-03-21T14:19:24.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:24 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:19:24.933 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:24 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:19:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:24.968 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:24.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:24.976 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:25.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.106 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:25.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:25.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:25.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:25.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:25.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:25.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:25.494 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:25.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:25.634 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:25.667 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:25.667 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:25.667 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[40510]: osdmap e362: 8 total, 8 up, 7 in 2024-03-21T14:19:25.667 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 37128/236241 objects degraded (15.716%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[29576]: osdmap e362: 8 total, 8 up, 7 in 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:19:25.668 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:25 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 37128/236241 objects degraded (15.716%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:25.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:25.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:25.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:25.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:25.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:25.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:25.913 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:19:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:25 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:25 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:25 smithi118 ceph-mon[33862]: osdmap e362: 8 total, 8 up, 7 in 2024-03-21T14:19:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:25 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:19:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:25 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:19:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:25 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 37128/236241 objects degraded (15.716%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:25.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:25.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:25.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:25.956 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:19:25.998 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:19:26.039 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:26.041 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:19:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:26.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:26.082 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:19:26.130 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:19:26.172 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:19:26.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:26.209 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:26.213 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:19:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:19:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:19:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:19:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:19:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:19:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:19:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:19:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:19:26.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:26.358 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:26.460 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:26.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:19:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:19:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: pgmap v775: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 963 op/s; 37128/236241 objects degraded (15.716%); 14 MiB/s, 222 objects/s recovering 2024-03-21T14:19:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:19:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:19:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: osdmap e363: 8 total, 8 up, 7 in 2024-03-21T14:19:26.492 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1930031449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:26.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:26.565 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:26.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.643 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:26.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:26.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:19:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: pgmap v775: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 963 op/s; 37128/236241 objects degraded (15.716%); 14 MiB/s, 222 objects/s recovering 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: osdmap e363: 8 total, 8 up, 7 in 2024-03-21T14:19:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1930031449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:26.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:26.889 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:26.924 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:26.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: pgmap v775: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 4.7 KiB/s rd, 27 KiB/s wr, 963 op/s; 37128/236241 objects degraded (15.716%); 14 MiB/s, 222 objects/s recovering 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: osdmap e363: 8 total, 8 up, 7 in 2024-03-21T14:19:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1930031449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:27.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:19:27.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:27.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:27.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:27.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:27.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:27.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:27.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:27.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:27.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:27.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:27.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:27.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:27.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:27.561 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:27.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:27.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:27 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:19:27.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:27 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:19:27.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:19:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:19:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:27.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:27.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:27.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:27 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:19:27.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:27 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:19:27.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:27 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:19:27.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:27 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:19:27.829 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:27.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:27.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:27.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:27.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:28.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:28.109 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:28.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.143 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:28.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:28.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.268 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:28.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:28.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:28.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:28.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:28.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:28.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:28.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:28.684 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[29576]: pgmap v777: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.06k op/s; 32174/219594 objects degraded (14.652%); 37 MiB/s, 599 objects/s recovering 2024-03-21T14:19:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[29576]: 6.3 deep-scrub starts 2024-03-21T14:19:28.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[29576]: 6.3 deep-scrub ok 2024-03-21T14:19:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[40510]: pgmap v777: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.06k op/s; 32174/219594 objects degraded (14.652%); 37 MiB/s, 599 objects/s recovering 2024-03-21T14:19:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[40510]: 6.3 deep-scrub starts 2024-03-21T14:19:28.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:28 smithi003 ceph-mon[40510]: 6.3 deep-scrub ok 2024-03-21T14:19:28.785 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:28.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:28.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:28 smithi118 ceph-mon[33862]: pgmap v777: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.06k op/s; 32174/219594 objects degraded (14.652%); 37 MiB/s, 599 objects/s recovering 2024-03-21T14:19:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:28 smithi118 ceph-mon[33862]: 6.3 deep-scrub starts 2024-03-21T14:19:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:28 smithi118 ceph-mon[33862]: 6.3 deep-scrub ok 2024-03-21T14:19:28.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:28.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:28.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:29.064 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.096 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:29.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:29.197 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:29.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:29.323 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:29.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.357 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:29.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:29.458 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:29.592 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:29.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.629 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:29.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:29.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:29.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:29.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:29.871 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:29.906 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:29.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:30.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:30.043 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:30.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:30.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:30.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:30.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:30.311 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:30.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:30.412 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.449 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:30.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:30.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:30.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:30.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:30.732 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:30.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:30.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:30.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:30 smithi003 ceph-mon[29576]: pgmap v778: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.17k op/s; 32174/215724 objects degraded (14.914%); 16 MiB/s, 261 objects/s recovering 2024-03-21T14:19:30.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:30 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 32174/215724 objects degraded (14.914%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:30.830 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:30 smithi003 ceph-mon[29576]: osdmap e364: 8 total, 8 up, 7 in 2024-03-21T14:19:30.830 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:30 smithi003 ceph-mon[40510]: pgmap v778: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.17k op/s; 32174/215724 objects degraded (14.914%); 16 MiB/s, 261 objects/s recovering 2024-03-21T14:19:30.831 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:30 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 32174/215724 objects degraded (14.914%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:30.831 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:30 smithi003 ceph-mon[40510]: osdmap e364: 8 total, 8 up, 7 in 2024-03-21T14:19:30.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:30.882 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:30.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:30.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:30 smithi118 ceph-mon[33862]: pgmap v778: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.17k op/s; 32174/215724 objects degraded (14.914%); 16 MiB/s, 261 objects/s recovering 2024-03-21T14:19:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:30 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 32174/215724 objects degraded (14.914%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:30.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:30 smithi118 ceph-mon[33862]: osdmap e364: 8 total, 8 up, 7 in 2024-03-21T14:19:30.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:31.007 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:31.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:31.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:31.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.145 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:31.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:31.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:31.269 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:31.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:31.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:31.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:31.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:31.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:31.536 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:31.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:31.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:31.773 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:31.808 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:31.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:31.945 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:31.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:31.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:32.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.072 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:32.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:32.173 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:32.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:32.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:32.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:32.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:32.450 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:32.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:32.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:32.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:19:32.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:32.773 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:32.847 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:32 smithi118 ceph-mon[33862]: pgmap v780: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 992 op/s; 32174/215724 objects degraded (14.914%); 16 MiB/s, 262 objects/s recovering 2024-03-21T14:19:32.847 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:32 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:32.847 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:32.875 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:32.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:32.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:33.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:32 smithi003 ceph-mon[29576]: pgmap v780: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 992 op/s; 32174/215724 objects degraded (14.914%); 16 MiB/s, 262 objects/s recovering 2024-03-21T14:19:33.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:32 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:33.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:33.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:32 smithi003 ceph-mon[40510]: pgmap v780: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 992 op/s; 32174/215724 objects degraded (14.914%); 16 MiB/s, 262 objects/s recovering 2024-03-21T14:19:33.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:32 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:19:33.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:33.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.060 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:33.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:33.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:33.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:33.305 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:33.339 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:33.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:33.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.474 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:33.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:33.575 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:33.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:33.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:33.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:33 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:19:33.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:33 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:19:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:33 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:19:33.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:33 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:19:33.756 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:33.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:33.900 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:33.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:33.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:33 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:19:33.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:33 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:19:34.001 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.037 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:34.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:34.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:34.172 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:34.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:34.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:34.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:34.455 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:34.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:34.567 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T14:19:34.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:34.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:34.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:34.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:34.843 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:34.874 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:34.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:34.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:34 smithi118 ceph-mon[33862]: pgmap v781: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 27555/196887 objects degraded (13.995%); 29 MiB/s, 460 objects/s recovering 2024-03-21T14:19:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2808570459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:19:34.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:34 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:19:34.975 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:34 smithi003 ceph-mon[29576]: pgmap v781: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 27555/196887 objects degraded (13.995%); 29 MiB/s, 460 objects/s recovering 2024-03-21T14:19:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2808570459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:19:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:34 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:19:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:34 smithi003 ceph-mon[40510]: pgmap v781: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 27555/196887 objects degraded (13.995%); 29 MiB/s, 460 objects/s recovering 2024-03-21T14:19:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2808570459' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:19:35.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:34 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:19:35.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:35.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:35.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:35.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:35.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:35.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:35.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:35.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:35.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:35.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:35.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:35.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:35.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:19:35.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:19:35.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:19:35.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: osdmap e365: 8 total, 8 up, 7 in 2024-03-21T14:19:35.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 27555/196887 objects degraded (13.995%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:35.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: osdmap e366: 8 total, 8 up, 7 in 2024-03-21T14:19:35.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: 10.a starting backfill to osd.0 from (0'0,0'0] MAX to 364'14977 2024-03-21T14:19:35.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[29576]: 10.a continuing backfill to osd.2 from (346'10400,364'14977] MIN to 364'14977 2024-03-21T14:19:35.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:19:35.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:19:35.816 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:19:35.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: osdmap e365: 8 total, 8 up, 7 in 2024-03-21T14:19:35.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 27555/196887 objects degraded (13.995%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:35.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: osdmap e366: 8 total, 8 up, 7 in 2024-03-21T14:19:35.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: 10.a starting backfill to osd.0 from (0'0,0'0] MAX to 364'14977 2024-03-21T14:19:35.817 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:35 smithi003 ceph-mon[40510]: 10.a continuing backfill to osd.2 from (346'10400,364'14977] MIN to 364'14977 2024-03-21T14:19:35.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: osdmap e365: 8 total, 8 up, 7 in 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 27555/196887 objects degraded (13.995%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: osdmap e366: 8 total, 8 up, 7 in 2024-03-21T14:19:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: 10.a starting backfill to osd.0 from (0'0,0'0] MAX to 364'14977 2024-03-21T14:19:35.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:35 smithi118 ceph-mon[33862]: 10.a continuing backfill to osd.2 from (346'10400,364'14977] MIN to 364'14977 2024-03-21T14:19:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:35.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:35.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:36.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:36.087 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:36.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:36.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:36.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:36.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:36.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:36.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-21T14:19:36.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:36.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:36.522 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:36.552 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:36.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:36.622 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:19:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:19:36.624 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:19:36.624 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:19:36.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:19:36.624 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219636 8.3 8:c2dee2e9:::benchmark_data_smithi003_110242_object88719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.068135789, 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99404183000000002, 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219636 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550756+0000", 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550757+0000", 2024-03-21T14:19:36.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.02e-07 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550761+0000", 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0810000000000004e-06 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550771+0000", 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4530000000000008e-06 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550778+0000", 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5909999999999998e-06 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550827+0000", 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8690000000000003e-05 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.550833+0000", 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1410000000000002e-06 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539253+0000", 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98841965700000001 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539380+0000", 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012699099999999999 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.539666+0000", 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028561599999999998 2024-03-21T14:19:36.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.542830+0000", 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0031648290000000001 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544576+0000", 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001746039 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544691+0000", 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011476000000000001 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544746+0000", 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4617999999999998e-05 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544798+0000", 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2462e-05 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219640 8.7 8:e03a6842:::benchmark_data_smithi003_110242_object88723:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.05816600700001, 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.000192481, 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219640 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560726+0000", 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560728+0000", 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.548e-06 2024-03-21T14:19:36.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560734+0000", 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5189999999999998e-06 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560752+0000", 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7683999999999999e-05 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560764+0000", 2024-03-21T14:19:36.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1558e-05 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560819+0000", 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5161000000000003e-05 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.560826+0000", 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8390000000000001e-06 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551363+0000", 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99053792699999998 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551445+0000", 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1199999999999995e-05 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.551758+0000", 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00031323899999999999 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558010+0000", 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062516619999999998 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558022+0000", 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2751e-05 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560813+0000", 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027905650000000001 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560868+0000", 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5031999999999998e-05 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560919+0000", 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0796e-05 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.638 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219647 8.7 8:e325f337:::benchmark_data_smithi003_110242_object88730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.05276820900002, 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0098845869999999, 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219647 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566124+0000", 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566125+0000", 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7699999999999992e-07 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566131+0000", 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5400000000000003e-06 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566142+0000", 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1029000000000001e-05 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566150+0000", 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8179999999999993e-06 2024-03-21T14:19:36.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566259+0000", 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109064 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.566268+0000", 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1850000000000004e-06 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554008+0000", 2024-03-21T14:19:36.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98774052300000004 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554113+0000", 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104929 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.554390+0000", 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027710000000000001 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.560968+0000", 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0065778310000000001 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569693+0000", 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0087252330000000006 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575871+0000", 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0061776979999999997 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575961+0000", 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9869000000000003e-05 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576009+0000", 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7790999999999997e-05 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219661 8.3 8:c78a9c88:::benchmark_data_smithi003_110242_object88744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.04117001700001, 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96728133199999999, 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219661 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577722+0000", 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577723+0000", 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.218e-06 2024-03-21T14:19:36.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577729+0000", 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4469999999999997e-06 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577741+0000", 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1929e-05 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577749+0000", 2024-03-21T14:19:36.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0420000000000003e-06 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577810+0000", 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0661000000000001e-05 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.577818+0000", 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5720000000000003e-06 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541402+0000", 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96358385700000004 2024-03-21T14:19:36.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541513+0000", 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110608 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.541765+0000", 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025253 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544855+0000", 2024-03-21T14:19:36.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0030902249999999998 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544866+0000", 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0219e-05 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544900+0000", 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4406999999999998e-05 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.544963+0000", 2024-03-21T14:19:36.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3510999999999996e-05 2024-03-21T14:19:36.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545004+0000", 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0105999999999999e-05 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219664 8.3 8:c1f39ff4:::benchmark_data_smithi003_110242_object88747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.03923473899999, 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96554897500000003, 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.653 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219664 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579657+0000", 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579659+0000", 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0979999999999999e-06 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579663+0000", 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6879999999999998e-06 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579675+0000", 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1701e-05 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579684+0000", 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3670000000000003e-06 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579738+0000", 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3353999999999998e-05 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.656 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.579746+0000", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1349999999999992e-06 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.542881+0000", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96313484800000004 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.543021+0000", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014037899999999999 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,5", 2024-03-21T14:19:36.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.543251+0000", 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000229776 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545019+0000", 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001768541 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545064+0000", 2024-03-21T14:19:36.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4567999999999997e-05 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545127+0000", 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3037e-05 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545169+0000", 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1650000000000003e-05 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.545206+0000", 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7833000000000002e-05 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219669 8.7 8:e640187a:::benchmark_data_smithi003_110242_object88752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.03638277699997, 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99373624599999999, 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.660 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219669 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582509+0000", 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582510+0000", 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3500000000000005e-07 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582515+0000", 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6190000000000002e-06 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582527+0000", 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1663e-05 2024-03-21T14:19:36.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582535+0000", 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9719999999999992e-06 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582589+0000", 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4258000000000001e-05 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.582594+0000", 2024-03-21T14:19:36.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5650000000000001e-06 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556001+0000", 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.97340653200000005 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556118+0000", 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116953 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.556396+0000", 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00027774400000000002 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569603+0000", 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013207429 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569729+0000", 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000126198 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576017+0000", 2024-03-21T14:19:36.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0062882509999999999 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576174+0000", 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015680300000000001 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576246+0000", 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1423999999999999e-05 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219677 8.7 8:e3d98eec:::benchmark_data_smithi003_110242_object88760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.030411555, 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98785003400000004, 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219677 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588481+0000", 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588482+0000", 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7999999999999993e-07 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588487+0000", 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2270000000000001e-06 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588498+0000", 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1511e-05 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588507+0000", 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6759999999999996e-06 2024-03-21T14:19:36.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588580+0000", 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3046000000000005e-05 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.588589+0000", 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8140000000000007e-06 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.670 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558336+0000", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96974661200000001 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558447+0000", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111644 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.558737+0000", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028978899999999999 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569659+0000", 2024-03-21T14:19:36.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010921759 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.569764+0000", 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105388 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576252+0000", 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064880720000000001 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576293+0000", 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1300000000000001e-05 2024-03-21T14:19:36.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.576331+0000", 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7215999999999997e-05 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15132.0:219708 8.7 8:e065f095:::benchmark_data_smithi003_110242_object88791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 446.00145722000002, 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.96333364799999999, 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.673 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15132", 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 219708 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617435+0000", 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617436+0000", 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2160000000000001e-06 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617442+0000", 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4369999999999998e-06 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617453+0000", 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1109e-05 2024-03-21T14:19:36.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617460+0000", 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6179999999999997e-06 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617520+0000", 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9939000000000003e-05 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:10.617530+0000", 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1679999999999993e-06 2024-03-21T14:19:36.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561003+0000", 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94347382000000002 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561126+0000", 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012228799999999999 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,3", 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.561456+0000", 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033023800000000001 2024-03-21T14:19:36.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575680+0000", 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014224647999999999 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.575752+0000", 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1209000000000002e-05 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580677+0000", 2024-03-21T14:19:36.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0049251019999999998 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580721+0000", 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4159000000000002e-05 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:12:11.580769+0000", 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7697000000000001e-05 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.679 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146548 9.0 9:0dd0b1d4:::benchmark_data_smithi003_135814_object5814:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.717159+0000", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 282.90173349100002, 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.029799326, 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146548 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717159+0000", 2024-03-21T14:19:36.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717159+0000", 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717160+0000", 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4890000000000001e-06 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717167+0000", 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1060000000000002e-06 2024-03-21T14:19:36.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717180+0000", 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2979000000000001e-05 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717189+0000", 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0010000000000007e-06 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.732497+0000", 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.015307731999999999 2024-03-21T14:19:36.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.732505+0000", 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4260000000000003e-06 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.741415+0000", 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0089090789999999 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.741552+0000", 2024-03-21T14:19:36.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013709200000000001 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0", 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.741683+0000", 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013169899999999999 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.745161+0000", 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003477876 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746810+0000", 2024-03-21T14:19:36.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016483299999999999 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746868+0000", 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8176e-05 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746958+0000", 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0340999999999994e-05 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146549 9.2 9:444f3cf6:::benchmark_data_smithi003_135814_object5815:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 282.901686337, 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.035862668, 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146549 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717206+0000", 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4099999999999999e-07 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717209+0000", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.915e-06 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717216+0000", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9530000000000002e-06 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.717222+0000", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1330000000000004e-06 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.727749+0000", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010526862999999999 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.729296+0000", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001546415 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.745813+0000", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0165168 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746169+0000", 2024-03-21T14:19:36.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00035624300000000001 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.746303+0000", 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013459600000000001 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750301+0000", 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039974140000000003 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.752966+0000", 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026654920000000002 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753001+0000", 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.4690000000000002e-05 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753069+0000", 2024-03-21T14:19:36.691 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7613000000000005e-05 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146566 9.2 9:478951ba:::benchmark_data_smithi003_135814_object5832:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-21T14:19:36.693 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.738747+0000", 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 282.88014490900002, 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0144947470000001, 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146566 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738747+0000", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738747+0000", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738749+0000", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5570000000000001e-06 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738755+0000", 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0839999999999997e-06 2024-03-21T14:19:36.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738766+0000", 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0776e-05 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738772+0000", 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2129999999999996e-06 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738824+0000", 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1814000000000002e-05 2024-03-21T14:19:36.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.738830+0000", 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8089999999999998e-06 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.748308+0000", 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0094779439999999 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.748412+0000", 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000104036 2024-03-21T14:19:36.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.748522+0000", 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011043 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.751822+0000", 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033004729999999999 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753120+0000", 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00129707 2024-03-21T14:19:36.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753161+0000", 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1202000000000003e-05 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753242+0000", 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1339000000000002e-05 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146570 9.2 9:454dc127:::benchmark_data_smithi003_135814_object5836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-21T14:19:36.699 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.742486+0000", 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 282.87640622800001, 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0111842019999999, 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146570 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742486+0000", 2024-03-21T14:19:36.700 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742486+0000", 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742487+0000", 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.229e-06 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742492+0000", 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9459999999999997e-06 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.701 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742502+0000", 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8090000000000004e-06 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742508+0000", 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9020000000000004e-06 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742560+0000", 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2111000000000003e-05 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.702 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.742567+0000", 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0500000000000003e-06 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750335+0000", 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.007768346 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750438+0000", 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000103065 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.703 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 7", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750506+0000", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7508000000000005e-05 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.751919+0000", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001412817 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753569+0000", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016506050000000001 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753600+0000", 2024-03-21T14:19:36.704 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0420999999999999e-05 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.753670+0000", 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0393000000000002e-05 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:146575 9.0 9:09bb5e96:::benchmark_data_smithi003_135814_object5841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:14:53.750996+0000", 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 282.867896347, 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99901830800000002, 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.705 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 146575 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.750996+0000", 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.706 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.750996+0000", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.750998+0000", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.759e-06 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751002+0000", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8230000000000002e-06 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751014+0000", 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1202e-05 2024-03-21T14:19:36.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751020+0000", 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8640000000000001e-06 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751064+0000", 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4823000000000001e-05 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:53.751072+0000", 2024-03-21T14:19:36.708 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2459999999999997e-06 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.747328+0000", 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.996256751 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.747439+0000", 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011100899999999999 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0", 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.747553+0000", 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011396999999999999 2024-03-21T14:19:36.709 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.749343+0000", 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0017892769999999999 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.749904+0000", 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00056149800000000001 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.749945+0000", 2024-03-21T14:19:36.710 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.1069999999999998e-05 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:14:54.750014+0000", 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9016000000000005e-05 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268614 9.f 9:f836142a:::benchmark_data_smithi003_135814_object127880:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863347+0000", 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.75554492500001, 2024-03-21T14:19:36.711 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0564403689999999, 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268614 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863347+0000", 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863347+0000", 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863349+0000", 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2589999999999999e-06 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863352+0000", 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8670000000000004e-06 2024-03-21T14:19:36.713 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863362+0000", 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6830000000000003e-06 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863372+0000", 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8260000000000005e-06 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.714 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864619+0000", 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012469849999999999 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864629+0000", 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0193999999999999e-05 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869854+0000", 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052248629999999997 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.715 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869864+0000", 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0397999999999999e-05 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.917770+0000", 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.047905265 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.919788+0000", 2024-03-21T14:19:36.716 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020180290000000002 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268616 9.f 9:f6a2344d:::benchmark_data_smithi003_135814_object127882:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.75550039699999, 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0566032329999999, 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.717 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268616 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.718 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863392+0000", 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7899999999999999e-07 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863395+0000", 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8279999999999999e-06 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863402+0000", 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3709999999999996e-06 2024-03-21T14:19:36.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863407+0000", 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9529999999999996e-06 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864661+0000", 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00125344 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.720 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864667+0000", 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9050000000000002e-06 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869887+0000", 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052205400000000001 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.869894+0000", 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6959999999999999e-06 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.721 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.919906+0000", 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0500118490000001 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.919995+0000", 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9172000000000006e-05 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268618 9.6 9:6081da32:::benchmark_data_smithi003_135814_object127884:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.722 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.75547724500001, 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.047590408, 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268618 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.723 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863415+0000", 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.46e-07 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.724 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863418+0000", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1940000000000001e-06 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863423+0000", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4079999999999997e-06 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863429+0000", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7649999999999999e-06 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.725 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864328+0000", 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00089905200000000001 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864335+0000", 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0129999999999996e-06 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876237+0000", 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011902233 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.726 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876247+0000", 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8160000000000005e-06 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.908396+0000", 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.032149414 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911005+0000", 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0026091669999999999 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.727 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268622 9.6 9:654a3d90:::benchmark_data_smithi003_135814_object127888:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863436+0000", 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.75545578500001, 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0484016620000001, 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268622 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.728 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863436+0000", 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863436+0000", 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863437+0000", 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8299999999999998e-07 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.729 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863439+0000", 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1710000000000001e-06 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863445+0000", 2024-03-21T14:19:36.730 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5140000000000001e-06 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863450+0000", 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3009999999999997e-06 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864386+0000", 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00093625900000000005 2024-03-21T14:19:36.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864392+0000", 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6679999999999998e-06 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876274+0000", 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011882622000000001 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876279+0000", 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0180000000000001e-06 2024-03-21T14:19:36.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.733 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.733 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911726+0000", 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.035446259 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911838+0000", 2024-03-21T14:19:36.734 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011246700000000001 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268634 9.b 9:d4935d58:::benchmark_data_smithi003_135814_object127900:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.863456+0000", 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.755435826, 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0497115850000001, 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.735 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268634 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863456+0000", 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863456+0000", 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.736 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863457+0000", 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7099999999999997e-07 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863459+0000", 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4059999999999999e-06 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863465+0000", 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.327e-06 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.737 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.863470+0000", 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0669999999999999e-06 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864720+0000", 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00125074 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864726+0000", 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5990000000000001e-06 2024-03-21T14:19:36.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.870446+0000", 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0057203009999999997 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.870455+0000", 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1789999999999997e-06 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.902031+0000", 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0315752300000001 2024-03-21T14:19:36.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.902166+0000", 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013569 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5", 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.902575+0000", 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040849799999999998 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913018+0000", 2024-03-21T14:19:36.740 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010443295 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913034+0000", 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5814000000000001e-05 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913086+0000", 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1659000000000001e-05 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913168+0000", 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2409000000000003e-05 2024-03-21T14:19:36.741 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268638 9.6 9:651f0ab7:::benchmark_data_smithi003_135814_object127904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.864113+0000", 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.75477955400001, 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0622911719999999, 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268638 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864113+0000", 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864113+0000", 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.743 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864114+0000", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7700000000000003e-07 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864118+0000", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.831e-06 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864129+0000", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0737000000000001e-05 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864138+0000", 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5420000000000005e-06 2024-03-21T14:19:36.744 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864623+0000", 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048560200000000002 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting_for_map not empty", 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.864631+0000", 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9389999999999997e-06 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876315+0000", 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011683567000000001 2024-03-21T14:19:36.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.876320+0000", 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7609999999999998e-06 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.911865+0000", 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0355454129999999 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.912392+0000", 2024-03-21T14:19:36.746 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00052676199999999996 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5", 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.912628+0000", 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00023668499999999999 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926185+0000", 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.013556456 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.747 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926228+0000", 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3429e-05 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926310+0000", 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1829999999999994e-05 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.926404+0000", 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3740999999999996e-05 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.748 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:268641 9.b 9:dd564974:::benchmark_data_smithi003_135814_object127907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:16:20.877108+0000", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 195.74178381900001, 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0432787109999999, 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 268641 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877108+0000", 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877108+0000", 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877110+0000", 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.668e-06 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877118+0000", 2024-03-21T14:19:36.750 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4699999999999996e-06 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877127+0000", 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9329999999999996e-06 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877134+0000", 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.7229999999999998e-06 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877185+0000", 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0804999999999999e-05 2024-03-21T14:19:36.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:20.877193+0000", 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8250000000000005e-06 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913205+0000", 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0360120150000001 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913311+0000", 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105954 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.752 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5", 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.913425+0000", 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113713 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920031+0000", 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066069259999999999 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920240+0000", 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020825800000000001 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.753 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920294+0000", 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4106e-05 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:16:21.920387+0000", 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3314999999999997e-05 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:19:36.754 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:19:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:36.755 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:36.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:36.856 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:36.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:36.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:36.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:36 smithi118 ceph-mon[33862]: pgmap v784: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s; 27555/196887 objects degraded (13.995%); 19 MiB/s, 311 objects/s recovering 2024-03-21T14:19:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:36 smithi118 ceph-mon[33862]: 10.4 starting backfill to osd.0 from (0'0,0'0] MAX to 364'17460 2024-03-21T14:19:36.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:36 smithi118 ceph-mon[33862]: osdmap e367: 8 total, 8 up, 7 in 2024-03-21T14:19:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-mon[29576]: pgmap v784: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s; 27555/196887 objects degraded (13.995%); 19 MiB/s, 311 objects/s recovering 2024-03-21T14:19:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-mon[29576]: 10.4 starting backfill to osd.0 from (0'0,0'0] MAX to 364'17460 2024-03-21T14:19:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-mon[29576]: osdmap e367: 8 total, 8 up, 7 in 2024-03-21T14:19:37.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-mon[40510]: pgmap v784: 158 pgs: 150 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.05k op/s; 27555/196887 objects degraded (13.995%); 19 MiB/s, 311 objects/s recovering 2024-03-21T14:19:37.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-mon[40510]: 10.4 starting backfill to osd.0 from (0'0,0'0] MAX to 364'17460 2024-03-21T14:19:37.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:36 smithi003 ceph-mon[40510]: osdmap e367: 8 total, 8 up, 7 in 2024-03-21T14:19:37.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:37.041 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:37.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:37.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.178 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:37.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:37.182 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:37.279 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:37.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:37.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:37.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:37.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:37.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:37.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:19:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:19:37.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:37.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:37.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:37.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:37.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:37 smithi003 ceph-mon[29576]: 1.9 scrub starts 2024-03-21T14:19:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:37 smithi003 ceph-mon[29576]: 1.9 scrub ok 2024-03-21T14:19:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:37 smithi003 ceph-mon[29576]: osdmap e368: 8 total, 8 up, 7 in 2024-03-21T14:19:38.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:37 smithi003 ceph-mon[40510]: 1.9 scrub starts 2024-03-21T14:19:38.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:37 smithi003 ceph-mon[40510]: 1.9 scrub ok 2024-03-21T14:19:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:37 smithi003 ceph-mon[40510]: osdmap e368: 8 total, 8 up, 7 in 2024-03-21T14:19:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:38.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:38.121 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:38.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:37 smithi118 ceph-mon[33862]: 1.9 scrub starts 2024-03-21T14:19:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:37 smithi118 ceph-mon[33862]: 1.9 scrub ok 2024-03-21T14:19:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:37 smithi118 ceph-mon[33862]: osdmap e368: 8 total, 8 up, 7 in 2024-03-21T14:19:38.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.295 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:38.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:38.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:38.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:38.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:38.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:38.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:38.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:38.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:38.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:38.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:38.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:38.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:38.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:38.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:38.997 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[29576]: pgmap v787: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 22417/184572 objects degraded (12.145%); 7588/184572 objects misplaced (4.111%); 49 MiB/s, 786 objects/s recovering 2024-03-21T14:19:38.998 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:19:38.998 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[29576]: osdmap e369: 8 total, 8 up, 7 in 2024-03-21T14:19:38.998 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:38.998 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[40510]: pgmap v787: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 22417/184572 objects degraded (12.145%); 7588/184572 objects misplaced (4.111%); 49 MiB/s, 786 objects/s recovering 2024-03-21T14:19:38.998 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:19:38.998 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:38 smithi003 ceph-mon[40510]: osdmap e369: 8 total, 8 up, 7 in 2024-03-21T14:19:39.096 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:39.125 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:38 smithi118 ceph-mon[33862]: pgmap v787: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 0 B/s wr, 1.02k op/s; 22417/184572 objects degraded (12.145%); 7588/184572 objects misplaced (4.111%); 49 MiB/s, 786 objects/s recovering 2024-03-21T14:19:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:38 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:19:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:38 smithi118 ceph-mon[33862]: osdmap e369: 8 total, 8 up, 7 in 2024-03-21T14:19:39.227 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:39.261 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:39.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:39.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:39.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.522 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:39.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:39.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:39.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:39.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:39.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:39.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:39.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:39.904 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:39.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:39.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:39 smithi003 ceph-mon[29576]: 10.b scrub starts 2024-03-21T14:19:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:39 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 22417/184572 objects degraded (12.145%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:39 smithi003 ceph-mon[29576]: osdmap e370: 8 total, 8 up, 7 in 2024-03-21T14:19:40.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:39 smithi003 ceph-mon[40510]: 10.b scrub starts 2024-03-21T14:19:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:39 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 22417/184572 objects degraded (12.145%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:39 smithi003 ceph-mon[40510]: osdmap e370: 8 total, 8 up, 7 in 2024-03-21T14:19:40.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:40.078 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:40.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:39 smithi118 ceph-mon[33862]: 10.b scrub starts 2024-03-21T14:19:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:39 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 22417/184572 objects degraded (12.145%), 8 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:39 smithi118 ceph-mon[33862]: osdmap e370: 8 total, 8 up, 7 in 2024-03-21T14:19:40.215 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:40.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:40.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:40.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:40.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:40.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:40.467 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:40.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:40.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:40.607 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:40.709 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:40.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:40.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:40.885 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:40.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:40.987 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:40 smithi003 ceph-mon[29576]: pgmap v789: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s; 22417/181668 objects degraded (12.340%); 7588/181668 objects misplaced (4.177%); 42 MiB/s, 670 objects/s recovering 2024-03-21T14:19:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:40 smithi003 ceph-mon[29576]: 10.b scrub ok 2024-03-21T14:19:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:40 smithi003 ceph-mon[40510]: pgmap v789: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s; 22417/181668 objects degraded (12.340%); 7588/181668 objects misplaced (4.177%); 42 MiB/s, 670 objects/s recovering 2024-03-21T14:19:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:40 smithi003 ceph-mon[40510]: 10.b scrub ok 2024-03-21T14:19:41.023 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:41.125 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:41.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:40 smithi118 ceph-mon[33862]: pgmap v789: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s; 22417/181668 objects degraded (12.340%); 7588/181668 objects misplaced (4.177%); 42 MiB/s, 670 objects/s recovering 2024-03-21T14:19:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:40 smithi118 ceph-mon[33862]: 10.b scrub ok 2024-03-21T14:19:41.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:41.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:41.383 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:41.416 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:41.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:41.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:41.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:41.674 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:41.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:41.809 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:41.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:41.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:41.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:41 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:19:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:41 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:19:42.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:41 smithi003 ceph-mon[29576]: osdmap e371: 8 total, 8 up, 7 in 2024-03-21T14:19:42.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:41 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:19:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:41 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:19:42.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:41 smithi003 ceph-mon[40510]: osdmap e371: 8 total, 8 up, 7 in 2024-03-21T14:19:42.049 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:42.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:42.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:41 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:19:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:41 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:19:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:41 smithi118 ceph-mon[33862]: osdmap e371: 8 total, 8 up, 7 in 2024-03-21T14:19:42.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:42.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:42.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:42.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:42.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:42.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:42.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:42.632 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:42.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:42.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:42.759 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:42.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:42.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:42.887 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:42.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:42.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:42.989 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:42 smithi003 ceph-mon[29576]: 10.2 scrub starts 2024-03-21T14:19:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:42 smithi003 ceph-mon[29576]: pgmap v792: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s; 22417/181668 objects degraded (12.340%); 7588/181668 objects misplaced (4.177%); 42 MiB/s, 675 objects/s recovering 2024-03-21T14:19:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:42 smithi003 ceph-mon[29576]: osdmap e372: 8 total, 8 up, 7 in 2024-03-21T14:19:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:42 smithi003 ceph-mon[40510]: 10.2 scrub starts 2024-03-21T14:19:43.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:42 smithi003 ceph-mon[40510]: pgmap v792: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s; 22417/181668 objects degraded (12.340%); 7588/181668 objects misplaced (4.177%); 42 MiB/s, 675 objects/s recovering 2024-03-21T14:19:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:42 smithi003 ceph-mon[40510]: osdmap e372: 8 total, 8 up, 7 in 2024-03-21T14:19:43.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:43.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:43.158 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:42 smithi118 ceph-mon[33862]: 10.2 scrub starts 2024-03-21T14:19:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:42 smithi118 ceph-mon[33862]: pgmap v792: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 148 active+clean, 3 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.08k op/s; 22417/181668 objects degraded (12.340%); 7588/181668 objects misplaced (4.177%); 42 MiB/s, 675 objects/s recovering 2024-03-21T14:19:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:42 smithi118 ceph-mon[33862]: osdmap e372: 8 total, 8 up, 7 in 2024-03-21T14:19:43.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.298 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:43.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:43.430 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:43.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:43.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.560 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:43.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:43.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:43.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:43.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:43.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:43.968 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:43.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[29576]: 10.2 scrub ok 2024-03-21T14:19:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:19:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:19:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[29576]: 3.f deep-scrub starts 2024-03-21T14:19:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[29576]: 3.f deep-scrub ok 2024-03-21T14:19:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[40510]: 10.2 scrub ok 2024-03-21T14:19:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:19:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:19:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[40510]: 3.f deep-scrub starts 2024-03-21T14:19:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:43 smithi003 ceph-mon[40510]: 3.f deep-scrub ok 2024-03-21T14:19:44.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:44.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:43 smithi118 ceph-mon[33862]: 10.2 scrub ok 2024-03-21T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:43 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:43 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:43 smithi118 ceph-mon[33862]: 3.f deep-scrub starts 2024-03-21T14:19:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:43 smithi118 ceph-mon[33862]: 3.f deep-scrub ok 2024-03-21T14:19:44.206 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:44.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:44.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:44.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:44.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:44.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:44.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:44.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.660 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:44.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:44.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:44.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:44.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:44.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:44.921 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:44.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:44.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[29576]: pgmap v794: 158 pgs: 1 active+remapped+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 901 B/s rd, 0 B/s wr, 896 op/s; 18144/166407 objects degraded (10.903%); 6979/166407 objects misplaced (4.194%); 28 MiB/s, 441 objects/s recovering 2024-03-21T14:19:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:19:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:19:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 18144/166407 objects degraded (10.903%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[40510]: pgmap v794: 158 pgs: 1 active+remapped+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 901 B/s rd, 0 B/s wr, 896 op/s; 18144/166407 objects degraded (10.903%); 6979/166407 objects misplaced (4.194%); 28 MiB/s, 441 objects/s recovering 2024-03-21T14:19:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:19:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:19:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:44 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 18144/166407 objects degraded (10.903%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:45.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:45.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:45.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:45.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:45.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:44 smithi118 ceph-mon[33862]: pgmap v794: 158 pgs: 1 active+remapped+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 901 B/s rd, 0 B/s wr, 896 op/s; 18144/166407 objects degraded (10.903%); 6979/166407 objects misplaced (4.194%); 28 MiB/s, 441 objects/s recovering 2024-03-21T14:19:45.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:44 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:19:45.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:44 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:19:45.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:44 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 18144/166407 objects degraded (10.903%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:45.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:45.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:45.318 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:45.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:45.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:45.451 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:45.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:45.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:45.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:45.718 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:45.820 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:45.858 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:45.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:45.961 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:46.004 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:46.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T14:19:46.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:46.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:46.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.315 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:46.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:46.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:46.577 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:46.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:46.749 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:19:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:19:46.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:46.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:46.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:46.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:46.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:47.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-mon[29576]: pgmap v795: 158 pgs: 1 active+remapped+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 848 op/s; 18144/166407 objects degraded (10.903%); 6979/166407 objects misplaced (4.194%); 26 MiB/s, 418 objects/s recovering 2024-03-21T14:19:47.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:19:47.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:19:47.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-mon[40510]: pgmap v795: 158 pgs: 1 active+remapped+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 848 op/s; 18144/166407 objects degraded (10.903%); 6979/166407 objects misplaced (4.194%); 26 MiB/s, 418 objects/s recovering 2024-03-21T14:19:47.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:19:47.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:46 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:19:47.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:47.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.046 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:47.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:47.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:47.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:46 smithi118 ceph-mon[33862]: pgmap v795: 158 pgs: 1 active+remapped+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull; 3.4 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 848 op/s; 18144/166407 objects degraded (10.903%); 6979/166407 objects misplaced (4.194%); 26 MiB/s, 418 objects/s recovering 2024-03-21T14:19:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:46 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:19:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:46 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:19:47.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:47.315 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:47.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:47.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:47.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:47.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:47.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:47.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:19:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:19:47.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:47.751 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:47.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:47.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:47.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:47.907 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:47.908 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:48.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:48.061 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:19:48.181 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:48.292 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:48.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.366 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:48.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:48.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:48.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.499 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:48.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:48.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:48.645 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:48.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:48.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:48.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:48.934 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:48.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:48.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:49.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:49.066 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:49.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:49.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[29576]: 5.17 deep-scrub starts 2024-03-21T14:19:49.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[29576]: pgmap v796: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 1.22k op/s; 14794/153504 objects degraded (9.638%); 6259/153504 objects misplaced (4.077%); 33 MiB/s, 530 objects/s recovering 2024-03-21T14:19:49.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[29576]: 5.17 deep-scrub ok 2024-03-21T14:19:49.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[29576]: 5.9 deep-scrub starts 2024-03-21T14:19:49.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[29576]: 5.9 deep-scrub ok 2024-03-21T14:19:49.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:49.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[40510]: 5.17 deep-scrub starts 2024-03-21T14:19:49.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[40510]: pgmap v796: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 1.22k op/s; 14794/153504 objects degraded (9.638%); 6259/153504 objects misplaced (4.077%); 33 MiB/s, 530 objects/s recovering 2024-03-21T14:19:49.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[40510]: 5.17 deep-scrub ok 2024-03-21T14:19:49.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[40510]: 5.9 deep-scrub starts 2024-03-21T14:19:49.115 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:48 smithi003 ceph-mon[40510]: 5.9 deep-scrub ok 2024-03-21T14:19:49.168 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:48 smithi118 ceph-mon[33862]: 5.17 deep-scrub starts 2024-03-21T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:48 smithi118 ceph-mon[33862]: pgmap v796: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 1.22k op/s; 14794/153504 objects degraded (9.638%); 6259/153504 objects misplaced (4.077%); 33 MiB/s, 530 objects/s recovering 2024-03-21T14:19:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:48 smithi118 ceph-mon[33862]: 5.17 deep-scrub ok 2024-03-21T14:19:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:48 smithi118 ceph-mon[33862]: 5.9 deep-scrub starts 2024-03-21T14:19:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:48 smithi118 ceph-mon[33862]: 5.9 deep-scrub ok 2024-03-21T14:19:49.191 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:49.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:49.293 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:49.326 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:49.373 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:19:49.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:19:49.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:19:49.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:19:49.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:19:49.374 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:19:49.383 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:19:49.413 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:19:49.415 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:19:49.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:49.453 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:49.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.581 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:49.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:49.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:49.706 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:49.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:49.846 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:49.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:49.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:49.983 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:49.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[29576]: 10.0 scrub starts 2024-03-21T14:19:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[29576]: osdmap e373: 8 total, 8 up, 7 in 2024-03-21T14:19:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3318351990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:50.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 14794/153504 objects degraded (9.638%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:50.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[29576]: osdmap e374: 8 total, 8 up, 7 in 2024-03-21T14:19:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[40510]: 10.0 scrub starts 2024-03-21T14:19:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[40510]: osdmap e373: 8 total, 8 up, 7 in 2024-03-21T14:19:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3318351990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 14794/153504 objects degraded (9.638%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:50.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:49 smithi003 ceph-mon[40510]: osdmap e374: 8 total, 8 up, 7 in 2024-03-21T14:19:50.084 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:50.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.113 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:50.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:49 smithi118 ceph-mon[33862]: 10.0 scrub starts 2024-03-21T14:19:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:49 smithi118 ceph-mon[33862]: osdmap e373: 8 total, 8 up, 7 in 2024-03-21T14:19:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3318351990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:19:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:49 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 14794/153504 objects degraded (9.638%), 7 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:49 smithi118 ceph-mon[33862]: osdmap e374: 8 total, 8 up, 7 in 2024-03-21T14:19:50.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:50.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:50.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:50.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:50.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:50.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:50.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:50.636 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:50.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:50.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:50.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:50.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:50.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:50.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:51.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[29576]: 10.0 scrub ok 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[29576]: pgmap v798: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 14794/149103 objects degraded (9.922%); 6259/149103 objects misplaced (4.198%); 32 MiB/s, 511 objects/s recovering 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[40510]: 10.0 scrub ok 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[40510]: pgmap v798: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 14794/149103 objects degraded (9.922%); 6259/149103 objects misplaced (4.198%); 32 MiB/s, 511 objects/s recovering 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:19:51.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:50 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:19:51.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.085 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:51.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:50 smithi118 ceph-mon[33862]: 10.0 scrub ok 2024-03-21T14:19:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:50 smithi118 ceph-mon[33862]: pgmap v798: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 14794/149103 objects degraded (9.922%); 6259/149103 objects misplaced (4.198%); 32 MiB/s, 511 objects/s recovering 2024-03-21T14:19:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:50 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:19:51.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:50 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:19:51.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:51.210 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:51.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:51.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.347 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:51.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:51.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:51.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.482 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:51.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:51.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:51.617 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:51.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:51.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:51.831 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:51 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:19:51.832 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:51 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:19:51.832 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:51 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:19:51.832 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:51 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:19:51.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:51.906 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:51.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:52.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:52.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:52.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:52.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:19:52.156 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:51 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:19:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:51 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:19:52.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:52.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:52.397 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:52.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:52.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:52.529 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:52.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:52.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:52.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.666 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:52.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:52.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:52.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:52.833 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:19:52.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:52 smithi003 ceph-mon[29576]: pgmap v800: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 721 op/s; 14794/149103 objects degraded (9.922%); 6259/149103 objects misplaced (4.198%); 12 MiB/s, 197 objects/s recovering 2024-03-21T14:19:52.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:52 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/681303810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:19:52.888 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:52 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:19:52.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:52 smithi003 ceph-mon[40510]: pgmap v800: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 721 op/s; 14794/149103 objects degraded (9.922%); 6259/149103 objects misplaced (4.198%); 12 MiB/s, 197 objects/s recovering 2024-03-21T14:19:52.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:52 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/681303810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:19:52.889 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:52 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:19:52.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:52.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:52.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:53.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:53.073 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:53.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:53.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:52 smithi118 ceph-mon[33862]: pgmap v800: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 150 active+clean, 2 active+undersized+degraded+remapped+backfill_wait, 3 active+undersized+degraded+remapped+backfilling; 3.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 721 op/s; 14794/149103 objects degraded (9.922%); 6259/149103 objects misplaced (4.198%); 12 MiB/s, 197 objects/s recovering 2024-03-21T14:19:53.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:52 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/681303810' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:19:53.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:52 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:19:53.175 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:53.214 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:53.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:53.351 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:53.453 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:53.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:53.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:53.625 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:53.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:53.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:53.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:53.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:53.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:53.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:53.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:53.998 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.030 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:54.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:54.113 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:53 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:19:54.113 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:53 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:19:54.113 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:53 smithi118 ceph-mon[33862]: osdmap e375: 8 total, 8 up, 7 in 2024-03-21T14:19:54.131 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:54.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:54.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:53 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:19:54.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:53 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:19:54.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:53 smithi003 ceph-mon[29576]: osdmap e375: 8 total, 8 up, 7 in 2024-03-21T14:19:54.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:53 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:19:54.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:53 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:19:54.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:53 smithi003 ceph-mon[40510]: osdmap e375: 8 total, 8 up, 7 in 2024-03-21T14:19:54.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:54.290 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:54.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:54.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:54.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:54.554 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:54.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:54.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:54.789 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:54.814 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:54.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:54.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:54.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:54.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:54.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:55.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:55.091 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:55.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:55.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:54 smithi003 ceph-mon[29576]: pgmap v802: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 11006/132486 objects degraded (8.307%); 5545/132486 objects misplaced (4.185%); 23 MiB/s, 373 objects/s recovering 2024-03-21T14:19:55.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:54 smithi003 ceph-mon[29576]: osdmap e376: 8 total, 8 up, 7 in 2024-03-21T14:19:55.147 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:54 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 11006/132486 objects degraded (8.307%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:55.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:54 smithi003 ceph-mon[40510]: pgmap v802: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 11006/132486 objects degraded (8.307%); 5545/132486 objects misplaced (4.185%); 23 MiB/s, 373 objects/s recovering 2024-03-21T14:19:55.147 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:54 smithi003 ceph-mon[40510]: osdmap e376: 8 total, 8 up, 7 in 2024-03-21T14:19:55.148 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:54 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 11006/132486 objects degraded (8.307%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:54 smithi118 ceph-mon[33862]: pgmap v802: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.17k op/s; 11006/132486 objects degraded (8.307%); 5545/132486 objects misplaced (4.185%); 23 MiB/s, 373 objects/s recovering 2024-03-21T14:19:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:54 smithi118 ceph-mon[33862]: osdmap e376: 8 total, 8 up, 7 in 2024-03-21T14:19:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:54 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 11006/132486 objects degraded (8.307%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T14:19:55.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:55.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:55.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:55.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:55.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:55.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:55.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:55.721 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:19:55.721 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:19:55.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:55.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:55.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:55 smithi118 ceph-mon[33862]: osdmap e377: 8 total, 8 up, 7 in 2024-03-21T14:19:55.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:55.981 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:55.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:55.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:55 smithi003 ceph-mon[29576]: osdmap e377: 8 total, 8 up, 7 in 2024-03-21T14:19:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:55 smithi003 ceph-mon[40510]: osdmap e377: 8 total, 8 up, 7 in 2024-03-21T14:19:56.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:56.115 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:56.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:56.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:56.255 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:56.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:56.396 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:56.498 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:56.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:56.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T14:19:56.638 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:56.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.671 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:56.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:19:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:19:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:19:56.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:56.819 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:56.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:56.922 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:19:56.950 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:56.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:19:57.052 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:57.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:19:57.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:57.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:57.140 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:56 smithi003 ceph-mon[29576]: pgmap v805: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 896 B/s rd, 0 B/s wr, 970 op/s; 11006/132486 objects degraded (8.307%); 5545/132486 objects misplaced (4.185%); 25 MiB/s, 392 objects/s recovering 2024-03-21T14:19:57.141 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:56 smithi003 ceph-mon[40510]: pgmap v805: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 896 B/s rd, 0 B/s wr, 970 op/s; 11006/132486 objects degraded (8.307%); 5545/132486 objects misplaced (4.185%); 25 MiB/s, 392 objects/s recovering 2024-03-21T14:19:57.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:56 smithi118 ceph-mon[33862]: pgmap v805: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.7 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 896 B/s rd, 0 B/s wr, 970 op/s; 11006/132486 objects degraded (8.307%); 5545/132486 objects misplaced (4.185%); 25 MiB/s, 392 objects/s recovering 2024-03-21T14:19:57.198 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:57.228 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:57.246 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:57.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:57.398 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:57.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:57.550 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:57.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:57.600 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:57.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:57.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:19:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:19:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:57.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.849 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:57.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:57.876 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-03-21T14:19:57.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:57.988 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:57.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:57.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:58.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1658410440' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:19:58.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1658410440' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:19:58.090 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:58.124 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:58.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1658410440' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:19:58.227 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:58.262 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:58.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:58.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:58.399 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:58.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:19:58.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:19:58.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:58.679 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:58.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:58.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:58.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:58.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T14:19:58.941 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:19:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:58.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:58.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:19:59.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:59.076 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[29576]: pgmap v806: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 4 active+undersized+degraded+remapped+backfilling; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.45k op/s; 7683/123057 objects degraded (6.243%); 4239/123057 objects misplaced (3.445%); 46 MiB/s, 738 objects/s recovering 2024-03-21T14:19:59.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1658410440' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:19:59.077 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[29576]: osdmap e378: 8 total, 8 up, 7 in 2024-03-21T14:19:59.078 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:59.078 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[40510]: pgmap v806: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 4 active+undersized+degraded+remapped+backfilling; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.45k op/s; 7683/123057 objects degraded (6.243%); 4239/123057 objects misplaced (3.445%); 46 MiB/s, 738 objects/s recovering 2024-03-21T14:19:59.078 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1658410440' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:19:59.078 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:58 smithi003 ceph-mon[40510]: osdmap e378: 8 total, 8 up, 7 in 2024-03-21T14:19:59.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:59.117 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:58 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:19:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:58 smithi118 ceph-mon[33862]: pgmap v806: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 151 active+clean, 4 active+undersized+degraded+remapped+backfilling; 2.5 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.45k op/s; 7683/123057 objects degraded (6.243%); 4239/123057 objects misplaced (3.445%); 46 MiB/s, 738 objects/s recovering 2024-03-21T14:19:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1658410440' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:19:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:58 smithi118 ceph-mon[33862]: osdmap e378: 8 total, 8 up, 7 in 2024-03-21T14:19:59.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:59.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:59.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:19:59.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:19:59.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.409 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:59.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:19:59.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:19:59.565 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:19:59.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:59.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:19:59.668 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:19:59.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:19:59.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:19:59.841 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:19:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:19:59.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:19:59.994 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:19:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:19:59.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:00.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:00.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:19:59 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 7683/123057 objects degraded (6.243%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T14:20:00.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:19:59 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 7683/123057 objects degraded (6.243%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T14:20:00.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:19:59 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 7683/123057 objects degraded (6.243%), 6 pgs degraded (PG_DEGRADED) 2024-03-21T14:20:00.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:00.272 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:00.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:00.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:00.412 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:00.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:00.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:00.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:00.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:00.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:00.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:00.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:00.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:00.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:00.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:00.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:01.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:01.125 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:01.167 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pgmap v808: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 152 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 847 op/s; 7540/117237 objects degraded (6.431%); 4239/117237 objects misplaced (3.616%); 24 MiB/s, 388 objects/s recovering 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; Degraded data redundancy: 7540/117237 objects degraded (6.431%), 5 pgs degraded 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: daemon osd.6 on smithi118 is in unknown state 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.3 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.4 is active+remapped+backfill_toofull, acting [4,3,1] 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: [WRN] PG_DEGRADED: Degraded data redundancy: 7540/117237 objects degraded (6.431%), 5 pgs degraded 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.3 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.5 is active+undersized+degraded+remapped+backfilling, acting [0,3] 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.a is active+undersized+degraded+remapped+backfilling, acting [5,7] 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:00 smithi118 ceph-mon[33862]: pg 10.f is active+undersized+degraded+remapped+backfilling, acting [3,4] 2024-03-21T14:20:01.227 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pgmap v808: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 152 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 847 op/s; 7540/117237 objects degraded (6.431%); 4239/117237 objects misplaced (3.616%); 24 MiB/s, 388 objects/s recovering 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; Degraded data redundancy: 7540/117237 objects degraded (6.431%), 5 pgs degraded 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: daemon osd.6 on smithi118 is in unknown state 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.3 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.4 is active+remapped+backfill_toofull, acting [4,3,1] 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: [WRN] PG_DEGRADED: Degraded data redundancy: 7540/117237 objects degraded (6.431%), 5 pgs degraded 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.3 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.5 is active+undersized+degraded+remapped+backfilling, acting [0,3] 2024-03-21T14:20:01.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.a is active+undersized+degraded+remapped+backfilling, acting [5,7] 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[29576]: pg 10.f is active+undersized+degraded+remapped+backfilling, acting [3,4] 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pgmap v808: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 152 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 847 op/s; 7540/117237 objects degraded (6.431%); 4239/117237 objects misplaced (3.616%); 24 MiB/s, 388 objects/s recovering 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); noscrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull; Degraded data redundancy: 7540/117237 objects degraded (6.431%), 5 pgs degraded 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: daemon osd.6 on smithi118 is in unknown state 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.3 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.4 is active+remapped+backfill_toofull, acting [4,3,1] 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: [WRN] PG_DEGRADED: Degraded data redundancy: 7540/117237 objects degraded (6.431%), 5 pgs degraded 2024-03-21T14:20:01.236 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.3 is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.5 is active+undersized+degraded+remapped+backfilling, acting [0,3] 2024-03-21T14:20:01.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.a is active+undersized+degraded+remapped+backfilling, acting [5,7] 2024-03-21T14:20:01.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.e is active+undersized+degraded+remapped+backfill_wait+backfill_toofull, acting [4,5] 2024-03-21T14:20:01.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:00 smithi003 ceph-mon[40510]: pg 10.f is active+undersized+degraded+remapped+backfilling, acting [3,4] 2024-03-21T14:20:01.257 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:01.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:01.386 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:01.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:01.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:01.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:01.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:01.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:01.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:01.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:01.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:01.914 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:01.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:01.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:02.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:02.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:02.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:02.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:01 smithi118 ceph-mon[33862]: osdmap e379: 8 total, 8 up, 7 in 2024-03-21T14:20:02.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:01 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:20:02.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:02.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:01 smithi003 ceph-mon[29576]: osdmap e379: 8 total, 8 up, 7 in 2024-03-21T14:20:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:01 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:20:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:01 smithi003 ceph-mon[40510]: osdmap e379: 8 total, 8 up, 7 in 2024-03-21T14:20:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:01 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:20:02.300 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:02.432 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:02.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:02.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:02.594 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:02.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:02.724 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:02.745 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:20:02.773 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:20:02.799 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:20:02.826 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:02.827 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:20:02.853 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:20:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.854 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:02.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:02.882 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:20:02.915 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: pgmap v810: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 152 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 774 op/s; 7540/117237 objects degraded (6.431%); 4239/117237 objects misplaced (3.616%); 22 MiB/s, 354 objects/s recovering 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: osdmap e380: 8 total, 8 up, 7 in 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:02.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:02.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3356919409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:02.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:02.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:02 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:02.950 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:20:02.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:20:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:20:02.958 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:20:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:20:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:20:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:20:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:20:02.959 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:20:02.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:02.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:03.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:03.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:03.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:03.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:03.237 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:03.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: pgmap v810: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 152 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 774 op/s; 7540/117237 objects degraded (6.431%); 4239/117237 objects misplaced (3.616%); 22 MiB/s, 354 objects/s recovering 2024-03-21T14:20:03.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: osdmap e380: 8 total, 8 up, 7 in 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3356919409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:03.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: pgmap v810: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 152 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.4 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 779 B/s rd, 0 B/s wr, 774 op/s; 7540/117237 objects degraded (6.431%); 4239/117237 objects misplaced (3.616%); 22 MiB/s, 354 objects/s recovering 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: osdmap e380: 8 total, 8 up, 7 in 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3356919409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:03.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:03.257 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:02 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:03.279 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:03.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:03.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:03.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:03.425 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:03.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:03.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:03.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:03.568 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:03.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:03.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:03.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:03.706 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:03.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:03.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:03.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:03.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:03.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:03.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:04.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:04.116 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:04.150 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:03 smithi118 ceph-mon[33862]: osdmap e381: 8 total, 8 up, 7 in 2024-03-21T14:20:04.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:04.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:03 smithi003 ceph-mon[29576]: osdmap e381: 8 total, 8 up, 7 in 2024-03-21T14:20:04.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:03 smithi003 ceph-mon[40510]: osdmap e381: 8 total, 8 up, 7 in 2024-03-21T14:20:04.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:04.382 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:04.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:04.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:04.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:04.557 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:04.659 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:04.693 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:04.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:04.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:04.929 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:04.962 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:04.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: pgmap v813: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 927 B/s rd, 0 B/s wr, 910 op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 27 MiB/s, 426 objects/s recovering 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 4210/102168 objects degraded (4.121%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T14:20:05.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[29576]: osdmap e382: 8 total, 8 up, 7 in 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: pgmap v813: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 927 B/s rd, 0 B/s wr, 910 op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 27 MiB/s, 426 objects/s recovering 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 4210/102168 objects degraded (4.121%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T14:20:05.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:04 smithi003 ceph-mon[40510]: osdmap e382: 8 total, 8 up, 7 in 2024-03-21T14:20:05.064 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:05.095 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:05.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: pgmap v813: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 927 B/s rd, 0 B/s wr, 910 op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 27 MiB/s, 426 objects/s recovering 2024-03-21T14:20:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:20:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:20:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:20:05.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:05.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 4210/102168 objects degraded (4.121%), 3 pgs degraded (PG_DEGRADED) 2024-03-21T14:20:05.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:04 smithi118 ceph-mon[33862]: osdmap e382: 8 total, 8 up, 7 in 2024-03-21T14:20:05.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:05.231 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:05.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:05.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:05.475 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:05.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:05.613 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:05.649 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:05.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:05.785 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:05.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:05.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:05.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:06.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:06.067 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:06.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:06.148 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:05 smithi003 ceph-mon[29576]: pgmap v814: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 31 MiB/s, 497 objects/s recovering 2024-03-21T14:20:06.148 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:05 smithi003 ceph-mon[40510]: pgmap v814: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 31 MiB/s, 497 objects/s recovering 2024-03-21T14:20:06.169 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:05 smithi118 ceph-mon[33862]: pgmap v814: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 31 MiB/s, 497 objects/s recovering 2024-03-21T14:20:06.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:06.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:06.326 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:06.428 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:06.457 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:06.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:06.586 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:06.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:06.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:06.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:06.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:20:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:20:06.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:06.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:06.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:06.962 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T14:20:06.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:06.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:06.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:07.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:07.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:07.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:07.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-mon[29576]: osdmap e383: 8 total, 8 up, 7 in 2024-03-21T14:20:07.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4187143757' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:20:07.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:20:07.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-mon[40510]: osdmap e383: 8 total, 8 up, 7 in 2024-03-21T14:20:07.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4187143757' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:20:07.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:06 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:20:07.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:07.309 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:06 smithi118 ceph-mon[33862]: osdmap e383: 8 total, 8 up, 7 in 2024-03-21T14:20:07.310 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:06 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4187143757' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:20:07.310 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:06 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:20:07.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:07.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:07.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:07.517 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:07.619 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:07.644 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:07.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:20:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:20:07.745 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:07.769 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:07.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:20:07.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:07.896 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:07.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:07.998 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:08.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:08.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:08.157 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:08.253 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:20:08.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[29576]: pgmap v817: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 25 MiB/s, 403 objects/s recovering 2024-03-21T14:20:08.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:20:08.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[29576]: osdmap e384: 8 total, 8 up, 7 in 2024-03-21T14:20:08.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1167716758' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:08.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[40510]: pgmap v817: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 25 MiB/s, 403 objects/s recovering 2024-03-21T14:20:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:20:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[40510]: osdmap e384: 8 total, 8 up, 7 in 2024-03-21T14:20:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1167716758' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:08.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:07 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:08.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:08.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:08.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:08.413 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:07 smithi118 ceph-mon[33862]: pgmap v817: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 153 active+clean, 3 active+undersized+degraded+remapped+backfilling; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s; 4210/102168 objects degraded (4.121%); 2871/102168 objects misplaced (2.810%); 25 MiB/s, 403 objects/s recovering 2024-03-21T14:20:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:07 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:20:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:07 smithi118 ceph-mon[33862]: osdmap e384: 8 total, 8 up, 7 in 2024-03-21T14:20:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1167716758' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:07 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:08.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:08.540 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:08.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:08.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:08.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:08.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:08.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:08.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:08.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:08.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:08.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:09.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:09.055 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:09.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:09.191 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:20:09.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[29576]: osdmap e385: 8 total, 8 up, 7 in 2024-03-21T14:20:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:20:09.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:08 smithi003 ceph-mon[40510]: osdmap e385: 8 total, 8 up, 7 in 2024-03-21T14:20:09.292 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:09.329 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:09.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:09.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:09.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:08 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:09.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:08 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:20:09.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:08 smithi118 ceph-mon[33862]: osdmap e385: 8 total, 8 up, 7 in 2024-03-21T14:20:09.465 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:09.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:09.602 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:09.692 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:20:09.692 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:20:09.692 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:20:09.692 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:20:09.692 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:20:09.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:09.751 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:09.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:09.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:09.881 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:09.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:09.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:10.006 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:10.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:10.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:10.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:10.234 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[29576]: pgmap v820: 158 pgs: 1 active+remapped+backfilling, 1 peering, 155 active+clean, 1 active+undersized+degraded+remapped+backfilling; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 421/89646 objects degraded (0.470%); 1368/89646 objects misplaced (1.526%); 63 MiB/s, 1.01k objects/s recovering 2024-03-21T14:20:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[29576]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T14:20:10.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[29576]: osdmap e386: 8 total, 8 up, 7 in 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[29576]: Health check update: Degraded data redundancy: 421/89646 objects degraded (0.470%), 1 pg degraded (PG_DEGRADED) 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[29576]: osdmap e387: 8 total, 8 up, 7 in 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[40510]: pgmap v820: 158 pgs: 1 active+remapped+backfilling, 1 peering, 155 active+clean, 1 active+undersized+degraded+remapped+backfilling; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 421/89646 objects degraded (0.470%); 1368/89646 objects misplaced (1.526%); 63 MiB/s, 1.01k objects/s recovering 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[40510]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[40510]: osdmap e386: 8 total, 8 up, 7 in 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[40510]: Health check update: Degraded data redundancy: 421/89646 objects degraded (0.470%), 1 pg degraded (PG_DEGRADED) 2024-03-21T14:20:10.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:09 smithi003 ceph-mon[40510]: osdmap e387: 8 total, 8 up, 7 in 2024-03-21T14:20:10.258 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:10.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:10.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:10.384 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:09 smithi118 ceph-mon[33862]: pgmap v820: 158 pgs: 1 active+remapped+backfilling, 1 peering, 155 active+clean, 1 active+undersized+degraded+remapped+backfilling; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 421/89646 objects degraded (0.470%); 1368/89646 objects misplaced (1.526%); 63 MiB/s, 1.01k objects/s recovering 2024-03-21T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:09 smithi118 ceph-mon[33862]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:09 smithi118 ceph-mon[33862]: osdmap e386: 8 total, 8 up, 7 in 2024-03-21T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:09 smithi118 ceph-mon[33862]: Health check update: Degraded data redundancy: 421/89646 objects degraded (0.470%), 1 pg degraded (PG_DEGRADED) 2024-03-21T14:20:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:09 smithi118 ceph-mon[33862]: osdmap e387: 8 total, 8 up, 7 in 2024-03-21T14:20:10.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:10.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:10.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:10.638 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:10.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:10.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:10.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:10.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:10.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:10.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:11.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:11.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:11.159 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:11.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:11.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:11.332 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:11.434 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:11.464 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:11.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:11.598 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:11.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:11.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:11.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:11.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:11.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:11.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:11.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:11.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:11 smithi003 ceph-mon[29576]: pgmap v823: 158 pgs: 1 active+remapped+backfilling, 1 peering, 155 active+clean, 1 active+undersized+degraded+remapped+backfilling; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 421/89646 objects degraded (0.470%); 1368/89646 objects misplaced (1.526%); 63 MiB/s, 1.01k objects/s recovering 2024-03-21T14:20:12.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:11 smithi003 ceph-mon[29576]: osdmap e388: 8 total, 8 up, 7 in 2024-03-21T14:20:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:11 smithi003 ceph-mon[40510]: pgmap v823: 158 pgs: 1 active+remapped+backfilling, 1 peering, 155 active+clean, 1 active+undersized+degraded+remapped+backfilling; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 421/89646 objects degraded (0.470%); 1368/89646 objects misplaced (1.526%); 63 MiB/s, 1.01k objects/s recovering 2024-03-21T14:20:12.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:11 smithi003 ceph-mon[40510]: osdmap e388: 8 total, 8 up, 7 in 2024-03-21T14:20:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.016 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:12.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:12.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:12.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:11 smithi118 ceph-mon[33862]: pgmap v823: 158 pgs: 1 active+remapped+backfilling, 1 peering, 155 active+clean, 1 active+undersized+degraded+remapped+backfilling; 1.8 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 421/89646 objects degraded (0.470%); 1368/89646 objects misplaced (1.526%); 63 MiB/s, 1.01k objects/s recovering 2024-03-21T14:20:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:11 smithi118 ceph-mon[33862]: osdmap e388: 8 total, 8 up, 7 in 2024-03-21T14:20:12.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T14:20:12.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:12.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:12.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.470 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:12.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:12.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:12.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.605 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:12.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:12.706 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:12.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:12.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:12.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:12.874 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:12.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:12.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:12 smithi003 ceph-mon[29576]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 421/89646 objects degraded (0.470%), 1 pg degraded) 2024-03-21T14:20:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:12 smithi003 ceph-mon[40510]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 421/89646 objects degraded (0.470%), 1 pg degraded) 2024-03-21T14:20:13.009 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:13.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:13.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:13.138 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:12 smithi118 ceph-mon[33862]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 421/89646 objects degraded (0.470%), 1 pg degraded) 2024-03-21T14:20:13.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:13.276 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:13.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:13.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:13.521 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:13.552 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:13.654 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:13.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.678 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:13.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:13.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.822 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:13.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:13.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:13.954 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:13.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:13 smithi003 ceph-mon[29576]: pgmap v825: 158 pgs: 1 peering, 157 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 27 MiB/s, 424 objects/s recovering 2024-03-21T14:20:14.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:13 smithi003 ceph-mon[40510]: pgmap v825: 158 pgs: 1 peering, 157 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 27 MiB/s, 424 objects/s recovering 2024-03-21T14:20:14.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:14.090 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:13 smithi118 ceph-mon[33862]: pgmap v825: 158 pgs: 1 peering, 157 active+clean; 1.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 0 B/s wr, 296 op/s; 27 MiB/s, 424 objects/s recovering 2024-03-21T14:20:14.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:14.226 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:14.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:14.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:14.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.501 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:14.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:14.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:14.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.639 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:14.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:14.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:14.764 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:14.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:14.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:14.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:14.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:15.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:15.035 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:15.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:15.172 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:15.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:15.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:15.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:15.443 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:15.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:15.545 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:15.581 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:15.683 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.719 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:15.820 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.856 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:15.945 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:15 smithi003 ceph-mon[29576]: pgmap v826: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 631 op/s; 18 MiB/s, 294 objects/s recovering 2024-03-21T14:20:15.945 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:15 smithi003 ceph-mon[40510]: pgmap v826: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 631 op/s; 18 MiB/s, 294 objects/s recovering 2024-03-21T14:20:15.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:15.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:15.989 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:15.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:16.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:16.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:16.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T14:20:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:15 smithi118 ceph-mon[33862]: pgmap v826: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 631 op/s; 18 MiB/s, 294 objects/s recovering 2024-03-21T14:20:16.227 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:16.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:16.296 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:16.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:16.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:16.418 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:16.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.450 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:16.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:16.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:16.559 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:20:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:20:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:20:16.580 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:16.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:16.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:16.713 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:16.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:16.857 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:16.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:16.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:16.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:16.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:17.094 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:17.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.130 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:17.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:17.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:17.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:17.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:17.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.419 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:17.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:17.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:17.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:17.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:17.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:17.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:20:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:20:17.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:17.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:17.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:17.942 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:17.978 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:17.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:17.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:17.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:20:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:17 smithi003 ceph-mon[29576]: pgmap v827: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 714 B/s rd, 0 B/s wr, 529 op/s; 15 MiB/s, 246 objects/s recovering 2024-03-21T14:20:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:18.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:17 smithi003 ceph-mon[40510]: pgmap v827: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 714 B/s rd, 0 B/s wr, 529 op/s; 15 MiB/s, 246 objects/s recovering 2024-03-21T14:20:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:18.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:18.080 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:18.143 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:17 smithi118 ceph-mon[33862]: pgmap v827: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 614 GiB / 626 GiB avail; 714 B/s rd, 0 B/s wr, 529 op/s; 15 MiB/s, 246 objects/s recovering 2024-03-21T14:20:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:18.245 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:18.276 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:18.378 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:18.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:18.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:18.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:18.540 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:18.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:18.708 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:18.810 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:18.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:18.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:18.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:18.976 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:18.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-03-21T14:20:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:18 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-03-21T14:20:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-03-21T14:20:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:18 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-03-21T14:20:19.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:19.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-03-21T14:20:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:18 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]: dispatch 2024-03-21T14:20:19.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:19.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:19.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:19.426 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:19.528 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:19.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:19.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:19.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:19.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:19.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:19.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:19.913 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:19.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:19.989 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:19.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:20.056 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[29576]: pgmap v828: 158 pgs: 158 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 13 MiB/s, 209 objects/s recovering 2024-03-21T14:20:20.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]': finished 2024-03-21T14:20:20.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[29576]: osdmap e389: 8 total, 8 up, 7 in 2024-03-21T14:20:20.057 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[29576]: osdmap e390: 8 total, 8 up, 7 in 2024-03-21T14:20:20.057 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[40510]: pgmap v828: 158 pgs: 158 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 13 MiB/s, 209 objects/s recovering 2024-03-21T14:20:20.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]': finished 2024-03-21T14:20:20.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[40510]: osdmap e389: 8 total, 8 up, 7 in 2024-03-21T14:20:20.058 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:19 smithi003 ceph-mon[40510]: osdmap e390: 8 total, 8 up, 7 in 2024-03-21T14:20:20.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:20.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:19 smithi118 ceph-mon[33862]: pgmap v828: 158 pgs: 158 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s; 13 MiB/s, 209 objects/s recovering 2024-03-21T14:20:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [4, 2]}]': finished 2024-03-21T14:20:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:19 smithi118 ceph-mon[33862]: osdmap e389: 8 total, 8 up, 7 in 2024-03-21T14:20:20.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:19 smithi118 ceph-mon[33862]: osdmap e390: 8 total, 8 up, 7 in 2024-03-21T14:20:20.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.270 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:20.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:20.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:20.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:20.507 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:20.536 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:20.638 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:20.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:20.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:20.787 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:20.788 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T14:20:20.816 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:20.820 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:20:20.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/200305374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:20:20.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:20 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:20:20.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1559157710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:20.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/200305374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:20:20.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:20 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:20:20.826 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1559157710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:20.852 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:20:20.873 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:20:20.893 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:20:20.913 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:20:20.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:20.934 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:20:20.941 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:20.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:20.954 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:20:20.973 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:20:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:20:20.974 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:20:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:20:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:20:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:20:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:20:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:20:20.975 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:20:21.043 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:21.170 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/200305374' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:20:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:20 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:20:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1559157710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:21.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.195 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:21.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:21.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:21.320 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:21.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:21.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:21.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:21.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:21.573 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:21.675 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:21.700 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:21.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:21.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:21.827 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:21.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:21.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:21.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:21.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[29576]: pgmap v831: 158 pgs: 158 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[29576]: osdmap e391: 8 total, 8 up, 7 in 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[29576]: osdmap e392: 8 total, 8 up, 7 in 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[40510]: pgmap v831: 158 pgs: 158 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:20:22.033 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[40510]: osdmap e391: 8 total, 8 up, 7 in 2024-03-21T14:20:22.034 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:21 smithi003 ceph-mon[40510]: osdmap e392: 8 total, 8 up, 7 in 2024-03-21T14:20:22.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:22.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:22.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:22.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:21 smithi118 ceph-mon[33862]: pgmap v831: 158 pgs: 158 active+clean; 1.2 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:20:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:21 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:20:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:21 smithi118 ceph-mon[33862]: osdmap e391: 8 total, 8 up, 7 in 2024-03-21T14:20:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:21 smithi118 ceph-mon[33862]: osdmap e392: 8 total, 8 up, 7 in 2024-03-21T14:20:22.206 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:22.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:22.307 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:22.333 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:22.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:22.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:22.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:22.587 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:22.689 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:22.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:22.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:22.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:22.840 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:22.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T14:20:22.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:22.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:22.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:23.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:23.216 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:23.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:23.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:23.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:23.403 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:23.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:23.505 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:23.537 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:23.638 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:23.674 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:23.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:23.776 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:23.818 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:23.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:23.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:24.057 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:24.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:23 smithi118 ceph-mon[33862]: pgmap v834: 158 pgs: 158 active+clean; 1.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 490 op/s; 82 B/s, 3 objects/s recovering 2024-03-21T14:20:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:23 smithi118 ceph-mon[33862]: osdmap e393: 8 total, 8 up, 7 in 2024-03-21T14:20:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:23 smithi118 ceph-mon[33862]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 390'18538 2024-03-21T14:20:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:23 smithi118 ceph-mon[33862]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 390'18071 2024-03-21T14:20:24.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[29576]: pgmap v834: 158 pgs: 158 active+clean; 1.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 490 op/s; 82 B/s, 3 objects/s recovering 2024-03-21T14:20:24.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[29576]: osdmap e393: 8 total, 8 up, 7 in 2024-03-21T14:20:24.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[29576]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 390'18538 2024-03-21T14:20:24.186 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[29576]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 390'18071 2024-03-21T14:20:24.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[40510]: pgmap v834: 158 pgs: 158 active+clean; 1.0 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 490 op/s; 82 B/s, 3 objects/s recovering 2024-03-21T14:20:24.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[40510]: osdmap e393: 8 total, 8 up, 7 in 2024-03-21T14:20:24.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[40510]: 10.7 starting backfill to osd.3 from (0'0,0'0] MAX to 390'18538 2024-03-21T14:20:24.187 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:23 smithi003 ceph-mon[40510]: 10.3 starting backfill to osd.3 from (0'0,0'0] MAX to 390'18071 2024-03-21T14:20:24.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:24.223 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:24.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:24.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:24.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:24.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:24.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:24.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:24.644 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:24.746 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:24.775 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:24.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:24.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:24.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:24.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:24.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:24 smithi003 ceph-mon[29576]: osdmap e394: 8 total, 8 up, 7 in 2024-03-21T14:20:24.967 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:24 smithi003 ceph-mon[40510]: osdmap e394: 8 total, 8 up, 7 in 2024-03-21T14:20:25.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:25.066 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:25.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:24 smithi118 ceph-mon[33862]: osdmap e394: 8 total, 8 up, 7 in 2024-03-21T14:20:25.197 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:25.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:25.328 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:25.430 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:25.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:25.566 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:25.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:25.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:25.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:25.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:25.871 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:25.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:25.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:25.992 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:20:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: pgmap v837: 158 pgs: 1 activating+degraded+remapped, 2 activating+remapped, 1 remapped+peering, 154 active+clean; 911 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 8.2 KiB/s rd, 55 KiB/s wr, 1.14k op/s; 1/44304 objects degraded (0.002%); 1772/44304 objects misplaced (4.000%); 52 KiB/s, 4 objects/s recovering 2024-03-21T14:20:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:20:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: Health check failed: Degraded data redundancy: 1/44304 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-03-21T14:20:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: osdmap e395: 8 total, 8 up, 7 in 2024-03-21T14:20:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[29576]: osdmap e396: 8 total, 8 up, 7 in 2024-03-21T14:20:26.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: pgmap v837: 158 pgs: 1 activating+degraded+remapped, 2 activating+remapped, 1 remapped+peering, 154 active+clean; 911 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 8.2 KiB/s rd, 55 KiB/s wr, 1.14k op/s; 1/44304 objects degraded (0.002%); 1772/44304 objects misplaced (4.000%); 52 KiB/s, 4 objects/s recovering 2024-03-21T14:20:26.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:26.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:26.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:20:26.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: Health check failed: Degraded data redundancy: 1/44304 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-03-21T14:20:26.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: osdmap e395: 8 total, 8 up, 7 in 2024-03-21T14:20:26.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:25 smithi003 ceph-mon[40510]: osdmap e396: 8 total, 8 up, 7 in 2024-03-21T14:20:26.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:26.044 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:26.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:26.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:26.176 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: pgmap v837: 158 pgs: 1 activating+degraded+remapped, 2 activating+remapped, 1 remapped+peering, 154 active+clean; 911 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 8.2 KiB/s rd, 55 KiB/s wr, 1.14k op/s; 1/44304 objects degraded (0.002%); 1772/44304 objects misplaced (4.000%); 52 KiB/s, 4 objects/s recovering 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: Health check failed: Degraded data redundancy: 1/44304 objects degraded (0.002%), 1 pg degraded (PG_DEGRADED) 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: osdmap e395: 8 total, 8 up, 7 in 2024-03-21T14:20:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:25 smithi118 ceph-mon[33862]: osdmap e396: 8 total, 8 up, 7 in 2024-03-21T14:20:26.278 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:26.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:26.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:26.423 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:26.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:26.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:26.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:26.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:26.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:26.743 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:26.745 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:20:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:20:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:20:26.845 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:26.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:26.877 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:26.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:26.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:26 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:26 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:26 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:26 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:27.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:27.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:27.116 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:27.154 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:26 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:26 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:27.257 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:27.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:27.418 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:27.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:27.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:27.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:27.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:20:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:20:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:27.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:27.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:27.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:27.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:27.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:27.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:27 smithi003 ceph-mon[29576]: pgmap v840: 158 pgs: 1 activating+degraded+remapped, 2 activating+remapped, 1 remapped+peering, 154 active+clean; 911 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 8.2 KiB/s rd, 55 KiB/s wr, 654 op/s; 1/44304 objects degraded (0.002%); 1772/44304 objects misplaced (4.000%); 52 KiB/s, 1 objects/s recovering 2024-03-21T14:20:28.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:27 smithi003 ceph-mon[40510]: pgmap v840: 158 pgs: 1 activating+degraded+remapped, 2 activating+remapped, 1 remapped+peering, 154 active+clean; 911 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 8.2 KiB/s rd, 55 KiB/s wr, 654 op/s; 1/44304 objects degraded (0.002%); 1772/44304 objects misplaced (4.000%); 52 KiB/s, 1 objects/s recovering 2024-03-21T14:20:28.071 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:28.100 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:27 smithi118 ceph-mon[33862]: pgmap v840: 158 pgs: 1 activating+degraded+remapped, 2 activating+remapped, 1 remapped+peering, 154 active+clean; 911 MiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 8.2 KiB/s rd, 55 KiB/s wr, 654 op/s; 1/44304 objects degraded (0.002%); 1772/44304 objects misplaced (4.000%); 52 KiB/s, 1 objects/s recovering 2024-03-21T14:20:28.201 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:28.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:28.251 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:28.251 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:28.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:28.375 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:28.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:28.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:28.561 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:28.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:28.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:28.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:28.830 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:28.855 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:20:28.931 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:28.971 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:28.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:28.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:29.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/899703088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:20:29.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:29.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/899703088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:20:29.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:29.102 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:29.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/899703088' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:20:29.204 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:29.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.239 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:29.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:29.340 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:29.379 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:20:29.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:29.497 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:29.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:29.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:29.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:29.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:29.747 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:29.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:29.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:29.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:29.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:29.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:29.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: pgmap v841: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 569 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 957 B/s rd, 0 B/s wr, 1.29k op/s; 763/27906 objects misplaced (2.734%); 4.1 MiB/s, 65 objects/s recovering 2024-03-21T14:20:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:20:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:20:30.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/44304 objects degraded (0.002%), 1 pg degraded) 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/899703088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: osdmap e397: 8 total, 8 up, 7 in 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[29576]: osdmap e398: 8 total, 8 up, 7 in 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: pgmap v841: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 569 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 957 B/s rd, 0 B/s wr, 1.29k op/s; 763/27906 objects misplaced (2.734%); 4.1 MiB/s, 65 objects/s recovering 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:20:30.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/44304 objects degraded (0.002%), 1 pg degraded) 2024-03-21T14:20:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/899703088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:20:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: osdmap e397: 8 total, 8 up, 7 in 2024-03-21T14:20:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:20:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:20:30.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:29 smithi003 ceph-mon[40510]: osdmap e398: 8 total, 8 up, 7 in 2024-03-21T14:20:30.066 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:30.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: pgmap v841: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 569 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 957 B/s rd, 0 B/s wr, 1.29k op/s; 763/27906 objects misplaced (2.734%); 4.1 MiB/s, 65 objects/s recovering 2024-03-21T14:20:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/44304 objects degraded (0.002%), 1 pg degraded) 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/899703088' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: osdmap e397: 8 total, 8 up, 7 in 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:20:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:29 smithi118 ceph-mon[33862]: osdmap e398: 8 total, 8 up, 7 in 2024-03-21T14:20:30.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:30.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:30.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:30.344 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.380 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:30.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:30.481 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:30.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:30.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:30.666 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:30.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:30.799 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:30.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:30.948 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:30.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:31.050 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:31.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:30 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:20:31.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:30 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:20:31.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:30 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:20:31.128 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:30 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:20:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:30 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:20:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:30 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:20:31.185 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:31.208 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:31.310 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:31.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:31.436 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.460 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:31.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:31.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:31.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.586 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:31.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:31.687 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:31.710 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:31.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:31.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:31.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:31.863 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:31.881 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:31.881 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:31.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:31.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:31.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:31.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:32.088 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:32.125 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:32.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: pgmap v844: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 569 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.03k op/s; 763/27906 objects misplaced (2.734%); 4.1 MiB/s, 65 objects/s recovering 2024-03-21T14:20:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:20:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: osdmap e399: 8 total, 8 up, 7 in 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 1.6 scrub starts 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 1.6 scrub ok 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 4.4 scrub starts 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: 4.4 scrub ok 2024-03-21T14:20:32.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:31 smithi118 ceph-mon[33862]: osdmap e400: 8 total, 8 up, 7 in 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: pgmap v844: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 569 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.03k op/s; 763/27906 objects misplaced (2.734%); 4.1 MiB/s, 65 objects/s recovering 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: osdmap e399: 8 total, 8 up, 7 in 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 1.6 scrub starts 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 1.6 scrub ok 2024-03-21T14:20:32.189 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 4.4 scrub starts 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: 4.4 scrub ok 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[29576]: osdmap e400: 8 total, 8 up, 7 in 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: pgmap v844: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 569 MiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 961 B/s rd, 0 B/s wr, 1.03k op/s; 763/27906 objects misplaced (2.734%); 4.1 MiB/s, 65 objects/s recovering 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: osdmap e399: 8 total, 8 up, 7 in 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 1.6 scrub starts 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 1.6 scrub ok 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:20:32.190 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:20:32.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 4.4 scrub starts 2024-03-21T14:20:32.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: 4.4 scrub ok 2024-03-21T14:20:32.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:31 smithi003 ceph-mon[40510]: osdmap e400: 8 total, 8 up, 7 in 2024-03-21T14:20:32.227 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:32.262 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:32.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:32.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:32.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:32.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:32.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:32.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:20:32.538 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:32.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:32.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:32.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:32.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:32.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:32.834 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:32.869 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:32.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:32.970 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:33.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:33.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:33.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:20:33.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:33.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:33.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:32 smithi118 ceph-mon[33862]: 3.1a scrub starts 2024-03-21T14:20:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:32 smithi118 ceph-mon[33862]: 3.1a scrub ok 2024-03-21T14:20:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:32 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:20:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:32 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:20:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:32 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:33.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[29576]: 3.1a scrub starts 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[29576]: 3.1a scrub ok 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[40510]: 3.1a scrub starts 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[40510]: 3.1a scrub ok 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:33.284 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:33.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:33.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:33.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:33.448 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:33.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:33.564 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:33.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.698 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:33.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:33.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.833 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:33.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:33.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:33.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:33.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:34.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:34.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.107 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:34.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:33 smithi118 ceph-mon[33862]: pgmap v847: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 451 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 763/22206 objects misplaced (3.436%) 2024-03-21T14:20:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:33 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:20:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:33 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:20:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:33 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:20:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:33 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:20:34.209 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:34.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[29576]: pgmap v847: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 451 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 763/22206 objects misplaced (3.436%) 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[40510]: pgmap v847: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 451 MiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 474 op/s; 763/22206 objects misplaced (3.436%) 2024-03-21T14:20:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:20:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:20:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:20:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:33 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:20:34.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:34.376 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:34.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:34.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:34.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:34.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:34.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:34.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:34.778 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:34.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:34.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:34.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:35.013 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:35.070 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:35.172 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:35.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:35.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:35.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:35.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:35.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:35.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:35.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:35.718 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 153373 objects 2024-03-21T14:20:35.718 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :124.499 2024-03-21T14:20:35.719 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:35.719 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:35.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:35.767 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:35.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:35.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:35.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:35.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:35 smithi118 ceph-mon[33862]: pgmap v848: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 257 MiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 194/12906 objects misplaced (1.503%); 2.4 MiB/s, 38 objects/s recovering 2024-03-21T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:35 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:35 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:35 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:20:35.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:35 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:20:35.995 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[29576]: pgmap v848: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 257 MiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 194/12906 objects misplaced (1.503%); 2.4 MiB/s, 38 objects/s recovering 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[40510]: pgmap v848: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 257 MiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 942 op/s; 194/12906 objects misplaced (1.503%); 2.4 MiB/s, 38 objects/s recovering 2024-03-21T14:20:36.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:20:36.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:20:36.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:20:36.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:35 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:20:36.019 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:36.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:36.121 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:36.146 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:36.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:36.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:36.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:36.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:36.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:36.400 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:36.502 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:36.526 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:36.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:36.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:36.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:36.700 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:36.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:36 smithi003 ceph-mon[29576]: 1.1 scrub starts 2024-03-21T14:20:36.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:36 smithi003 ceph-mon[29576]: 1.1 scrub ok 2024-03-21T14:20:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:20:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:20:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:20:36.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:36 smithi003 ceph-mon[40510]: 1.1 scrub starts 2024-03-21T14:20:36.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:36 smithi003 ceph-mon[40510]: 1.1 scrub ok 2024-03-21T14:20:36.756 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.782 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:36.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:36.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:36.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:36.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:36.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:37.009 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:37.033 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.034 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:36 smithi118 ceph-mon[33862]: 1.1 scrub starts 2024-03-21T14:20:37.034 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:36 smithi118 ceph-mon[33862]: 1.1 scrub ok 2024-03-21T14:20:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:37.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:37.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:37.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:37.285 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:37.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.413 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:37.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:20:37.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:37.519 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:20:37.519 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:20:37.531 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:20:37.538 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:37.552 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:20:37.572 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:20:37.593 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:20:37.613 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:20:37.634 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:20:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:20:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:20:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:20:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:20:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:20:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:20:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:20:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:20:37.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:37.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:20:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:20:37.725 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T14:20:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[29576]: pgmap v849: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 257 MiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 194/12906 objects misplaced (1.503%); 2.0 MiB/s, 31 objects/s recovering 2024-03-21T14:20:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[29576]: 1.a scrub starts 2024-03-21T14:20:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[29576]: 1.a scrub ok 2024-03-21T14:20:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4073205849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:20:37.754 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2749405532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:37.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[40510]: pgmap v849: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 257 MiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 194/12906 objects misplaced (1.503%); 2.0 MiB/s, 31 objects/s recovering 2024-03-21T14:20:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[40510]: 1.a scrub starts 2024-03-21T14:20:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[40510]: 1.a scrub ok 2024-03-21T14:20:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4073205849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:20:37.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2749405532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:37.755 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:37.767 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:37.792 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:37.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:37.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:38.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:38.043 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:38.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:38.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:38.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:37 smithi118 ceph-mon[33862]: pgmap v849: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 257 MiB data, 7.3 GiB used, 618 GiB / 626 GiB avail; 0 B/s wr, 773 op/s; 194/12906 objects misplaced (1.503%); 2.0 MiB/s, 31 objects/s recovering 2024-03-21T14:20:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:37 smithi118 ceph-mon[33862]: 1.a scrub starts 2024-03-21T14:20:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:37 smithi118 ceph-mon[33862]: 1.a scrub ok 2024-03-21T14:20:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4073205849' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:20:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2749405532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:38.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:38.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:38.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:38.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:38.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:38.523 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:38.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:38.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:38.648 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.675 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:38.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:38.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:38.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:38.754 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: 1.8 scrub starts 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: 1.8 scrub ok 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4073205849' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: osdmap e401: 8 total, 8 up, 7 in 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:38.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: 1.8 scrub starts 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: 1.8 scrub ok 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4073205849' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: osdmap e401: 8 total, 8 up, 7 in 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:20:38.756 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:38 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:20:38.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:38.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:38.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:38.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:38.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:38.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: 1.8 scrub starts 2024-03-21T14:20:38.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: 1.8 scrub ok 2024-03-21T14:20:38.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4073205849' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:20:38.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: osdmap e401: 8 total, 8 up, 7 in 2024-03-21T14:20:38.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:20:38.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:38 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:20:38.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T14:20:39.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.054 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:39.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:39.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:39.183 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:39.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:39.313 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:39.415 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:39.441 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:39.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:39.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:39.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:39.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:39.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:39.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:39.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:39.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:39.926 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:39.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:39.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[29576]: pgmap v851: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-21T14:20:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[29576]: 6.17 scrub starts 2024-03-21T14:20:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[29576]: 6.17 scrub ok 2024-03-21T14:20:40.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[29576]: osdmap e402: 8 total, 8 up, 7 in 2024-03-21T14:20:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[40510]: pgmap v851: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-21T14:20:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[40510]: 6.17 scrub starts 2024-03-21T14:20:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[40510]: 6.17 scrub ok 2024-03-21T14:20:40.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:39 smithi003 ceph-mon[40510]: osdmap e402: 8 total, 8 up, 7 in 2024-03-21T14:20:40.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:40.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:40.117 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:39 smithi118 ceph-mon[33862]: pgmap v851: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 1.24k op/s; 1.7 MiB/s, 27 objects/s recovering 2024-03-21T14:20:40.117 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:39 smithi118 ceph-mon[33862]: 6.17 scrub starts 2024-03-21T14:20:40.117 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:39 smithi118 ceph-mon[33862]: 6.17 scrub ok 2024-03-21T14:20:40.117 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:39 smithi118 ceph-mon[33862]: osdmap e402: 8 total, 8 up, 7 in 2024-03-21T14:20:40.169 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:40.181 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:40.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.207 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:40.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:40.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:40.333 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:40.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:40.463 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:40.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:40.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:40.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:40.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:40.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:40.767 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-21T14:20:40.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-21T14:20:40.817 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:40.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:40.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:40.972 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:40.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:40 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:20:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:40 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:20:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:40 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:20:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:40 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:20:41.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:41.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:40 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:20:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:40 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:20:41.202 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:41.230 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:41.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:41.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:41.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:41.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:41.613 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:41.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:20:41.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:41.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:41.757 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:20:41.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:41.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:41.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:41.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[29576]: pgmap v853: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 1.5 MiB/s, 25 objects/s recovering 2024-03-21T14:20:41.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:20:41.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:20:41.953 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1883907685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:41.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[40510]: pgmap v853: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 1.5 MiB/s, 25 objects/s recovering 2024-03-21T14:20:41.953 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:20:41.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:20:41.954 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1883907685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:41.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:41.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:41.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:41.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:42.090 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:41 smithi118 ceph-mon[33862]: pgmap v853: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 892 op/s; 1.5 MiB/s, 25 objects/s recovering 2024-03-21T14:20:42.090 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:41 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:20:42.090 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:41 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:20:42.090 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1883907685' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:20:42.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:42.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:42.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:42.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:42.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:42.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:42.371 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:42.473 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:42.498 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:42.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:42.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:42.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:42.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:42.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:42.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:42.879 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:42.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:42.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1883907685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: osdmap e403: 8 total, 8 up, 7 in 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:20:43.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1883907685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: osdmap e403: 8 total, 8 up, 7 in 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:20:43.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:42 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:20:43.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.006 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:43.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1883907685' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: osdmap e403: 8 total, 8 up, 7 in 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:20:43.060 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:42 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:20:43.085 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:191648 8.18 8:1e671e02:::benchmark_data_smithi003_110242_object60731:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 542.33437199299999, 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99582953799999996, 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.086 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 191648 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749047+0000", 2024-03-21T14:20:43.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749048+0000", 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6000000000000003e-07 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749052+0000", 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.544e-06 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749074+0000", 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.1254999999999999e-05 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749082+0000", 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5469999999999997e-06 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749377+0000", 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029512199999999998 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:40.749386+0000", 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8370000000000003e-06 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739054+0000", 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98966760600000003 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739210+0000", 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015671000000000001 2024-03-21T14:20:43.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.739479+0000", 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00026850799999999998 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744724+0000", 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052452610000000002 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744795+0000", 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0326000000000004e-05 2024-03-21T14:20:43.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744804+0000", 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3680000000000008e-06 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744841+0000", 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7134000000000001e-05 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:11:41.744877+0000", 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5559999999999998e-05 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:219191 8.5 8:a034a08e:::benchmark_data_smithi003_110242_object88274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 513.55500518099996, 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0133928029999999, 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:43.093 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 219191 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528414+0000", 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528415+0000", 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3300000000000001e-07 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528419+0000", 2024-03-21T14:20:43.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0219999999999998e-06 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528427+0000", 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5499999999999995e-06 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528435+0000", 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.199e-06 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528488+0000", 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3267999999999999e-05 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:09.528497+0000", 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6379999999999999e-06 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536200+0000", 2024-03-21T14:20:43.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.007702536 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536291+0000", 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0702e-05 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,7", 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.536577+0000", 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00028596900000000001 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.098 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.539703+0000", 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003126396 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541643+0000", 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001940083 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541733+0000", 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0246000000000003e-05 2024-03-21T14:20:43.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541768+0000", 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4150000000000003e-05 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:10.541807+0000", 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9211000000000001e-05 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24350.0:2962 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94450053602048 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e296)", 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:58.649678+0000", 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 284.43374060000002, 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9659854800000001, 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24350", 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2962 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649678+0000", 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649678+0000", 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649679+0000", 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6400000000000002e-07 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649685+0000", 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4090000000000002e-06 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649693+0000", 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6959999999999994e-06 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649701+0000", 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4109999999999999e-06 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649762+0000", 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1153000000000002e-05 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:58.649771+0000", 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6660000000000003e-06 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.596536+0000", 2024-03-21T14:20:43.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94676457700000005 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.596548+0000", 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1653e-05 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606082+0000", 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0095339270000001 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.106 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606255+0000", 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017383599999999999 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.615664+0000", 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0094084879999999996 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24350.0:2963 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94450053602048 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e297)", 2024-03-21T14:20:43.107 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:15:59.599358+0000", 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 283.48406100099999, 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0164483479999999, 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24350", 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 2963 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599358+0000", 2024-03-21T14:20:43.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599358+0000", 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599360+0000", 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.575e-06 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599367+0000", 2024-03-21T14:20:43.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8530000000000003e-06 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599378+0000", 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1440000000000001e-05 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599387+0000", 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.636e-06 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599503+0000", 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000116882 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:15:59.599515+0000", 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1060000000000001e-05 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606611+0000", 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0070960019999999 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.606687+0000", 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6530000000000001e-05 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:00.615806+0000", 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0091193699999999999 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.24350.0:3182 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94450053602048 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-03-21T14:20:43.112 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:17:47.116552+0000", 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 175.966867444, 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0192424769999999, 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "started", 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.24350", 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1675140515", 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 3182 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116552+0000", 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116552+0000", 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116553+0000", 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.136e-06 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116558+0000", 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.164e-06 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116572+0000", 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3713000000000001e-05 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.116580+0000", 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.123e-06 2024-03-21T14:20:43.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.117294+0000", 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000713855 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op must wait for map", 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.117300+0000", 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8260000000000004e-06 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.125149+0000", 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0078487069999999999 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:47.125158+0000", 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.9199999999999993e-06 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:48.132480+0000", 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.00732208 2024-03-21T14:20:43.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:48.132614+0000", 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000134101 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:17:48.135794+0000", 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031798519999999999 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:244541 10.4 10:25760c27:::benchmark_data_smithi003_160672_object91165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:34.553688+0000", 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.529730780999998, 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1997849039999999, 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244541 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553688+0000", 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553688+0000", 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553690+0000", 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-06 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553695+0000", 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0379999999999999e-06 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553705+0000", 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0858e-05 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553713+0000", 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0150000000000003e-06 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553786+0000", 2024-03-21T14:20:43.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2883e-05 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.553798+0000", 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1793999999999999e-05 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670537+0000", 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.116739369 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670552+0000", 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4158999999999999e-05 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.749439+0000", 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.078887624 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.749576+0000", 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000136983 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.749658+0000", 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1378999999999995e-05 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753095+0000", 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034375759999999999 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753137+0000", 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1597000000000002e-05 2024-03-21T14:20:43.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753237+0000", 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000100633 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753297+0000", 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9413999999999997e-05 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753366+0000", 2024-03-21T14:20:43.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9418000000000005e-05 2024-03-21T14:20:43.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753473+0000", 2024-03-21T14:20:43.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000106931 2024-03-21T14:20:43.129 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.129 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.129 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:244590 10.4 10:27f2178d:::benchmark_data_smithi003_160672_object91214:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:34.613953+0000", 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.469466213000004, 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.146247767, 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244590 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.130 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.613953+0000", 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.613953+0000", 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.613954+0000", 2024-03-21T14:20:43.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0500000000000002e-07 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.613958+0000", 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2860000000000001e-06 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.613968+0000", 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0316000000000001e-05 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.613976+0000", 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9419999999999995e-06 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.614027+0000", 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1044999999999997e-05 2024-03-21T14:20:43.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.614036+0000", 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3499999999999997e-06 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670592+0000", 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.056556021999999997 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670599+0000", 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0990000000000001e-06 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.751716+0000", 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0811172259999999 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.751793+0000", 2024-03-21T14:20:43.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6808000000000001e-05 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.751864+0000", 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1217000000000006e-05 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753674+0000", 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001809827 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.757019+0000", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0033447260000000001 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.757250+0000", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000231449 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760096+0000", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00284564 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760133+0000", 2024-03-21T14:20:43.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6773000000000002e-05 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760201+0000", 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8136e-05 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:244605 10.4 10:28059364:::benchmark_data_smithi003_160672_object91229:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:34.626121+0000", 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.457298000999998, 2024-03-21T14:20:43.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.134235514, 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244605 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626121+0000", 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626121+0000", 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626122+0000", 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5299999999999999e-07 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626124+0000", 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.6560000000000001e-06 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626132+0000", 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3989999999999998e-06 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626137+0000", 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7620000000000001e-06 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626196+0000", 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8604999999999999e-05 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.626208+0000", 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.206e-05 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670636+0000", 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.044428002000000001 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670646+0000", 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.027e-05 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753715+0000", 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0830686 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753834+0000", 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118621 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.753954+0000", 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120625 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.757214+0000", 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032603060000000001 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.759979+0000", 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0027649189999999998 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760020+0000", 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0649999999999999e-05 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760267+0000", 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00024655199999999998 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760297+0000", 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0139999999999999e-05 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760357+0000", 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9893999999999999e-05 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.145 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:244634 10.4 10:28be6c71:::benchmark_data_smithi003_160672_object91258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:34.651971+0000", 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.431448243999995, 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1085270030000001, 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244634 2024-03-21T14:20:43.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.651971+0000", 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.651971+0000", 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.651972+0000", 2024-03-21T14:20:43.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2900000000000002e-07 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.651976+0000", 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2930000000000002e-06 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.651985+0000", 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5399999999999996e-06 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.651990+0000", 2024-03-21T14:20:43.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5099999999999998e-06 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.652068+0000", 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7843000000000007e-05 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.652079+0000", 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1222e-05 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670673+0000", 2024-03-21T14:20:43.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.018593937000000001 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670678+0000", 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4240000000000001e-06 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.755350+0000", 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0846715099999999 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.755473+0000", 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012345800000000001 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.755599+0000", 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012561399999999999 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.758653+0000", 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030544410000000002 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760058+0000", 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014047 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760231+0000", 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017284599999999999 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760390+0000", 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000158708 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760424+0000", 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3915999999999999e-05 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760498+0000", 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4212e-05 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:244644 10.4 10:2e6a7f44:::benchmark_data_smithi003_160672_object91268:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:34.665343+0000", 2024-03-21T14:20:43.153 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.418075889999997, 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.098884594, 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244644 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665343+0000", 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665343+0000", 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665344+0000", 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2099999999999996e-07 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665347+0000", 2024-03-21T14:20:43.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.943e-06 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665354+0000", 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4059999999999999e-06 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665360+0000", 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6149999999999996e-06 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665408+0000", 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8099999999999997e-05 2024-03-21T14:20:43.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.665416+0000", 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6850000000000001e-06 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670704+0000", 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0052884849999999999 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.670714+0000", 2024-03-21T14:20:43.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8209999999999997e-06 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.757283+0000", 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.086569331 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.757407+0000", 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012345600000000001 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.757479+0000", 2024-03-21T14:20:43.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2467999999999998e-05 2024-03-21T14:20:43.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.159 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:43.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760066+0000", 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002587102 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764024+0000", 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039576699999999999 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764061+0000", 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6563000000000002e-05 2024-03-21T14:20:43.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764114+0000", 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2981e-05 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764152+0000", 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8347999999999999e-05 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764228+0000", 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5899e-05 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:244647 10.4 10:2b446316:::benchmark_data_smithi003_160672_object91271:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e366)", 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:34.676750+0000", 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 68.406668972999995, 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0878472779999999, 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.163 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 244647 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676750+0000", 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676750+0000", 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676751+0000", 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.78e-07 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676755+0000", 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9910000000000004e-06 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676766+0000", 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.099e-05 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676774+0000", 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5610000000000001e-06 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676831+0000", 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7933999999999998e-05 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:34.676845+0000", 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3525e-05 2024-03-21T14:20:43.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.758703+0000", 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.081857987 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.758814+0000", 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000110913 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.758949+0000", 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000135234 2024-03-21T14:20:43.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.760238+0000", 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0012892509999999999 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764303+0000", 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040644670000000004 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764344+0000", 2024-03-21T14:20:43.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0874999999999997e-05 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764401+0000", 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7623e-05 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764466+0000", 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4339999999999997e-05 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:35.764597+0000", 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013170900000000001 2024-03-21T14:20:43.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278304 10.4 10:283d363b:::benchmark_data_smithi003_160672_object124928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.676141+0000", 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.407277835000002, 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0449981450000001, 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.170 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278304 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676141+0000", 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676141+0000", 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676142+0000", 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1650000000000001e-06 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676148+0000", 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6520000000000003e-06 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676162+0000", 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3858e-05 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676172+0000", 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5920000000000007e-06 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676228+0000", 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6548000000000001e-05 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676444+0000", 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021573799999999999 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676591+0000", 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014695800000000001 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.682437+0000", 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058462890000000002 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.685276+0000", 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0028391810000000001 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.685289+0000", 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2941e-05 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721055+0000", 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0357660070000001 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721139+0000", 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4215999999999995e-05 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.175 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278306 10.4 10:2332fb0b:::benchmark_data_smithi003_160672_object124930:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e386)", 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.676195+0000", 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.407224051, 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0454411729999999, 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278306 2024-03-21T14:20:43.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676195+0000", 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676195+0000", 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676196+0000", 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1600000000000001e-07 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676199+0000", 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0460000000000001e-06 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676206+0000", 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.401e-06 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.178 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.676212+0000", 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1890000000000004e-06 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.679200+0000", 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029876400000000002 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.679360+0000", 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000160342 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,1,3", 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.679455+0000", 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4254999999999998e-05 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.685322+0000", 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0058675230000000004 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.685331+0000", 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7020000000000007e-06 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.180 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721573+0000", 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.036241872 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721636+0000", 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3587e-05 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278317 10.4 10:21b72867:::benchmark_data_smithi003_160672_object124941:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.181 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.690859+0000", 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.392560508000003, 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0726314589999999, 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278317 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690859+0000", 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690859+0000", 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690860+0000", 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.178e-06 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690865+0000", 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5470000000000004e-06 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.183 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690879+0000", 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3563999999999999e-05 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690887+0000", 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4970000000000001e-06 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690968+0000", 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.072e-05 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.184 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.690978+0000", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9280000000000005e-06 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721652+0000", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.030674098 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721735+0000", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3294999999999994e-05 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.721819+0000", 2024-03-21T14:20:43.185 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3141999999999998e-05 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726210+0000", 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043911970000000003 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726227+0000", 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6942000000000001e-05 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.763419+0000", 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037192416999999998 2024-03-21T14:20:43.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.763456+0000", 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6424999999999998e-05 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.763490+0000", 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4508999999999998e-05 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.187 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278328 10.4 10:29209135:::benchmark_data_smithi003_160672_object124952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.703410+0000", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.380008652000001, 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0630221289999999, 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278328 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703410+0000", 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.188 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703410+0000", 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703412+0000", 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4130000000000001e-06 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703418+0000", 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9819999999999996e-06 2024-03-21T14:20:43.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703432+0000", 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4635e-05 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703442+0000", 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.645e-06 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703504+0000", 2024-03-21T14:20:43.190 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2088000000000006e-05 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.703525+0000", 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.0967000000000001e-05 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.723534+0000", 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.020008955 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.191 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.723655+0000", 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120444 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.723780+0000", 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012514000000000001 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726235+0000", 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002454819 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.192 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726259+0000", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4839999999999999e-05 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766327+0000", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.040067545000000003 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766384+0000", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6956999999999997e-05 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.193 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766433+0000", 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8699000000000002e-05 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278336 10.4 10:2c531c98:::benchmark_data_smithi003_160672_object124960:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.723824+0000", 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.359595022000001, 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0427399049999999, 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.194 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278336 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723824+0000", 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723824+0000", 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.195 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723831+0000", 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3839999999999999e-06 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723836+0000", 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6959999999999996e-06 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723846+0000", 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4280000000000002e-06 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.196 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723853+0000", 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.401e-06 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723896+0000", 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3321000000000001e-05 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.723903+0000", 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0210000000000002e-06 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.197 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.725097+0000", 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0011938600000001 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.725171+0000", 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4004000000000005e-05 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.725244+0000", 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2702999999999998e-05 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.198 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.728665+0000", 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003421398 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.728680+0000", 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.434e-05 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766490+0000", 2024-03-21T14:20:43.199 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.037810847000000002 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766524+0000", 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.3887999999999998e-05 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766564+0000", 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9613999999999997e-05 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.200 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278344 10.4 10:2155ba6b:::benchmark_data_smithi003_160672_object124968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.736595+0000", 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.346824265000002, 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0301097109999999, 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.201 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278344 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736595+0000", 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736595+0000", 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.202 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736595+0000", 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7000000000000005e-07 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736598+0000", 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4710000000000001e-06 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736606+0000", 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8099999999999998e-06 2024-03-21T14:20:43.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736610+0000", 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.0010000000000003e-06 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736646+0000", 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6581000000000001e-05 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.736652+0000", 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4240000000000001e-06 2024-03-21T14:20:43.204 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726297+0000", 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98964512299999996 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726406+0000", 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00010942600000000001 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.726531+0000", 2024-03-21T14:20:43.205 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000125189 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.731056+0000", 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0045242060000000002 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.731104+0000", 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8149999999999998e-05 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.206 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766615+0000", 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035511210000000001 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766665+0000", 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9799999999999998e-05 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766705+0000", 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9749999999999997e-05 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.207 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278345 10.4 10:2369fb8c:::benchmark_data_smithi003_160672_object124969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.737237+0000", 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.346182476999999, 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0296139550000001, 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278345 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.208 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737237+0000", 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737237+0000", 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737237+0000", 2024-03-21T14:20:43.209 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2099999999999996e-07 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737241+0000", 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.287e-06 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737251+0000", 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0365e-05 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.210 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737259+0000", 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9750000000000007e-06 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737309+0000", 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0204000000000002e-05 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.737317+0000", 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1929999999999993e-06 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.211 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.727547+0000", 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99022928499999996 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.727616+0000", 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9393999999999993e-05 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.727684+0000", 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8163000000000005e-05 2024-03-21T14:20:43.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.731157+0000", 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0034723179999999998 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.731171+0000", 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4259e-05 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766778+0000", 2024-03-21T14:20:43.213 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035607315 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766820+0000", 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1896e-05 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766851+0000", 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0580000000000002e-05 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278359 10.4 10:2ffee8aa:::benchmark_data_smithi003_160672_object124983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.749700+0000", 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.333719297000002, 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0172782650000001, 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278359 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.215 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749700+0000", 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749700+0000", 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749701+0000", 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1260000000000001e-06 2024-03-21T14:20:43.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749705+0000", 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.76e-06 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749714+0000", 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5370000000000003e-06 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749721+0000", 2024-03-21T14:20:43.217 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6030000000000001e-06 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749776+0000", 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5077999999999997e-05 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.749785+0000", 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3670000000000003e-06 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.218 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.728705+0000", 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97891926600000001 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.728796+0000", 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.1612000000000003e-05 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.728870+0000", 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.394e-05 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.219 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.731178+0000", 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023075959999999999 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.731197+0000", 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9805000000000001e-05 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766902+0000", 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.035704459000000001 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.220 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766949+0000", 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7098000000000002e-05 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.766978+0000", 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9017999999999999e-05 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:278367 10.4 10:22117a3d:::benchmark_data_smithi003_160672_object124991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e387)", 2024-03-21T14:20:43.221 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:20:09.753528+0000", 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 33.329890902000002, 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0164473089999999, 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 278367 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:43.222 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753528+0000", 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753528+0000", 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753529+0000", 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8100000000000002e-07 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.223 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753532+0000", 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5410000000000001e-06 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753543+0000", 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0515e-05 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753550+0000", 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9829999999999999e-06 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753598+0000", 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7654999999999997e-05 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:09.753603+0000", 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4929999999999998e-06 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.729836+0000", 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.97623279699999999 2024-03-21T14:20:43.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.729958+0000", 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012250999999999999 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,3", 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.730081+0000", 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000122454 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.732470+0000", 2024-03-21T14:20:43.226 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023893719999999999 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.733816+0000", 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013462319999999999 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.769873+0000", 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.036057049000000001 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:43.227 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.769936+0000", 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2786000000000005e-05 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:20:10.769975+0000", 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9240999999999999e-05 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:43.228 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:20:43.252 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:43.353 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.377 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:43.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:43.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:43.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:43.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:43.629 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:43.731 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:43.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:43.776 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T14:20:43.780 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:43.781 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:43.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:43.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:43.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:43.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: pgmap v855: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 675 op/s 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3624522825' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:20:44.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: pgmap v855: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 675 op/s 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3624522825' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:20:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:43 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:20:44.008 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:44.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:44.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:44.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:20:44.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:20:44.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: pgmap v855: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 0 B/s wr, 675 op/s 2024-03-21T14:20:44.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3624522825' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:20:44.160 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:20:44.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:20:44.161 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:43 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:20:44.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:20:44.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:44.259 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:44.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:44.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:44.427 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:44.486 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.511 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:44.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:44.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T14:20:44.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.639 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:44.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:44.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:44.765 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:44.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:20:44.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:20:44.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:20:44.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[29576]: osdmap e404: 8 total, 8 up, 7 in 2024-03-21T14:20:44.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[40510]: osdmap e404: 8 total, 8 up, 7 in 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:20:44.812 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:44 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:20:44.863 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:44 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:20:44.864 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:44 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:20:44.864 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:44 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:20:44.864 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:44 smithi118 ceph-mon[33862]: osdmap e404: 8 total, 8 up, 7 in 2024-03-21T14:20:44.864 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:44 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:20:44.864 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:44 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:20:44.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:44.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:44.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:44.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:45.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:45.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:45.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:45.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:45.245 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.271 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:45.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:45.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:45.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.398 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:45.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:45.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:45.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:45.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:45.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:45.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:45.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:45.752 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:20:45.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:45.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: pgmap v857: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: osdmap e405: 8 total, 8 up, 7 in 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: 6.f deep-scrub starts 2024-03-21T14:20:45.889 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:45 smithi118 ceph-mon[33862]: 6.f deep-scrub ok 2024-03-21T14:20:45.901 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:45.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:46.003 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:46.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: pgmap v857: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:20:46.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: osdmap e405: 8 total, 8 up, 7 in 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: 6.f deep-scrub starts 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[40510]: 6.f deep-scrub ok 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: pgmap v857: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: osdmap e405: 8 total, 8 up, 7 in 2024-03-21T14:20:46.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:20:46.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:20:46.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: 6.f deep-scrub starts 2024-03-21T14:20:46.006 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:45 smithi003 ceph-mon[29576]: 6.f deep-scrub ok 2024-03-21T14:20:46.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:46.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:46.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.157 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:46.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:46.171 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:20:46.172 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:20:46.172 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-03-21T14:20:46.172 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:20:46.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:46.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:46.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:46.407 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:46.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.535 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:46.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:46.635 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:46.659 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:46.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:20:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:20:46.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:46.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:20:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:46.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:46.809 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:46 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:20:46.809 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:46 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:20:46.810 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:46 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:20:46.810 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:46 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:20:46.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:46.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:46.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:47.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:47.026 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:47.038 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:47.053 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:47.053 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:47.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:47.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:47.193 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:20:47.194 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:46 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:20:47.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:47.288 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:47.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:47.390 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:47.414 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:47.431 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:20:47.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.541 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:47.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:47.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:47.665 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:47.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:47.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:47.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:47.856 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:20:47.875 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:20:47.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:47.896 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:20:47.916 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:20:47.919 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:47.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:47.937 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:20:47.958 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:20:47.978 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: 4.1f deep-scrub starts 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: 4.1f deep-scrub ok 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: pgmap v859: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: 10.c deep-scrub starts 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: 10.c deep-scrub ok 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:47.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: 4.1f deep-scrub starts 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: 4.1f deep-scrub ok 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: pgmap v859: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: 10.c deep-scrub starts 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: 10.c deep-scrub ok 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:47.980 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:47.999 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:20:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:20:48.021 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:48.045 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:48.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:48.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: 4.1f deep-scrub starts 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: 4.1f deep-scrub ok 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: pgmap v859: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: 10.c deep-scrub starts 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: 10.c deep-scrub ok 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:48.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:20:48.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:20:48.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:48.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.297 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:48.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:48.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.422 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:48.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:48.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:48.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:48.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:48.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:48.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:48.798 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:48.802 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_3' removed 2024-03-21T14:20:48.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:48.908 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:48.908 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:48.908 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:48.908 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:48 smithi118 ceph-mon[33862]: osdmap e406: 8 total, 8 up, 7 in 2024-03-21T14:20:48.908 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1762543578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:48.908 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1604156811' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:20:48.908 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:48 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:20:48.909 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:48 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:20:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:48.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:49.026 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:49.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:49.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:49.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:49.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[29576]: osdmap e406: 8 total, 8 up, 7 in 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1762543578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1604156811' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[40510]: osdmap e406: 8 total, 8 up, 7 in 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1762543578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1604156811' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:20:49.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:48 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:20:49.280 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:49.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.305 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:49.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:49.406 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:49.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:49.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:49.532 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:49.555 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:49.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:20:49.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:49.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:49.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:49.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:49.885 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:20:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:20:49.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: pgmap v861: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1604156811' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: osdmap e407: 8 total, 8 up, 7 in 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: 5.d deep-scrub starts 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: 5.d deep-scrub ok 2024-03-21T14:20:49.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:49 smithi118 ceph-mon[33862]: osdmap e408: 8 total, 8 up, 7 in 2024-03-21T14:20:49.934 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:49.958 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:49.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:50.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:50.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:50.186 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:50.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:20:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: pgmap v861: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1604156811' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: osdmap e407: 8 total, 8 up, 7 in 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: 5.d deep-scrub starts 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: 5.d deep-scrub ok 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[29576]: osdmap e408: 8 total, 8 up, 7 in 2024-03-21T14:20:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: pgmap v861: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1604156811' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: osdmap e407: 8 total, 8 up, 7 in 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: 5.d deep-scrub starts 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: 5.d deep-scrub ok 2024-03-21T14:20:50.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:49 smithi003 ceph-mon[40510]: osdmap e408: 8 total, 8 up, 7 in 2024-03-21T14:20:50.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:50.336 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:50.411 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-21T14:20:50.412 INFO:tasks.radosbench:Beginning radosbench... 2024-03-21T14:20:50.412 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-21T14:20:50.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-21T14:20:50.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:50.478 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:50.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:50.604 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:50.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:50.706 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:50.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:50.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:50.855 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:50.884 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:20:50.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:50.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:50.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[29576]: 4.a deep-scrub starts 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[29576]: 4.a deep-scrub ok 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[40510]: 4.a deep-scrub starts 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[40510]: 4.a deep-scrub ok 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:20:51.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:50 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:20:51.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:50 smithi118 ceph-mon[33862]: 4.a deep-scrub starts 2024-03-21T14:20:51.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:50 smithi118 ceph-mon[33862]: 4.a deep-scrub ok 2024-03-21T14:20:51.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:50 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:20:51.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:50 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:20:51.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:50 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:20:51.055 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:50 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:20:51.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:51.107 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:51.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:51.208 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:51.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:51.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:51.464 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.488 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:51.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:51.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:51.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:51.624 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:51.624 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:51.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:51.738 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:51.840 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.865 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:51.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:51 smithi118 ceph-mon[33862]: pgmap v864: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 763 B/s rd, 0 op/s 2024-03-21T14:20:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:51 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:20:51.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:51 smithi118 ceph-mon[33862]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T14:20:51.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:51 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:20:51.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:51 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:20:51.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:51 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:20:51.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:51.990 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:51.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:52.092 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:52.105 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[29576]: pgmap v864: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 763 B/s rd, 0 op/s 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[29576]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[40510]: pgmap v864: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 619 GiB / 626 GiB avail; 763 B/s rd, 0 op/s 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[40510]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-21T14:20:52.106 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:20:52.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:20:52.107 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:51 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:20:52.117 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:52.205 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:52.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:52.244 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:52.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:20:52.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:52.379 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:52.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:52.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:52.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:52.556 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:52.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:52.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:52.733 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.757 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:52.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:52.772 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:52.815 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:20:52.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:52.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:52.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3287081947' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:20:52.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3287081947' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:20:52.895 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:52 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:20:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:20:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:20:52.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:20:52.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:20:52.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:20:52.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:20:52.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3287081947' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:20:52.932 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:52 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:20:52.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.011 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:53.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:53.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:53.137 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:53.239 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:53.264 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:53.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:53.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:53.389 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:53.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:53.491 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:53.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:53.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:53.639 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:53.741 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.769 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:53.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:53.822 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T14:20:53.870 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:53.895 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:53.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:53.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:53.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:54.003 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: pgmap v865: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:20:54.003 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: 5.18 scrub starts 2024-03-21T14:20:54.003 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: 5.18 scrub ok 2024-03-21T14:20:54.003 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: 1.4 scrub starts 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: 1.4 scrub ok 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: osdmap e409: 8 total, 8 up, 7 in 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1302346085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:53 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: pgmap v865: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:20:54.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: 5.18 scrub starts 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: 5.18 scrub ok 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: 1.4 scrub starts 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: 1.4 scrub ok 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: osdmap e409: 8 total, 8 up, 7 in 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1302346085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: pgmap v865: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: 5.18 scrub starts 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: 5.18 scrub ok 2024-03-21T14:20:54.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: 1.4 scrub starts 2024-03-21T14:20:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: 1.4 scrub ok 2024-03-21T14:20:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:20:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:20:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: osdmap e409: 8 total, 8 up, 7 in 2024-03-21T14:20:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1302346085' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:20:54.006 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:53 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:20:54.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:54.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:54.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:54.244 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:54.245 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:54.250 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:54.273 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:54.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.401 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:54.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:54.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:54.525 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:54.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:54.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:54.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.652 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:54.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:54.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:54.777 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:54.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:54 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:20:54.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:54 smithi003 ceph-mon[29576]: osdmap e410: 8 total, 8 up, 7 in 2024-03-21T14:20:54.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:54 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:20:54.833 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:54 smithi003 ceph-mon[40510]: osdmap e410: 8 total, 8 up, 7 in 2024-03-21T14:20:54.858 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T14:20:54.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:20:54.879 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:54.904 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:54.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:54.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:55.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.039 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:55.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:55.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:20:55.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.165 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:55.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:54 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:20:55.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:54 smithi118 ceph-mon[33862]: osdmap e410: 8 total, 8 up, 7 in 2024-03-21T14:20:55.266 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.291 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:55.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:55.291 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:55.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:55.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:55.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:55.518 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.543 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:55.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:55.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:55.669 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:55.770 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:55.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.795 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:55.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:55.896 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:55.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:55.922 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:55.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:55 smithi003 ceph-mon[29576]: pgmap v868: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 958 B/s rd, 0 op/s 2024-03-21T14:20:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:55 smithi003 ceph-mon[40510]: pgmap v868: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 958 B/s rd, 0 op/s 2024-03-21T14:20:56.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:56.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:56.173 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:56.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:56.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:55 smithi118 ceph-mon[33862]: pgmap v868: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 958 B/s rd, 0 op/s 2024-03-21T14:20:56.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:20:56.275 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:56.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:56.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:56.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:56.526 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:56.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.611 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:56.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:56.712 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:56.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:56.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:56.752 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:20:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:20:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:20:56.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:56.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:56.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:56.988 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:56.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:57.082 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:20:57.082 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:20:57.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:57.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:57.215 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:57.239 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:57.320 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:20:57.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:57.346 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:20:57.364 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:57.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:20:57.366 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:20:57.385 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:20:57.405 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:20:57.426 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:20:57.445 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:20:57.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:57.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:20:57.468 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:20:57.468 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:20:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:20:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:20:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:20:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:20:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:20:57.469 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:20:57.493 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:57.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:57.595 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:57.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.620 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:57.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:57.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:20:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:20:57.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:57.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:57.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:57.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:20:57.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:57.870 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:57.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:57.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:20:57.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:57.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:20:58.014 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:20:58.092 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_4' created 2024-03-21T14:20:58.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:58.121 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:58.223 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:58.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:58.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:58 smithi003 ceph-mon[29576]: pgmap v869: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 794 B/s rd, 0 op/s 2024-03-21T14:20:58.290 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3949037799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:58.290 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:58 smithi003 ceph-mon[40510]: pgmap v869: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 794 B/s rd, 0 op/s 2024-03-21T14:20:58.291 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3949037799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:58.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:58.365 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:58 smithi118 ceph-mon[33862]: pgmap v869: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 794 B/s rd, 0 op/s 2024-03-21T14:20:58.365 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3949037799' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:20:58.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:58.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:58.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.500 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:20:58.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:20:58.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:58.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:20:58.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:58.750 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:58.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:58.852 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:58.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:58.877 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:58.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:58.977 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.003 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:59.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:59.070 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-21T14:20:59.106 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:59.129 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:59 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1863447224' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T14:20:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1863447224' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-21T14:20:59.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:20:59 smithi118 ceph-mon[33862]: osdmap e411: 8 total, 8 up, 7 in 2024-03-21T14:20:59.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:20:59.255 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:20:59.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1863447224' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1863447224' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[29576]: osdmap e411: 8 total, 8 up, 7 in 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1863447224' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1863447224' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-21T14:20:59.289 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:20:59 smithi003 ceph-mon[40510]: osdmap e411: 8 total, 8 up, 7 in 2024-03-21T14:20:59.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:20:59.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:59.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:20:59.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:20:59.508 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:59.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:20:59.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:20:59.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.634 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:59.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:20:59.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:20:59.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:20:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:20:59.793 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:20:59.793 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:20:59.793 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:20:59.793 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218154 8.17 8:e9ce89ea:::benchmark_data_smithi003_110242_object87237:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:07.964667+0000", 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 531.82719368699998, 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0141021400000001, 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218154 2024-03-21T14:20:59.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964667+0000", 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964667+0000", 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.795 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964668+0000", 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8999999999999996e-07 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964671+0000", 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6059999999999999e-06 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964679+0000", 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4919999999999996e-06 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.796 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964686+0000", 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8689999999999998e-06 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964753+0000", 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7691e-05 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.797 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.964761+0000", 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9570000000000002e-06 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.974133+0000", 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0093721529999999 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.974256+0000", 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012236699999999999 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.798 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.974600+0000", 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034449200000000001 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978523+0000", 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0039230080000000004 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978568+0000", 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5030000000000001e-05 2024-03-21T14:20:59.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978691+0000", 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000122984 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978732+0000", 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.1128999999999999e-05 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.800 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978769+0000", 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6671999999999997e-05 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218177 8.17 8:e895dc9a:::benchmark_data_smithi003_110242_object87260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:07.984637+0000", 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 531.807223927, 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99749402899999995, 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.801 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218177 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984637+0000", 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.802 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984637+0000", 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984638+0000", 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.547e-06 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984643+0000", 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6609999999999999e-06 2024-03-21T14:20:59.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984652+0000", 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2539999999999998e-06 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984659+0000", 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0500000000000003e-06 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984707+0000", 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7985999999999998e-05 2024-03-21T14:20:59.804 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.984716+0000", 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4570000000000005e-06 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.976740+0000", 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99202411899999998 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.976877+0000", 2024-03-21T14:20:59.805 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013677700000000001 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.977128+0000", 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00025135400000000002 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978779+0000", 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001651146 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.806 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982000+0000", 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003220503 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982052+0000", 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2018000000000002e-05 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982086+0000", 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4187999999999999e-05 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.807 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982131+0000", 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4969000000000003e-05 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.810 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218191 8.17 8:e950946d:::benchmark_data_smithi003_110242_object87274:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:07.992480+0000", 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 531.799381079, 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99410021699999995, 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218191 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.811 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992480+0000", 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992480+0000", 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992481+0000", 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.285e-06 2024-03-21T14:20:59.812 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992486+0000", 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3519999999999997e-06 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992500+0000", 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3328e-05 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992508+0000", 2024-03-21T14:20:59.813 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3340000000000002e-06 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992565+0000", 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7463999999999997e-05 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:07.992571+0000", 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.071e-06 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.814 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978807+0000", 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98623560600000004 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.978924+0000", 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0001171 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.979134+0000", 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000209724 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.815 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982183+0000", 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0030487520000000001 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982198+0000", 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5091e-05 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986498+0000", 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0043003549999999996 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.816 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986536+0000", 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8401e-05 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986580+0000", 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.3353999999999999e-05 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218274 8.17 8:e974f6a2:::benchmark_data_smithi003_110242_object87357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.089562+0000", 2024-03-21T14:20:59.817 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 531.70229878099997, 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89721891499999995, 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218274 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089562+0000", 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089562+0000", 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089563+0000", 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1769999999999999e-06 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089568+0000", 2024-03-21T14:20:59.819 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8099999999999997e-06 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089578+0000", 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.031e-05 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089585+0000", 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6900000000000003e-06 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.820 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089658+0000", 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3218000000000003e-05 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.089669+0000", 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.096e-05 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.980361+0000", 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89069226700000004 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.821 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.980477+0000", 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115708 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.980702+0000", 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000224692 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982267+0000", 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001565227 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.822 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.983750+0000", 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001482997 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986695+0000", 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0029450180000000002 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986746+0000", 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1450999999999999e-05 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.823 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986781+0000", 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4390000000000001e-05 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:218280 8.17 8:edc70fe4:::benchmark_data_smithi003_110242_object87363:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:08.094569+0000", 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 531.697291554, 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89239709199999995, 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.824 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 218280 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094569+0000", 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094569+0000", 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.825 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094571+0000", 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5519999999999999e-06 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094579+0000", 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0739999999999993e-06 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094591+0000", 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2330000000000001e-05 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.826 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094599+0000", 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.1000000000000004e-06 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094659+0000", 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9682000000000002e-05 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.094668+0000", 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.0950000000000002e-06 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.827 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982289+0000", 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88762121900000002 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982357+0000", 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8149000000000001e-05 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.982496+0000", 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000138947 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.828 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986635+0000", 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041387170000000001 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986834+0000", 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019904999999999999 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986847+0000", 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2480000000000001e-05 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.829 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986922+0000", 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5099999999999996e-05 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:08.986966+0000", 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.4597e-05 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.830 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228274 8.6 8:60f051e7:::benchmark_data_smithi003_110242_object97357:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.562509+0000", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 519.229351318, 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99705723000000002, 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228274 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562509+0000", 2024-03-21T14:20:59.831 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562509+0000", 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562511+0000", 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9269999999999999e-06 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562519+0000", 2024-03-21T14:20:59.832 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9959999999999993e-06 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562533+0000", 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3838e-05 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.562545+0000", 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2347000000000001e-05 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.563036+0000", 2024-03-21T14:20:59.833 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00049081699999999995 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.563049+0000", 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2536e-05 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.544628+0000", 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98157942899999995 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.834 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.544752+0000", 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012386 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.545144+0000", 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00039143199999999999 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556353+0000", 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011209566000000001 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.835 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559313+0000", 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002959602 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559349+0000", 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6090999999999997e-05 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559429+0000", 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9766999999999997e-05 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.836 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559467+0000", 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.8642e-05 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559567+0000", 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9380000000000001e-05 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228413 8.6 8:67cc4f9b:::benchmark_data_smithi003_110242_object97496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.646732+0000", 2024-03-21T14:20:59.837 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 519.14512834200002, 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91498634899999998, 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228413 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646732+0000", 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.838 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646732+0000", 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646734+0000", 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.593e-06 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646740+0000", 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8479999999999997e-06 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.839 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646752+0000", 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2476e-05 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646761+0000", 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5229999999999995e-06 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646825+0000", 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3948999999999994e-05 2024-03-21T14:20:59.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.646834+0000", 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7729999999999994e-06 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556388+0000", 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.90955385300000002 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556477+0000", 2024-03-21T14:20:59.841 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8911000000000003e-05 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.556691+0000", 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00021380800000000001 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559386+0000", 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002694723 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.842 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559394+0000", 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3219999999999993e-06 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561494+0000", 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020995200000000001 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561599+0000", 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000105534 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.843 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561635+0000", 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.6341999999999999e-05 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561719+0000", 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3174e-05 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15132.0:228445 8.6 8:60e0e6c0:::benchmark_data_smithi003_110242_object97528:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e225)", 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:12:20.667121+0000", 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 519.12473918600006, 2024-03-21T14:20:59.844 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89474894699999996, 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15132", 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/1946303460", 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 228445 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667121+0000", 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.845 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667121+0000", 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667123+0000", 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.53e-06 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667129+0000", 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.964e-06 2024-03-21T14:20:59.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667143+0000", 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3947999999999999e-05 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667151+0000", 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2770000000000005e-06 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667212+0000", 2024-03-21T14:20:59.847 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0946999999999998e-05 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:20.667221+0000", 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6729999999999998e-06 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.557860+0000", 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89063941499999999 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.558000+0000", 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013936800000000001 2024-03-21T14:20:59.848 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,2,6", 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.558236+0000", 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00023654500000000001 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.559399+0000", 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011625610000000001 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.849 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561529+0000", 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021306739999999999 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561560+0000", 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0176e-05 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561755+0000", 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000195596 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.850 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561788+0000", 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2755000000000001e-05 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:12:21.561870+0000", 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2517999999999998e-05 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268617 9.7 9:eb313569:::benchmark_data_smithi003_135814_object127883:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.862294+0000", 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.92956688800001, 2024-03-21T14:20:59.851 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.036352554, 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268617 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862294+0000", 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.852 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862294+0000", 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862295+0000", 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4399999999999998e-07 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862300+0000", 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8189999999999999e-06 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.853 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.866340+0000", 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00404061 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.866358+0000", 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7586000000000001e-05 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868673+0000", 2024-03-21T14:20:59.854 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023157519999999999 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868683+0000", 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8130000000000007e-06 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898576+0000", 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0298928190000001 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.855 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898646+0000", 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0210999999999996e-05 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268623 9.5 9:ab08bb5c:::benchmark_data_smithi003_135814_object127889:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.868534+0000", 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.92332632900002, 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0204479879999999, 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.856 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268623 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868534+0000", 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868534+0000", 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.857 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868535+0000", 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0920000000000001e-06 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868541+0000", 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6110000000000001e-06 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868551+0000", 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.747e-06 2024-03-21T14:20:59.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868561+0000", 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9510000000000001e-06 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.869234+0000", 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000673517 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.869244+0000", 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3009999999999995e-06 2024-03-21T14:20:59.859 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.887465+0000", 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.018221448 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.888982+0000", 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0015173210000000001 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.860 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268627 9.8 9:125f1641:::benchmark_data_smithi003_135814_object127893:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.868574+0000", 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.923286308, 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0303467589999999, 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268627 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.861 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868574+0000", 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868574+0000", 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868575+0000", 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.58e-07 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.862 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.863 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868578+0000", 2024-03-21T14:20:59.863 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9009999999999998e-06 2024-03-21T14:20:59.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.863 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868585+0000", 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8929999999999999e-06 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868590+0000", 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7540000000000003e-06 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870331+0000", 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017408759999999999 2024-03-21T14:20:59.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870342+0000", 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.075e-05 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897810+0000", 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0274681649999999 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898921+0000", 2024-03-21T14:20:59.866 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011109620000000001 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268628 9.7 9:ea86f15e:::benchmark_data_smithi003_135814_object127894:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870340+0000", 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.92152055499997, 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.031124903, 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.867 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268628 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870340+0000", 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870340+0000", 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.868 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870341+0000", 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0270000000000001e-06 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870346+0000", 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7530000000000001e-06 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870357+0000", 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0767e-05 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.869 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870364+0000", 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7659999999999998e-06 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870662+0000", 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029719099999999998 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870669+0000", 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8609999999999998e-06 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.870 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898862+0000", 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0281925110000001 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for missing object", 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898997+0000", 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013489700000000001 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.901394+0000", 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023972780000000001 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.871 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.901465+0000", 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0852000000000005e-05 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268630 9.8 9:1462fa6d:::benchmark_data_smithi003_135814_object127896:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.92147979800001, 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.038780203, 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.872 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268630 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.873 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-07 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870385+0000", 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.292e-06 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870391+0000", 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5060000000000001e-06 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.874 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870406+0000", 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.518e-05 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870452+0000", 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5859999999999998e-05 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870461+0000", 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2709999999999998e-06 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.875 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.899383+0000", 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0289211490000001 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.899704+0000", 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032184800000000001 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.899902+0000", 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019798600000000001 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.908928+0000", 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090258820000000007 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.908987+0000", 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8446999999999998e-05 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909061+0000", 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4682000000000001e-05 2024-03-21T14:20:59.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909113+0000", 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1447999999999999e-05 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909161+0000", 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8149999999999998e-05 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.878 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268633 9.8 9:17488b50:::benchmark_data_smithi003_135814_object127899:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870422+0000", 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.92143852700002, 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0421929080000001, 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268633 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.879 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870422+0000", 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870422+0000", 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870423+0000", 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9100000000000004e-07 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.880 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870426+0000", 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.286e-06 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870433+0000", 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870439+0000", 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9359999999999996e-06 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.881 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870488+0000", 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9209999999999998e-05 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870496+0000", 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9629999999999998e-06 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.906627+0000", 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.036130526 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.882 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.906759+0000", 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131731 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.906887+0000", 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128763 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909014+0000", 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021263699999999998 2024-03-21T14:20:59.883 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.910484+0000", 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014707749999999999 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912494+0000", 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020092780000000002 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912557+0000", 2024-03-21T14:20:59.884 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3280000000000004e-05 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912615+0000", 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8069000000000003e-05 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268639 9.5 9:a8974fa7:::benchmark_data_smithi003_135814_object127905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.92137587100001, 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.027248454, 2024-03-21T14:20:59.885 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268639 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.886 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6199999999999998e-07 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870488+0000", 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7389999999999999e-06 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.887 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870494+0000", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1299999999999998e-06 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870501+0000", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.46e-06 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.871813+0000", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001312548 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.888 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.871823+0000", 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8940000000000005e-06 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.889526+0000", 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0177033390000001 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.889846+0000", 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031953599999999998 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:20:59.889 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.890019+0000", 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017282699999999999 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897136+0000", 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071172700000000002 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897191+0000", 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4558999999999997e-05 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.890 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897670+0000", 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047937000000000001 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897706+0000", 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5887999999999999e-05 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897733+0000", 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7432000000000001e-05 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.891 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268640 9.8 9:19908561:::benchmark_data_smithi003_135814_object127906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.873233+0000", 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.91862779299998, 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0394783320000001, 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268640 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.892 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873233+0000", 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873233+0000", 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873234+0000", 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2699999999999999e-06 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.893 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873239+0000", 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8899999999999998e-06 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873249+0000", 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5200000000000003e-06 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873256+0000", 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.644e-06 2024-03-21T14:20:59.894 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873307+0000", 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0921000000000003e-05 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873316+0000", 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2439999999999999e-06 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909178+0000", 2024-03-21T14:20:59.895 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0358614159999999 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909251+0000", 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3392999999999993e-05 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909320+0000", 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8437999999999998e-05 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.896 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.910646+0000", 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013268010000000001 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.911254+0000", 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060741600000000005 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912654+0000", 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001399874 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.897 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912686+0000", 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.273e-05 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912711+0000", 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4774999999999999e-05 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268643 9.5 9:aae2a19c:::benchmark_data_smithi003_135814_object127909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:20:59.898 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.880225+0000", 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 278.911635762, 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0218630360000001, 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268643 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880225+0000", 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880225+0000", 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880226+0000", 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0800000000000004e-07 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880230+0000", 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9530000000000001e-06 2024-03-21T14:20:59.900 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880236+0000", 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4189999999999999e-06 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880241+0000", 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.933e-06 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880293+0000", 2024-03-21T14:20:59.901 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2160000000000002e-05 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880307+0000", 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.343e-05 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.894036+0000", 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.013729887 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.902 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.894152+0000", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115622 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.894270+0000", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118305 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897147+0000", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002876768 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898277+0000", 2024-03-21T14:20:59.903 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011301 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.901985+0000", 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003708065 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.902042+0000", 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6913999999999998e-05 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.904 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.902088+0000", 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5772000000000002e-05 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:183429 10.c 10:36aa9b50:::benchmark_data_smithi003_160672_object30053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:18:49.902736+0000", 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 129.889124189, 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.11657621899999999, 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.905 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 183429 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.902736+0000", 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.906 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.902736+0000", 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.902738+0000", 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1060000000000001e-06 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.902743+0000", 2024-03-21T14:20:59.907 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0420000000000002e-06 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.902754+0000", 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0902e-05 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.902761+0000", 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8669999999999994e-06 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.908 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.904058+0000", 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001297004 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.904163+0000", 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104647 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,4", 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.904267+0000", 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000104064 2024-03-21T14:20:59.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.906602+0000", 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023349429999999999 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:49.907606+0000", 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001003473 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.910 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:50.019183+0000", 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.111577894 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:50.019270+0000", 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.6940000000000004e-05 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:18:50.019313+0000", 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.2336999999999998e-05 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.911 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:198695 10.c 10:3fe971f3:::benchmark_data_smithi003_160672_object45319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e352)", 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:01.133085+0000", 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 118.65877567699999, 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.115769997, 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 198695 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.912 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.133085+0000", 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.133085+0000", 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.133085+0000", 2024-03-21T14:20:59.913 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.51e-07 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.133088+0000", 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.0580000000000002e-06 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.133098+0000", 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.3000000000000007e-06 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.914 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.133103+0000", 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.5289999999999999e-06 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.135732+0000", 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0026282520000000002 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.135847+0000", 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011517499999999999 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,4", 2024-03-21T14:20:59.915 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.135962+0000", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011555700000000001 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.137965+0000", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020027780000000002 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.138594+0000", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00062912000000000005 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.248729+0000", 2024-03-21T14:20:59.916 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.110134933 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.248805+0000", 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.5838000000000001e-05 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:01.248855+0000", 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0006000000000002e-05 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.15666.0:249247 10.c 10:3bfb53c3:::benchmark_data_smithi003_160672_object95871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e369)", 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:19:39.310969+0000", 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 80.480891795999995, 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.14371515200000001, 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.15666", 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 249247 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.918 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.310969+0000", 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.310969+0000", 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.310970+0000", 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6199999999999999e-06 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.919 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.310978+0000", 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1860000000000003e-06 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.310994+0000", 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6056999999999999e-05 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.311004+0000", 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0032e-05 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.920 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.311070+0000", 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5968000000000003e-05 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.311195+0000", 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012526900000000001 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,4", 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.311313+0000", 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118025 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.921 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.313786+0000", 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0024729919999999998 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.314315+0000", 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052944500000000002 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.454510+0000", 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.14019432200000001 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.922 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.454635+0000", 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012472199999999999 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:19:39.454684+0000", 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9514000000000001e-05 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:20:59.923 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:20:59.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:20:59.947 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:20:59.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:00.048 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.073 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:00.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:00.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:00 smithi118 ceph-mon[33862]: pgmap v871: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:00 smithi118 ceph-mon[33862]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:21:00.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:00 smithi118 ceph-mon[33862]: osdmap e412: 8 total, 8 up, 7 in 2024-03-21T14:21:00.197 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:00.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:00.299 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:00.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:21:00.323 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:00.343 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:00 smithi003 ceph-mon[29576]: pgmap v871: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:21:00.343 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:00 smithi003 ceph-mon[29576]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:21:00.344 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:00 smithi003 ceph-mon[29576]: osdmap e412: 8 total, 8 up, 7 in 2024-03-21T14:21:00.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:00 smithi003 ceph-mon[40510]: pgmap v871: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:21:00.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:00 smithi003 ceph-mon[40510]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-21T14:21:00.344 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:00 smithi003 ceph-mon[40510]: osdmap e412: 8 total, 8 up, 7 in 2024-03-21T14:21:00.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:00.448 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:00.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:00.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:00.573 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:00.675 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:00.699 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:00.723 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:21:00.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:00.830 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:00.932 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:00.956 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:00.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:01.005 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:21:01.059 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.084 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:01.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:01.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:01.208 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:01.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:01.336 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:01.376 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:01 smithi003 ceph-mon[29576]: osdmap e413: 8 total, 8 up, 7 in 2024-03-21T14:21:01.376 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:01 smithi003 ceph-mon[40510]: osdmap e413: 8 total, 8 up, 7 in 2024-03-21T14:21:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:01 smithi118 ceph-mon[33862]: osdmap e413: 8 total, 8 up, 7 in 2024-03-21T14:21:01.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:01.462 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:01.564 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:01.587 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:01.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:01.713 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:01.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:01.838 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:01.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:01.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:01.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:02.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:02.072 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:02.073 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:02.089 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:02.190 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:02.214 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:02 smithi003 ceph-mon[29576]: pgmap v874: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:02 smithi003 ceph-mon[40510]: pgmap v874: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:02.309 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:21:02.312 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:21:02.316 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:02.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:02.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:02.414 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:02 smithi118 ceph-mon[33862]: pgmap v874: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:02.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:02.466 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:02.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:02.567 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:02.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:02.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:02.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:02.716 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:02.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:02.818 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:02.841 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:02.885 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:21:02.885 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:21:02.885 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 0 2024-03-21T14:21:02.886 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:21:02.943 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:02.967 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:02.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:02.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:03.069 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:03.093 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:03.194 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:03.219 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:03.316 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:21:03.320 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:03.348 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:03 smithi118 ceph-mon[33862]: pgmap v875: 158 pgs: 13 unknown, 145 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:03 smithi118 ceph-mon[33862]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:21:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:03 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:03.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:03 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/966161542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:21:03.450 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:03.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:03.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.602 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:03.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:03.627 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T14:21:03.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[29576]: pgmap v875: 158 pgs: 13 unknown, 145 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:03.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[29576]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:21:03.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/966161542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[40510]: pgmap v875: 158 pgs: 13 unknown, 145 active+clean; 582 KiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[40510]: from='client.25772 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:03.695 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:03 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/966161542' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:21:03.703 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:03.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:03.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:03.854 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:03.867 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:21:03.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:03.980 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:03.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:04.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:04.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:04.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:04.233 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:04.335 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.361 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:04.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:04.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:04.485 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:04.509 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":875,"stamp":"2024-03-21T14:21:02.144980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2168,"num_read_kb":1920,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":3868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2297,"ondisk_log_size":2297,"up":435,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":442,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6570848,"kb_used_data":1712896,"kb_used_omap":17,"kb_used_meta":4857838,"kb_avail":649645216,"statfs":{"total":671965249536,"available":665236701184,"internally_reserved":0,"allocated":1754005504,"data_stored":1746361212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17623,"internal_metadata":4974426921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001432"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":748,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098605+0000","last_change":"2024-03-21T14:18:38.441703+0000","last_active":"2024-03-21T14:21:00.098605+0000","last_peered":"2024-03-21T14:21:00.098605+0000","last_clean":"2024-03-21T14:21:00.098605+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:21:00.098605+0000","last_undegraded":"2024-03-21T14:21:00.098605+0000","last_fullsized":"2024-03-21T14:21:00.098605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:38.441623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:38.441623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":763,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815527+0000","last_change":"2024-03-21T14:19:09.223055+0000","last_active":"2024-03-21T14:20:53.815527+0000","last_peered":"2024-03-21T14:20:53.815527+0000","last_clean":"2024-03-21T14:20:53.815527+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:20:53.815527+0000","last_undegraded":"2024-03-21T14:20:53.815527+0000","last_fullsized":"2024-03-21T14:20:53.815527+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:09.222911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:19:09.222911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":318,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797278+0000","last_change":"2024-03-21T14:20:42.558129+0000","last_active":"2024-03-21T14:20:54.797278+0000","last_peered":"2024-03-21T14:20:54.797278+0000","last_clean":"2024-03-21T14:20:54.797278+0000","last_became_active":"2024-03-21T14:19:34.695760+0000","last_became_peered":"2024-03-21T14:19:34.695760+0000","last_unstale":"2024-03-21T14:20:54.797278+0000","last_undegraded":"2024-03-21T14:20:54.797278+0000","last_fullsized":"2024-03-21T14:20:54.797278+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.558048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.558048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:58.393037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":814,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098312+0000","last_change":"2024-03-21T14:20:41.426448+0000","last_active":"2024-03-21T14:21:00.098312+0000","last_peered":"2024-03-21T14:21:00.098312+0000","last_clean":"2024-03-21T14:21:00.098312+0000","last_became_active":"2024-03-21T14:20:21.782187+0000","last_became_peered":"2024-03-21T14:20:21.782187+0000","last_unstale":"2024-03-21T14:21:00.098312+0000","last_undegraded":"2024-03-21T14:21:00.098312+0000","last_fullsized":"2024-03-21T14:21:00.098312+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:20:41.426344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:41.426344+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.597745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":47,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797547+0000","last_change":"2024-03-21T14:20:21.782736+0000","last_active":"2024-03-21T14:20:54.797547+0000","last_peered":"2024-03-21T14:20:54.797547+0000","last_clean":"2024-03-21T14:20:54.797547+0000","last_became_active":"2024-03-21T14:20:21.782401+0000","last_became_peered":"2024-03-21T14:20:21.782401+0000","last_unstale":"2024-03-21T14:20:54.797547+0000","last_undegraded":"2024-03-21T14:20:54.797547+0000","last_fullsized":"2024-03-21T14:20:54.797547+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:15.807973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.807973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":767,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098199+0000","last_change":"2024-03-21T14:20:52.477097+0000","last_active":"2024-03-21T14:21:00.098199+0000","last_peered":"2024-03-21T14:21:00.098199+0000","last_clean":"2024-03-21T14:21:00.098199+0000","last_became_active":"2024-03-21T14:19:14.689459+0000","last_became_peered":"2024-03-21T14:19:14.689459+0000","last_unstale":"2024-03-21T14:21:00.098199+0000","last_undegraded":"2024-03-21T14:21:00.098199+0000","last_fullsized":"2024-03-21T14:21:00.098199+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:52.477009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.477009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:00.750211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":772,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798196+0000","last_change":"2024-03-21T14:20:37.939217+0000","last_active":"2024-03-21T14:20:54.798196+0000","last_peered":"2024-03-21T14:20:54.798196+0000","last_clean":"2024-03-21T14:20:54.798196+0000","last_became_active":"2024-03-21T14:19:14.695703+0000","last_became_peered":"2024-03-21T14:19:14.695703+0000","last_unstale":"2024-03-21T14:20:54.798196+0000","last_undegraded":"2024-03-21T14:20:54.798196+0000","last_fullsized":"2024-03-21T14:20:54.798196+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:37.939080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.939080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:47.422281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":810,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798421+0000","last_change":"2024-03-21T14:20:40.919011+0000","last_active":"2024-03-21T14:20:54.798421+0000","last_peered":"2024-03-21T14:20:54.798421+0000","last_clean":"2024-03-21T14:20:54.798421+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:20:54.798421+0000","last_undegraded":"2024-03-21T14:20:54.798421+0000","last_fullsized":"2024-03-21T14:20:54.798421+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:20:40.918895+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:20:40.918895+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:07.224584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":623,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.814833+0000","last_change":"2024-03-21T14:19:34.691274+0000","last_active":"2024-03-21T14:20:53.814833+0000","last_peered":"2024-03-21T14:20:53.814833+0000","last_clean":"2024-03-21T14:20:53.814833+0000","last_became_active":"2024-03-21T14:19:34.690300+0000","last_became_peered":"2024-03-21T14:19:34.690300+0000","last_unstale":"2024-03-21T14:20:53.814833+0000","last_undegraded":"2024-03-21T14:20:53.814833+0000","last_fullsized":"2024-03-21T14:20:53.814833+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:23.464551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:23.464551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.098108+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":768,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.783028+0000","last_change":"2024-03-21T14:20:29.536936+0000","last_active":"2024-03-21T14:21:00.783028+0000","last_peered":"2024-03-21T14:21:00.783028+0000","last_clean":"2024-03-21T14:21:00.783028+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:21:00.783028+0000","last_undegraded":"2024-03-21T14:21:00.783028+0000","last_fullsized":"2024-03-21T14:21:00.783028+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:29.536820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:20:29.536820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:29.770498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":568,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099059+0000","last_change":"2024-03-21T14:17:49.153687+0000","last_active":"2024-03-21T14:21:00.099059+0000","last_peered":"2024-03-21T14:21:00.099059+0000","last_clean":"2024-03-21T14:21:00.099059+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:21:00.099059+0000","last_undegraded":"2024-03-21T14:21:00.099059+0000","last_fullsized":"2024-03-21T14:21:00.099059+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:49.153572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:17:49.153572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370508+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":773,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098825+0000","last_change":"2024-03-21T14:17:21.179086+0000","last_active":"2024-03-21T14:21:00.098825+0000","last_peered":"2024-03-21T14:21:00.098825+0000","last_clean":"2024-03-21T14:21:00.098825+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:21:00.098825+0000","last_undegraded":"2024-03-21T14:21:00.098825+0000","last_fullsized":"2024-03-21T14:21:00.098825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:21.178967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:21.178967+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370459+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":712,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815691+0000","last_change":"2024-03-21T14:20:21.776714+0000","last_active":"2024-03-21T14:20:53.815691+0000","last_peered":"2024-03-21T14:20:53.815691+0000","last_clean":"2024-03-21T14:20:53.815691+0000","last_became_active":"2024-03-21T14:20:21.776360+0000","last_became_peered":"2024-03-21T14:20:21.776360+0000","last_unstale":"2024-03-21T14:20:53.815691+0000","last_undegraded":"2024-03-21T14:20:53.815691+0000","last_fullsized":"2024-03-21T14:20:53.815691+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:18.436540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:18.436540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.098069+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":769,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398503+0000","last_change":"2024-03-21T14:19:45.482626+0000","last_active":"2024-03-21T14:21:00.398503+0000","last_peered":"2024-03-21T14:21:00.398503+0000","last_clean":"2024-03-21T14:21:00.398503+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:21:00.398503+0000","last_undegraded":"2024-03-21T14:21:00.398503+0000","last_fullsized":"2024-03-21T14:21:00.398503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:45.482513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:19:45.482513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:10.547844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":234,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098571+0000","last_change":"2024-03-21T14:17:09.374978+0000","last_active":"2024-03-21T14:21:00.098571+0000","last_peered":"2024-03-21T14:21:00.098571+0000","last_clean":"2024-03-21T14:21:00.098571+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:21:00.098571+0000","last_undegraded":"2024-03-21T14:21:00.098571+0000","last_fullsized":"2024-03-21T14:21:00.098571+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:09.374855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:09.374855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370329+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":769,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098665+0000","last_change":"2024-03-21T14:18:04.289351+0000","last_active":"2024-03-21T14:21:00.098665+0000","last_peered":"2024-03-21T14:21:00.098665+0000","last_clean":"2024-03-21T14:21:00.098665+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:21:00.098665+0000","last_undegraded":"2024-03-21T14:21:00.098665+0000","last_fullsized":"2024-03-21T14:21:00.098665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:18:04.289228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:04.289228+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370524+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":748,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099517+0000","last_change":"2024-03-21T14:17:06.425215+0000","last_active":"2024-03-21T14:21:00.099517+0000","last_peered":"2024-03-21T14:21:00.099517+0000","last_clean":"2024-03-21T14:21:00.099517+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:21:00.099517+0000","last_undegraded":"2024-03-21T14:21:00.099517+0000","last_fullsized":"2024-03-21T14:21:00.099517+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:06.425104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:06.425104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370346+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":755,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817364+0000","last_change":"2024-03-21T14:18:34.187073+0000","last_active":"2024-03-21T14:20:53.817364+0000","last_peered":"2024-03-21T14:20:53.817364+0000","last_clean":"2024-03-21T14:20:53.817364+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:20:53.817364+0000","last_undegraded":"2024-03-21T14:20:53.817364+0000","last_fullsized":"2024-03-21T14:20:53.817364+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:34.186897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:34.186897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097792+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":619,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098615+0000","last_change":"2024-03-21T14:19:34.686397+0000","last_active":"2024-03-21T14:21:00.098615+0000","last_peered":"2024-03-21T14:21:00.098615+0000","last_clean":"2024-03-21T14:21:00.098615+0000","last_became_active":"2024-03-21T14:19:34.686106+0000","last_became_peered":"2024-03-21T14:19:34.686106+0000","last_unstale":"2024-03-21T14:21:00.098615+0000","last_undegraded":"2024-03-21T14:21:00.098615+0000","last_fullsized":"2024-03-21T14:21:00.098615+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:33.422232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:18:33.422232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370604+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'27","reported_seq":661,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798205+0000","last_change":"2024-03-21T14:19:14.701974+0000","last_active":"2024-03-21T14:20:54.798205+0000","last_peered":"2024-03-21T14:20:54.798205+0000","last_clean":"2024-03-21T14:20:54.798205+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:20:54.798205+0000","last_undegraded":"2024-03-21T14:20:54.798205+0000","last_fullsized":"2024-03-21T14:20:54.798205+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:14:48.508532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:48.508532+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569063+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798515+0000","last_change":"2024-03-21T14:17:05.017847+0000","last_active":"2024-03-21T14:20:54.798515+0000","last_peered":"2024-03-21T14:20:54.798515+0000","last_clean":"2024-03-21T14:20:54.798515+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:20:54.798515+0000","last_undegraded":"2024-03-21T14:20:54.798515+0000","last_fullsized":"2024-03-21T14:20:54.798515+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.017731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.017731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":745,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098830+0000","last_change":"2024-03-21T14:19:14.691492+0000","last_active":"2024-03-21T14:21:00.098830+0000","last_peered":"2024-03-21T14:21:00.098830+0000","last_clean":"2024-03-21T14:21:00.098830+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:21:00.098830+0000","last_undegraded":"2024-03-21T14:21:00.098830+0000","last_fullsized":"2024-03-21T14:21:00.098830+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.679529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.679529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370225+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":761,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098934+0000","last_change":"2024-03-21T14:20:31.407178+0000","last_active":"2024-03-21T14:21:00.098934+0000","last_peered":"2024-03-21T14:21:00.098934+0000","last_clean":"2024-03-21T14:21:00.098934+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:21:00.098934+0000","last_undegraded":"2024-03-21T14:21:00.098934+0000","last_fullsized":"2024-03-21T14:21:00.098934+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.407065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.407065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:43.751635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":791,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798311+0000","last_change":"2024-03-21T14:18:03.889554+0000","last_active":"2024-03-21T14:20:54.798311+0000","last_peered":"2024-03-21T14:20:54.798311+0000","last_clean":"2024-03-21T14:20:54.798311+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:20:54.798311+0000","last_undegraded":"2024-03-21T14:20:54.798311+0000","last_fullsized":"2024-03-21T14:20:54.798311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:18:03.889428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:03.889428+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":262,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797935+0000","last_change":"2024-03-21T14:19:14.687146+0000","last_active":"2024-03-21T14:20:54.797935+0000","last_peered":"2024-03-21T14:20:54.797935+0000","last_clean":"2024-03-21T14:20:54.797935+0000","last_became_active":"2024-03-21T14:19:14.686523+0000","last_became_peered":"2024-03-21T14:19:14.686523+0000","last_unstale":"2024-03-21T14:20:54.797935+0000","last_undegraded":"2024-03-21T14:20:54.797935+0000","last_fullsized":"2024-03-21T14:20:54.797935+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.547775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.568976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"360'14","reported_seq":366,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099587+0000","last_change":"2024-03-21T14:19:15.820967+0000","last_active":"2024-03-21T14:21:00.099587+0000","last_peered":"2024-03-21T14:21:00.099587+0000","last_clean":"2024-03-21T14:21:00.099587+0000","last_became_active":"2024-03-21T14:19:15.688331+0000","last_became_peered":"2024-03-21T14:19:15.688331+0000","last_unstale":"2024-03-21T14:21:00.099587+0000","last_undegraded":"2024-03-21T14:21:00.099587+0000","last_fullsized":"2024-03-21T14:21:00.099587+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'10","last_scrub_stamp":"2024-03-21T14:16:59.343213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:59.343213+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370363+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":780,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815427+0000","last_change":"2024-03-21T14:20:37.151817+0000","last_active":"2024-03-21T14:20:53.815427+0000","last_peered":"2024-03-21T14:20:53.815427+0000","last_clean":"2024-03-21T14:20:53.815427+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:20:53.815427+0000","last_undegraded":"2024-03-21T14:20:53.815427+0000","last_fullsized":"2024-03-21T14:20:53.815427+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:20:37.151721+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.151721+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:49.468465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":483,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399311+0000","last_change":"2024-03-21T14:20:21.799582+0000","last_active":"2024-03-21T14:21:00.399311+0000","last_peered":"2024-03-21T14:21:00.399311+0000","last_clean":"2024-03-21T14:21:00.399311+0000","last_became_active":"2024-03-21T14:20:21.790051+0000","last_became_peered":"2024-03-21T14:20:21.790051+0000","last_unstale":"2024-03-21T14:21:00.399311+0000","last_undegraded":"2024-03-21T14:21:00.399311+0000","last_fullsized":"2024-03-21T14:21:00.399311+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:19:21.606220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:21.606220+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":751,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817014+0000","last_change":"2024-03-21T14:17:58.261975+0000","last_active":"2024-03-21T14:20:53.817014+0000","last_peered":"2024-03-21T14:20:53.817014+0000","last_clean":"2024-03-21T14:20:53.817014+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:20:53.817014+0000","last_undegraded":"2024-03-21T14:20:53.817014+0000","last_fullsized":"2024-03-21T14:20:53.817014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:58.261849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:58.261849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097392+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":425,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399262+0000","last_change":"2024-03-21T14:20:49.417668+0000","last_active":"2024-03-21T14:21:00.399262+0000","last_peered":"2024-03-21T14:21:00.399262+0000","last_clean":"2024-03-21T14:21:00.399262+0000","last_became_active":"2024-03-21T14:19:14.675463+0000","last_became_peered":"2024-03-21T14:19:14.675463+0000","last_unstale":"2024-03-21T14:21:00.399262+0000","last_undegraded":"2024-03-21T14:21:00.399262+0000","last_fullsized":"2024-03-21T14:21:00.399262+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:49.417593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.417593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:15.841246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":784,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797503+0000","last_change":"2024-03-21T14:20:45.936416+0000","last_active":"2024-03-21T14:20:54.797503+0000","last_peered":"2024-03-21T14:20:54.797503+0000","last_clean":"2024-03-21T14:20:54.797503+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:20:54.797503+0000","last_undegraded":"2024-03-21T14:20:54.797503+0000","last_fullsized":"2024-03-21T14:20:54.797503+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.936318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.936318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:14.932353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":495,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817561+0000","last_change":"2024-03-21T14:20:46.036522+0000","last_active":"2024-03-21T14:20:53.817561+0000","last_peered":"2024-03-21T14:20:53.817561+0000","last_clean":"2024-03-21T14:20:53.817561+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:20:53.817561+0000","last_undegraded":"2024-03-21T14:20:53.817561+0000","last_fullsized":"2024-03-21T14:20:53.817561+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.036377+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:08.476100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":117,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798650+0000","last_change":"2024-03-21T14:19:14.680267+0000","last_active":"2024-03-21T14:20:54.798650+0000","last_peered":"2024-03-21T14:20:54.798650+0000","last_clean":"2024-03-21T14:20:54.798650+0000","last_became_active":"2024-03-21T14:19:14.679790+0000","last_became_peered":"2024-03-21T14:19:14.679790+0000","last_unstale":"2024-03-21T14:20:54.798650+0000","last_undegraded":"2024-03-21T14:20:54.798650+0000","last_fullsized":"2024-03-21T14:20:54.798650+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:49.859204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:49.859204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569111+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":442,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098196+0000","last_change":"2024-03-21T14:18:42.513377+0000","last_active":"2024-03-21T14:21:00.098196+0000","last_peered":"2024-03-21T14:21:00.098196+0000","last_clean":"2024-03-21T14:21:00.098196+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:21:00.098196+0000","last_undegraded":"2024-03-21T14:21:00.098196+0000","last_fullsized":"2024-03-21T14:21:00.098196+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.513167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370635+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":230,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797056+0000","last_change":"2024-03-21T14:16:56.847010+0000","last_active":"2024-03-21T14:20:54.797056+0000","last_peered":"2024-03-21T14:20:54.797056+0000","last_clean":"2024-03-21T14:20:54.797056+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:20:54.797056+0000","last_undegraded":"2024-03-21T14:20:54.797056+0000","last_fullsized":"2024-03-21T14:20:54.797056+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:56.846916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:56.846916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"275'10","reported_seq":793,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797599+0000","last_change":"2024-03-21T14:18:45.023742+0000","last_active":"2024-03-21T14:20:54.797599+0000","last_peered":"2024-03-21T14:20:54.797599+0000","last_clean":"2024-03-21T14:20:54.797599+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:20:54.797599+0000","last_undegraded":"2024-03-21T14:20:54.797599+0000","last_fullsized":"2024-03-21T14:20:54.797599+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'10","last_scrub_stamp":"2024-03-21T14:18:45.023596+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:18:45.023596+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":775,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798130+0000","last_change":"2024-03-21T14:19:34.693613+0000","last_active":"2024-03-21T14:20:54.798130+0000","last_peered":"2024-03-21T14:20:54.798130+0000","last_clean":"2024-03-21T14:20:54.798130+0000","last_became_active":"2024-03-21T14:19:34.693279+0000","last_became_peered":"2024-03-21T14:19:34.693279+0000","last_unstale":"2024-03-21T14:20:54.798130+0000","last_undegraded":"2024-03-21T14:20:54.798130+0000","last_fullsized":"2024-03-21T14:20:54.798130+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:07.939188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:17:07.939188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":791,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398631+0000","last_change":"2024-03-21T14:20:30.520958+0000","last_active":"2024-03-21T14:21:00.398631+0000","last_peered":"2024-03-21T14:21:00.398631+0000","last_clean":"2024-03-21T14:21:00.398631+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:21:00.398631+0000","last_undegraded":"2024-03-21T14:21:00.398631+0000","last_fullsized":"2024-03-21T14:21:00.398631+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:30.520827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.520827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:42.586304+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":233,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817206+0000","last_change":"2024-03-21T14:19:42.258049+0000","last_active":"2024-03-21T14:20:53.817206+0000","last_peered":"2024-03-21T14:20:53.817206+0000","last_clean":"2024-03-21T14:20:53.817206+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:20:53.817206+0000","last_undegraded":"2024-03-21T14:20:53.817206+0000","last_fullsized":"2024-03-21T14:20:53.817206+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:19:42.257908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:19:42.257908+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:09.908704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":821,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798386+0000","last_change":"2024-03-21T14:18:42.061939+0000","last_active":"2024-03-21T14:20:54.798386+0000","last_peered":"2024-03-21T14:20:54.798386+0000","last_clean":"2024-03-21T14:20:54.798386+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:20:54.798386+0000","last_undegraded":"2024-03-21T14:20:54.798386+0000","last_fullsized":"2024-03-21T14:20:54.798386+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:18:42.061791+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.061791+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":239,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.814662+0000","last_change":"2024-03-21T14:19:50.405106+0000","last_active":"2024-03-21T14:20:53.814662+0000","last_peered":"2024-03-21T14:20:53.814662+0000","last_clean":"2024-03-21T14:20:53.814662+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:20:53.814662+0000","last_undegraded":"2024-03-21T14:20:53.814662+0000","last_fullsized":"2024-03-21T14:20:53.814662+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:50.404999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:50.404999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.098184+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":755,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797969+0000","last_change":"2024-03-21T14:17:19.983621+0000","last_active":"2024-03-21T14:20:54.797969+0000","last_peered":"2024-03-21T14:20:54.797969+0000","last_clean":"2024-03-21T14:20:54.797969+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:20:54.797969+0000","last_undegraded":"2024-03-21T14:20:54.797969+0000","last_fullsized":"2024-03-21T14:20:54.797969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:19.983497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:17:19.983497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":785,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398916+0000","last_change":"2024-03-21T14:20:45.456427+0000","last_active":"2024-03-21T14:21:00.398916+0000","last_peered":"2024-03-21T14:21:00.398916+0000","last_clean":"2024-03-21T14:21:00.398916+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:21:00.398916+0000","last_undegraded":"2024-03-21T14:21:00.398916+0000","last_fullsized":"2024-03-21T14:21:00.398916+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.456309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.456309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.470305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":809,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.816517+0000","last_change":"2024-03-21T14:19:25.467111+0000","last_active":"2024-03-21T14:20:53.816517+0000","last_peered":"2024-03-21T14:20:53.816517+0000","last_clean":"2024-03-21T14:20:53.816517+0000","last_became_active":"2024-03-21T14:19:14.682980+0000","last_became_peered":"2024-03-21T14:19:14.682980+0000","last_unstale":"2024-03-21T14:20:53.816517+0000","last_undegraded":"2024-03-21T14:20:53.816517+0000","last_fullsized":"2024-03-21T14:20:53.816517+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:19:25.465595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:25.465595+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.098147+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":55,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797819+0000","last_change":"2024-03-21T14:20:42.950594+0000","last_active":"2024-03-21T14:20:54.797819+0000","last_peered":"2024-03-21T14:20:54.797819+0000","last_clean":"2024-03-21T14:20:54.797819+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:20:54.797819+0000","last_undegraded":"2024-03-21T14:20:54.797819+0000","last_fullsized":"2024-03-21T14:20:54.797819+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.950500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.950500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.148493+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":775,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.816426+0000","last_change":"2024-03-21T14:20:21.778599+0000","last_active":"2024-03-21T14:20:53.816426+0000","last_peered":"2024-03-21T14:20:53.816426+0000","last_clean":"2024-03-21T14:20:53.816426+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:20:53.816426+0000","last_undegraded":"2024-03-21T14:20:53.816426+0000","last_fullsized":"2024-03-21T14:20:53.816426+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:57.200762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:57.200762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097473+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":775,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098681+0000","last_change":"2024-03-21T14:19:14.691071+0000","last_active":"2024-03-21T14:21:00.098681+0000","last_peered":"2024-03-21T14:21:00.098681+0000","last_clean":"2024-03-21T14:21:00.098681+0000","last_became_active":"2024-03-21T14:19:14.690097+0000","last_became_peered":"2024-03-21T14:19:14.690097+0000","last_unstale":"2024-03-21T14:21:00.098681+0000","last_undegraded":"2024-03-21T14:21:00.098681+0000","last_fullsized":"2024-03-21T14:21:00.098681+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:47.020235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:47.020235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":836,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098711+0000","last_change":"2024-03-21T14:19:15.685722+0000","last_active":"2024-03-21T14:21:00.098711+0000","last_peered":"2024-03-21T14:21:00.098711+0000","last_clean":"2024-03-21T14:21:00.098711+0000","last_became_active":"2024-03-21T14:19:15.685220+0000","last_became_peered":"2024-03-21T14:19:15.685220+0000","last_unstale":"2024-03-21T14:21:00.098711+0000","last_undegraded":"2024-03-21T14:21:00.098711+0000","last_fullsized":"2024-03-21T14:21:00.098711+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:16:13.416403+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.416403+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":642,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399145+0000","last_change":"2024-03-21T14:20:21.784715+0000","last_active":"2024-03-21T14:21:00.399145+0000","last_peered":"2024-03-21T14:21:00.399145+0000","last_clean":"2024-03-21T14:21:00.399145+0000","last_became_active":"2024-03-21T14:20:21.784416+0000","last_became_peered":"2024-03-21T14:20:21.784416+0000","last_unstale":"2024-03-21T14:21:00.399145+0000","last_undegraded":"2024-03-21T14:21:00.399145+0000","last_fullsized":"2024-03-21T14:21:00.399145+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:49.467034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:49.467034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:06.681753+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":957,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817175+0000","last_change":"2024-03-21T14:18:37.151442+0000","last_active":"2024-03-21T14:20:53.817175+0000","last_peered":"2024-03-21T14:20:53.817175+0000","last_clean":"2024-03-21T14:20:53.817175+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:20:53.817175+0000","last_undegraded":"2024-03-21T14:20:53.817175+0000","last_fullsized":"2024-03-21T14:20:53.817175+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'10","last_scrub_stamp":"2024-03-21T14:18:37.151302+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:18:37.151302+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097678+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":779,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398746+0000","last_change":"2024-03-21T14:19:22.612161+0000","last_active":"2024-03-21T14:21:00.398746+0000","last_peered":"2024-03-21T14:21:00.398746+0000","last_clean":"2024-03-21T14:21:00.398746+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:21:00.398746+0000","last_undegraded":"2024-03-21T14:21:00.398746+0000","last_fullsized":"2024-03-21T14:21:00.398746+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:22.612041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:19:22.612041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":786,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797838+0000","last_change":"2024-03-21T14:16:12.543045+0000","last_active":"2024-03-21T14:20:54.797838+0000","last_peered":"2024-03-21T14:20:54.797838+0000","last_clean":"2024-03-21T14:20:54.797838+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:20:54.797838+0000","last_undegraded":"2024-03-21T14:20:54.797838+0000","last_fullsized":"2024-03-21T14:20:54.797838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:16:12.542909+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:16:12.542909+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":749,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099529+0000","last_change":"2024-03-21T14:20:38.602527+0000","last_active":"2024-03-21T14:21:00.099529+0000","last_peered":"2024-03-21T14:21:00.099529+0000","last_clean":"2024-03-21T14:21:00.099529+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:21:00.099529+0000","last_undegraded":"2024-03-21T14:21:00.099529+0000","last_fullsized":"2024-03-21T14:21:00.099529+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:38.602405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:20:38.602405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:02.464900+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":66,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798172+0000","last_change":"2024-03-21T14:20:44.917571+0000","last_active":"2024-03-21T14:20:54.798172+0000","last_peered":"2024-03-21T14:20:54.798172+0000","last_clean":"2024-03-21T14:20:54.798172+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:20:54.798172+0000","last_undegraded":"2024-03-21T14:20:54.798172+0000","last_fullsized":"2024-03-21T14:20:54.798172+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:20:44.917480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.917480+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:55.408449+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":47,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797711+0000","last_change":"2024-03-21T14:20:21.784789+0000","last_active":"2024-03-21T14:20:54.797711+0000","last_peered":"2024-03-21T14:20:54.797711+0000","last_clean":"2024-03-21T14:20:54.797711+0000","last_became_active":"2024-03-21T14:20:21.784349+0000","last_became_peered":"2024-03-21T14:20:21.784349+0000","last_unstale":"2024-03-21T14:20:54.797711+0000","last_undegraded":"2024-03-21T14:20:54.797711+0000","last_fullsized":"2024-03-21T14:20:54.797711+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:40.140340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:18:40.140340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"338'19","reported_seq":653,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399195+0000","last_change":"2024-03-21T14:19:43.545025+0000","last_active":"2024-03-21T14:21:00.399195+0000","last_peered":"2024-03-21T14:21:00.399195+0000","last_clean":"2024-03-21T14:21:00.399195+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:21:00.399195+0000","last_undegraded":"2024-03-21T14:21:00.399195+0000","last_fullsized":"2024-03-21T14:21:00.399195+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'19","last_scrub_stamp":"2024-03-21T14:19:43.544784+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:19:43.544784+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":592,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798459+0000","last_change":"2024-03-21T14:19:14.699974+0000","last_active":"2024-03-21T14:20:54.798459+0000","last_peered":"2024-03-21T14:20:54.798459+0000","last_clean":"2024-03-21T14:20:54.798459+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:20:54.798459+0000","last_undegraded":"2024-03-21T14:20:54.798459+0000","last_fullsized":"2024-03-21T14:20:54.798459+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:43.502435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:43.502435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798704+0000","last_change":"2024-03-21T14:20:21.787792+0000","last_active":"2024-03-21T14:20:54.798704+0000","last_peered":"2024-03-21T14:20:54.798704+0000","last_clean":"2024-03-21T14:20:54.798704+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:20:54.798704+0000","last_undegraded":"2024-03-21T14:20:54.798704+0000","last_fullsized":"2024-03-21T14:20:54.798704+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.399758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:08.885507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":747,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098544+0000","last_change":"2024-03-21T14:17:05.374072+0000","last_active":"2024-03-21T14:21:00.098544+0000","last_peered":"2024-03-21T14:21:00.098544+0000","last_clean":"2024-03-21T14:21:00.098544+0000","last_became_active":"2024-03-21T14:16:34.616172+0000","last_became_peered":"2024-03-21T14:16:34.616172+0000","last_unstale":"2024-03-21T14:21:00.098544+0000","last_undegraded":"2024-03-21T14:21:00.098544+0000","last_fullsized":"2024-03-21T14:21:00.098544+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.373895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.373895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370428+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":452,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798402+0000","last_change":"2024-03-21T14:17:16.885851+0000","last_active":"2024-03-21T14:20:54.798402+0000","last_peered":"2024-03-21T14:20:54.798402+0000","last_clean":"2024-03-21T14:20:54.798402+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:20:54.798402+0000","last_undegraded":"2024-03-21T14:20:54.798402+0000","last_fullsized":"2024-03-21T14:20:54.798402+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:16.885709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.885709+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":779,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398902+0000","last_change":"2024-03-21T14:20:34.570950+0000","last_active":"2024-03-21T14:21:00.398902+0000","last_peered":"2024-03-21T14:21:00.398902+0000","last_clean":"2024-03-21T14:21:00.398902+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:21:00.398902+0000","last_undegraded":"2024-03-21T14:21:00.398902+0000","last_fullsized":"2024-03-21T14:21:00.398902+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:34.570712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:34.570712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:43.732723+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":229,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.796996+0000","last_change":"2024-03-21T14:17:03.781272+0000","last_active":"2024-03-21T14:20:54.796996+0000","last_peered":"2024-03-21T14:20:54.796996+0000","last_clean":"2024-03-21T14:20:54.796996+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:20:54.796996+0000","last_undegraded":"2024-03-21T14:20:54.796996+0000","last_fullsized":"2024-03-21T14:20:54.796996+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:03.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":413,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797348+0000","last_change":"2024-03-21T14:19:34.691021+0000","last_active":"2024-03-21T14:20:54.797348+0000","last_peered":"2024-03-21T14:20:54.797348+0000","last_clean":"2024-03-21T14:20:54.797348+0000","last_became_active":"2024-03-21T14:19:34.690247+0000","last_became_peered":"2024-03-21T14:19:34.690247+0000","last_unstale":"2024-03-21T14:20:54.797348+0000","last_undegraded":"2024-03-21T14:20:54.797348+0000","last_fullsized":"2024-03-21T14:20:54.797348+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:07.829823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:19:07.829823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":55,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798525+0000","last_change":"2024-03-21T14:20:21.796765+0000","last_active":"2024-03-21T14:20:54.798525+0000","last_peered":"2024-03-21T14:20:54.798525+0000","last_clean":"2024-03-21T14:20:54.798525+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:20:54.798525+0000","last_undegraded":"2024-03-21T14:20:54.798525+0000","last_fullsized":"2024-03-21T14:20:54.798525+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:19:32.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:32.302574+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:59.964639+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":637,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815460+0000","last_change":"2024-03-21T14:19:11.368761+0000","last_active":"2024-03-21T14:20:53.815460+0000","last_peered":"2024-03-21T14:20:53.815460+0000","last_clean":"2024-03-21T14:20:53.815460+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:20:53.815460+0000","last_undegraded":"2024-03-21T14:20:53.815460+0000","last_fullsized":"2024-03-21T14:20:53.815460+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:11.368572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:19:11.368572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097990+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":816,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399219+0000","last_change":"2024-03-21T14:19:42.441128+0000","last_active":"2024-03-21T14:21:00.399219+0000","last_peered":"2024-03-21T14:21:00.399219+0000","last_clean":"2024-03-21T14:21:00.399219+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:21:00.399219+0000","last_undegraded":"2024-03-21T14:21:00.399219+0000","last_fullsized":"2024-03-21T14:21:00.399219+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_clean_scrub_stamp":"2024-03-21T14:19:42.440997+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":698,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398502+0000","last_change":"2024-03-21T14:20:39.417922+0000","last_active":"2024-03-21T14:21:00.398502+0000","last_peered":"2024-03-21T14:21:00.398502+0000","last_clean":"2024-03-21T14:21:00.398502+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:21:00.398502+0000","last_undegraded":"2024-03-21T14:21:00.398502+0000","last_fullsized":"2024-03-21T14:21:00.398502+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:20:39.417781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:39.417781+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:43.468791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"372'10","reported_seq":1011,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398711+0000","last_change":"2024-03-21T14:19:47.479886+0000","last_active":"2024-03-21T14:21:00.398711+0000","last_peered":"2024-03-21T14:21:00.398711+0000","last_clean":"2024-03-21T14:21:00.398711+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:21:00.398711+0000","last_undegraded":"2024-03-21T14:21:00.398711+0000","last_fullsized":"2024-03-21T14:21:00.398711+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.479770+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":497,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797696+0000","last_change":"2024-03-21T14:18:26.997703+0000","last_active":"2024-03-21T14:20:54.797696+0000","last_peered":"2024-03-21T14:20:54.797696+0000","last_clean":"2024-03-21T14:20:54.797696+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:20:54.797696+0000","last_undegraded":"2024-03-21T14:20:54.797696+0000","last_fullsized":"2024-03-21T14:20:54.797696+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:26.997615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:26.997615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":794,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399242+0000","last_change":"2024-03-21T14:20:31.503763+0000","last_active":"2024-03-21T14:21:00.399242+0000","last_peered":"2024-03-21T14:21:00.399242+0000","last_clean":"2024-03-21T14:21:00.399242+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:21:00.399242+0000","last_undegraded":"2024-03-21T14:21:00.399242+0000","last_fullsized":"2024-03-21T14:21:00.399242+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.503687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.503687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:35.612376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":843,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098630+0000","last_change":"2024-03-21T14:20:21.797847+0000","last_active":"2024-03-21T14:21:00.098630+0000","last_peered":"2024-03-21T14:21:00.098630+0000","last_clean":"2024-03-21T14:21:00.098630+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:21:00.098630+0000","last_undegraded":"2024-03-21T14:21:00.098630+0000","last_fullsized":"2024-03-21T14:21:00.098630+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:18:29.432719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.432719+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370590+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":604,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797276+0000","last_change":"2024-03-21T14:19:14.699748+0000","last_active":"2024-03-21T14:20:54.797276+0000","last_peered":"2024-03-21T14:20:54.797276+0000","last_clean":"2024-03-21T14:20:54.797276+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:20:54.797276+0000","last_undegraded":"2024-03-21T14:20:54.797276+0000","last_fullsized":"2024-03-21T14:20:54.797276+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:39.425147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:14:39.425147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":125,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797397+0000","last_change":"2024-03-21T14:20:44.543695+0000","last_active":"2024-03-21T14:20:54.797397+0000","last_peered":"2024-03-21T14:20:54.797397+0000","last_clean":"2024-03-21T14:20:54.797397+0000","last_became_active":"2024-03-21T14:19:14.700785+0000","last_became_peered":"2024-03-21T14:19:14.700785+0000","last_unstale":"2024-03-21T14:20:54.797397+0000","last_undegraded":"2024-03-21T14:20:54.797397+0000","last_fullsized":"2024-03-21T14:20:54.797397+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:44.543611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.543611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:50.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":565,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798800+0000","last_change":"2024-03-21T14:20:51.545141+0000","last_active":"2024-03-21T14:20:54.798800+0000","last_peered":"2024-03-21T14:20:54.798800+0000","last_clean":"2024-03-21T14:20:54.798800+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:20:54.798800+0000","last_undegraded":"2024-03-21T14:20:54.798800+0000","last_fullsized":"2024-03-21T14:20:54.798800+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:51.545057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.545057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:54.778605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":741,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798500+0000","last_change":"2024-03-21T14:17:54.944166+0000","last_active":"2024-03-21T14:20:54.798500+0000","last_peered":"2024-03-21T14:20:54.798500+0000","last_clean":"2024-03-21T14:20:54.798500+0000","last_became_active":"2024-03-21T14:16:34.618652+0000","last_became_peered":"2024-03-21T14:16:34.618652+0000","last_unstale":"2024-03-21T14:20:54.798500+0000","last_undegraded":"2024-03-21T14:20:54.798500+0000","last_fullsized":"2024-03-21T14:20:54.798500+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:54.944088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:54.944088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":721,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.816258+0000","last_change":"2024-03-21T14:20:49.145974+0000","last_active":"2024-03-21T14:20:53.816258+0000","last_peered":"2024-03-21T14:20:53.816258+0000","last_clean":"2024-03-21T14:20:53.816258+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:20:53.816258+0000","last_undegraded":"2024-03-21T14:20:53.816258+0000","last_fullsized":"2024-03-21T14:20:53.816258+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.145792+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:53.951598+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":13,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398969+0000","last_change":"2024-03-21T14:20:59.109491+0000","last_active":"2024-03-21T14:21:00.398969+0000","last_peered":"2024-03-21T14:21:00.398969+0000","last_clean":"2024-03-21T14:21:00.398969+0000","last_became_active":"2024-03-21T14:20:59.109188+0000","last_became_peered":"2024-03-21T14:20:59.109188+0000","last_unstale":"2024-03-21T14:21:00.398969+0000","last_undegraded":"2024-03-21T14:21:00.398969+0000","last_fullsized":"2024-03-21T14:21:00.398969+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:20.048059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":124,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.796789+0000","last_change":"2024-03-21T14:19:34.691261+0000","last_active":"2024-03-21T14:20:54.796789+0000","last_peered":"2024-03-21T14:20:54.796789+0000","last_clean":"2024-03-21T14:20:54.796789+0000","last_became_active":"2024-03-21T14:19:34.690841+0000","last_became_peered":"2024-03-21T14:19:34.690841+0000","last_unstale":"2024-03-21T14:20:54.796789+0000","last_undegraded":"2024-03-21T14:20:54.796789+0000","last_fullsized":"2024-03-21T14:20:54.796789+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.865488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.865488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569096+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":604,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798799+0000","last_change":"2024-03-21T14:19:14.692130+0000","last_active":"2024-03-21T14:20:54.798799+0000","last_peered":"2024-03-21T14:20:54.798799+0000","last_clean":"2024-03-21T14:20:54.798799+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:20:54.798799+0000","last_undegraded":"2024-03-21T14:20:54.798799+0000","last_fullsized":"2024-03-21T14:20:54.798799+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.532653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.532653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":564,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.796868+0000","last_change":"2024-03-21T14:18:40.092328+0000","last_active":"2024-03-21T14:20:54.796868+0000","last_peered":"2024-03-21T14:20:54.796868+0000","last_clean":"2024-03-21T14:20:54.796868+0000","last_became_active":"2024-03-21T14:16:21.865349+0000","last_became_peered":"2024-03-21T14:16:21.865349+0000","last_unstale":"2024-03-21T14:20:54.796868+0000","last_undegraded":"2024-03-21T14:20:54.796868+0000","last_fullsized":"2024-03-21T14:20:54.796868+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:40.092182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:18:40.092182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":259,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.799027+0000","last_change":"2024-03-21T14:19:14.692833+0000","last_active":"2024-03-21T14:20:54.799027+0000","last_peered":"2024-03-21T14:20:54.799027+0000","last_clean":"2024-03-21T14:20:54.799027+0000","last_became_active":"2024-03-21T14:19:14.692456+0000","last_became_peered":"2024-03-21T14:19:14.692456+0000","last_unstale":"2024-03-21T14:20:54.799027+0000","last_undegraded":"2024-03-21T14:20:54.799027+0000","last_fullsized":"2024-03-21T14:20:54.799027+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.159386+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.568996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":740,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797422+0000","last_change":"2024-03-21T14:20:35.508783+0000","last_active":"2024-03-21T14:20:54.797422+0000","last_peered":"2024-03-21T14:20:54.797422+0000","last_clean":"2024-03-21T14:20:54.797422+0000","last_became_active":"2024-03-21T14:19:14.689666+0000","last_became_peered":"2024-03-21T14:19:14.689666+0000","last_unstale":"2024-03-21T14:20:54.797422+0000","last_undegraded":"2024-03-21T14:20:54.797422+0000","last_fullsized":"2024-03-21T14:20:54.797422+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:35.508660+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:35.508660+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.775681+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":752,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798303+0000","last_change":"2024-03-21T14:20:50.816435+0000","last_active":"2024-03-21T14:20:54.798303+0000","last_peered":"2024-03-21T14:20:54.798303+0000","last_clean":"2024-03-21T14:20:54.798303+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:20:54.798303+0000","last_undegraded":"2024-03-21T14:20:54.798303+0000","last_fullsized":"2024-03-21T14:20:54.798303+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:50.816334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:50.816334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:03.549011+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":264,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815728+0000","last_change":"2024-03-21T14:19:14.319436+0000","last_active":"2024-03-21T14:20:53.815728+0000","last_peered":"2024-03-21T14:20:53.815728+0000","last_clean":"2024-03-21T14:20:53.815728+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:20:53.815728+0000","last_undegraded":"2024-03-21T14:20:53.815728+0000","last_fullsized":"2024-03-21T14:20:53.815728+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:19:14.319346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:19:14.319346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.098031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":759,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098501+0000","last_change":"2024-03-21T14:16:57.382377+0000","last_active":"2024-03-21T14:21:00.098501+0000","last_peered":"2024-03-21T14:21:00.098501+0000","last_clean":"2024-03-21T14:21:00.098501+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:21:00.098501+0000","last_undegraded":"2024-03-21T14:21:00.098501+0000","last_fullsized":"2024-03-21T14:21:00.098501+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.382291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.382291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"360'54","reported_seq":678,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798042+0000","last_change":"2024-03-21T14:20:31.540383+0000","last_active":"2024-03-21T14:20:54.798042+0000","last_peered":"2024-03-21T14:20:54.798042+0000","last_clean":"2024-03-21T14:20:54.798042+0000","last_became_active":"2024-03-21T14:19:14.693616+0000","last_became_peered":"2024-03-21T14:19:14.693616+0000","last_unstale":"2024-03-21T14:20:54.798042+0000","last_undegraded":"2024-03-21T14:20:54.798042+0000","last_fullsized":"2024-03-21T14:20:54.798042+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'54","last_scrub_stamp":"2024-03-21T14:20:31.540283+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.540283+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:40.298858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":630,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817450+0000","last_change":"2024-03-21T14:19:34.691493+0000","last_active":"2024-03-21T14:20:53.817450+0000","last_peered":"2024-03-21T14:20:53.817450+0000","last_clean":"2024-03-21T14:20:53.817450+0000","last_became_active":"2024-03-21T14:19:34.690636+0000","last_became_peered":"2024-03-21T14:19:34.690636+0000","last_unstale":"2024-03-21T14:20:53.817450+0000","last_undegraded":"2024-03-21T14:20:53.817450+0000","last_fullsized":"2024-03-21T14:20:53.817450+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:01.122304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:01.122304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097433+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":984,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098348+0000","last_change":"2024-03-21T14:16:58.345839+0000","last_active":"2024-03-21T14:21:00.098348+0000","last_peered":"2024-03-21T14:21:00.098348+0000","last_clean":"2024-03-21T14:21:00.098348+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:21:00.098348+0000","last_undegraded":"2024-03-21T14:21:00.098348+0000","last_fullsized":"2024-03-21T14:21:00.098348+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:58.345739+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:58.345739+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":773,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.399343+0000","last_change":"2024-03-21T14:20:51.457667+0000","last_active":"2024-03-21T14:21:00.399343+0000","last_peered":"2024-03-21T14:21:00.399343+0000","last_clean":"2024-03-21T14:21:00.399343+0000","last_became_active":"2024-03-21T14:19:14.685026+0000","last_became_peered":"2024-03-21T14:19:14.685026+0000","last_unstale":"2024-03-21T14:21:00.399343+0000","last_undegraded":"2024-03-21T14:21:00.399343+0000","last_fullsized":"2024-03-21T14:21:00.399343+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:51.457549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.457549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:18.394890+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":772,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815188+0000","last_change":"2024-03-21T14:19:17.321024+0000","last_active":"2024-03-21T14:20:53.815188+0000","last_peered":"2024-03-21T14:20:53.815188+0000","last_clean":"2024-03-21T14:20:53.815188+0000","last_became_active":"2024-03-21T14:19:14.697007+0000","last_became_peered":"2024-03-21T14:19:14.697007+0000","last_unstale":"2024-03-21T14:20:53.815188+0000","last_undegraded":"2024-03-21T14:20:53.815188+0000","last_fullsized":"2024-03-21T14:20:53.815188+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:17.320902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:19:17.320902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097860+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":607,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797018+0000","last_change":"2024-03-21T14:20:48.551092+0000","last_active":"2024-03-21T14:20:54.797018+0000","last_peered":"2024-03-21T14:20:54.797018+0000","last_clean":"2024-03-21T14:20:54.797018+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:20:54.797018+0000","last_undegraded":"2024-03-21T14:20:54.797018+0000","last_fullsized":"2024-03-21T14:20:54.797018+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.550975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.550975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:10.129850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":850,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817369+0000","last_change":"2024-03-21T14:19:34.717581+0000","last_active":"2024-03-21T14:20:53.817369+0000","last_peered":"2024-03-21T14:20:53.817369+0000","last_clean":"2024-03-21T14:20:53.817369+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:20:53.817369+0000","last_undegraded":"2024-03-21T14:20:53.817369+0000","last_fullsized":"2024-03-21T14:20:53.817369+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:18:41.125165+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:18:41.125165+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097636+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":582,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798851+0000","last_change":"2024-03-21T14:19:14.683429+0000","last_active":"2024-03-21T14:20:54.798851+0000","last_peered":"2024-03-21T14:20:54.798851+0000","last_clean":"2024-03-21T14:20:54.798851+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:20:54.798851+0000","last_undegraded":"2024-03-21T14:20:54.798851+0000","last_fullsized":"2024-03-21T14:20:54.798851+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:07.244640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.244640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":229,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797022+0000","last_change":"2024-03-21T14:16:55.947536+0000","last_active":"2024-03-21T14:20:54.797022+0000","last_peered":"2024-03-21T14:20:54.797022+0000","last_clean":"2024-03-21T14:20:54.797022+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:20:54.797022+0000","last_undegraded":"2024-03-21T14:20:54.797022+0000","last_fullsized":"2024-03-21T14:20:54.797022+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:55.947438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:55.947438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1084,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398360+0000","last_change":"2024-03-21T14:19:40.473390+0000","last_active":"2024-03-21T14:21:00.398360+0000","last_peered":"2024-03-21T14:21:00.398360+0000","last_clean":"2024-03-21T14:21:00.398360+0000","last_became_active":"2024-03-21T14:19:37.727165+0000","last_became_peered":"2024-03-21T14:19:37.727165+0000","last_unstale":"2024-03-21T14:21:00.398360+0000","last_undegraded":"2024-03-21T14:21:00.398360+0000","last_fullsized":"2024-03-21T14:21:00.398360+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:40.473280+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:40.473280+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:01.733220+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":778,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797893+0000","last_change":"2024-03-21T14:20:49.815275+0000","last_active":"2024-03-21T14:20:54.797893+0000","last_peered":"2024-03-21T14:20:54.797893+0000","last_clean":"2024-03-21T14:20:54.797893+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:20:54.797893+0000","last_undegraded":"2024-03-21T14:20:54.797893+0000","last_fullsized":"2024-03-21T14:20:54.797893+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:49.815144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.815144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:11.062071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":622,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099044+0000","last_change":"2024-03-21T14:17:59.176788+0000","last_active":"2024-03-21T14:21:00.099044+0000","last_peered":"2024-03-21T14:21:00.099044+0000","last_clean":"2024-03-21T14:21:00.099044+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:21:00.099044+0000","last_undegraded":"2024-03-21T14:21:00.099044+0000","last_fullsized":"2024-03-21T14:21:00.099044+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'29","last_scrub_stamp":"2024-03-21T14:17:59.176541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:59.176541+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":124,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798567+0000","last_change":"2024-03-21T14:20:33.512686+0000","last_active":"2024-03-21T14:20:54.798567+0000","last_peered":"2024-03-21T14:20:54.798567+0000","last_clean":"2024-03-21T14:20:54.798567+0000","last_became_active":"2024-03-21T14:19:14.690145+0000","last_became_peered":"2024-03-21T14:19:14.690145+0000","last_unstale":"2024-03-21T14:20:54.798567+0000","last_undegraded":"2024-03-21T14:20:54.798567+0000","last_fullsized":"2024-03-21T14:20:54.798567+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:33.512571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:33.512571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:00.208164+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":834,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.816603+0000","last_change":"2024-03-21T14:18:43.149509+0000","last_active":"2024-03-21T14:20:53.816603+0000","last_peered":"2024-03-21T14:20:53.816603+0000","last_clean":"2024-03-21T14:20:53.816603+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:20:53.816603+0000","last_undegraded":"2024-03-21T14:20:53.816603+0000","last_fullsized":"2024-03-21T14:20:53.816603+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:18:43.148036+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:18:43.148036+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097757+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":778,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817035+0000","last_change":"2024-03-21T14:20:48.131092+0000","last_active":"2024-03-21T14:20:53.817035+0000","last_peered":"2024-03-21T14:20:53.817035+0000","last_clean":"2024-03-21T14:20:53.817035+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:20:53.817035+0000","last_undegraded":"2024-03-21T14:20:53.817035+0000","last_fullsized":"2024-03-21T14:20:53.817035+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.130989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.130989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:04.073822+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":569,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098484+0000","last_change":"2024-03-21T14:20:21.778652+0000","last_active":"2024-03-21T14:21:00.098484+0000","last_peered":"2024-03-21T14:21:00.098484+0000","last_clean":"2024-03-21T14:21:00.098484+0000","last_became_active":"2024-03-21T14:20:21.778326+0000","last_became_peered":"2024-03-21T14:20:21.778326+0000","last_unstale":"2024-03-21T14:21:00.098484+0000","last_undegraded":"2024-03-21T14:21:00.098484+0000","last_fullsized":"2024-03-21T14:21:00.098484+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:50.207374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:50.207374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370492+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1779,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:56.267240+0000","last_change":"2024-03-21T14:18:35.154897+0000","last_active":"2024-03-21T14:20:56.267240+0000","last_peered":"2024-03-21T14:20:56.267240+0000","last_clean":"2024-03-21T14:20:56.267240+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:20:56.267240+0000","last_undegraded":"2024-03-21T14:20:56.267240+0000","last_fullsized":"2024-03-21T14:20:56.267240+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:18:35.154808+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:35.154808+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097551+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1038,"num_read_kb":1033,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":707,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817534+0000","last_change":"2024-03-21T14:19:34.715158+0000","last_active":"2024-03-21T14:20:53.817534+0000","last_peered":"2024-03-21T14:20:53.817534+0000","last_clean":"2024-03-21T14:20:53.817534+0000","last_became_active":"2024-03-21T14:19:34.692301+0000","last_became_peered":"2024-03-21T14:19:34.692301+0000","last_unstale":"2024-03-21T14:20:53.817534+0000","last_undegraded":"2024-03-21T14:20:53.817534+0000","last_fullsized":"2024-03-21T14:20:53.817534+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:19:24.502253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:24.502253+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097950+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":231,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817666+0000","last_change":"2024-03-21T14:18:04.170207+0000","last_active":"2024-03-21T14:20:53.817666+0000","last_peered":"2024-03-21T14:20:53.817666+0000","last_clean":"2024-03-21T14:20:53.817666+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:20:53.817666+0000","last_undegraded":"2024-03-21T14:20:53.817666+0000","last_fullsized":"2024-03-21T14:20:53.817666+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:04.170084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:18:04.170084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097511+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":570,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798062+0000","last_change":"2024-03-21T14:20:30.945601+0000","last_active":"2024-03-21T14:20:54.798062+0000","last_peered":"2024-03-21T14:20:54.798062+0000","last_clean":"2024-03-21T14:20:54.798062+0000","last_became_active":"2024-03-21T14:20:24.825637+0000","last_became_peered":"2024-03-21T14:20:24.825637+0000","last_unstale":"2024-03-21T14:20:54.798062+0000","last_undegraded":"2024-03-21T14:20:54.798062+0000","last_fullsized":"2024-03-21T14:20:54.798062+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:30.945471+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.945471+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:41.790389+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":767,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.815606+0000","last_change":"2024-03-21T14:19:34.274417+0000","last_active":"2024-03-21T14:20:53.815606+0000","last_peered":"2024-03-21T14:20:53.815606+0000","last_clean":"2024-03-21T14:20:53.815606+0000","last_became_active":"2024-03-21T14:19:14.697998+0000","last_became_peered":"2024-03-21T14:19:14.697998+0000","last_unstale":"2024-03-21T14:20:53.815606+0000","last_undegraded":"2024-03-21T14:20:53.815606+0000","last_fullsized":"2024-03-21T14:20:53.815606+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:34.274201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:19:34.274201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:00.600251+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":773,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817134+0000","last_change":"2024-03-21T14:19:26.469534+0000","last_active":"2024-03-21T14:20:53.817134+0000","last_peered":"2024-03-21T14:20:53.817134+0000","last_clean":"2024-03-21T14:20:53.817134+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:20:53.817134+0000","last_undegraded":"2024-03-21T14:20:53.817134+0000","last_fullsized":"2024-03-21T14:20:53.817134+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:26.469255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:19:26.469255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097893+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":501,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.782675+0000","last_change":"2024-03-21T14:20:31.530370+0000","last_active":"2024-03-21T14:21:00.782675+0000","last_peered":"2024-03-21T14:21:00.782675+0000","last_clean":"2024-03-21T14:21:00.782675+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:21:00.782675+0000","last_undegraded":"2024-03-21T14:21:00.782675+0000","last_fullsized":"2024-03-21T14:21:00.782675+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:20:31.530270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.530270+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.391998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":128,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798634+0000","last_change":"2024-03-21T14:20:34.551705+0000","last_active":"2024-03-21T14:20:54.798634+0000","last_peered":"2024-03-21T14:20:54.798634+0000","last_clean":"2024-03-21T14:20:54.798634+0000","last_became_active":"2024-03-21T14:19:14.675538+0000","last_became_peered":"2024-03-21T14:19:14.675538+0000","last_unstale":"2024-03-21T14:20:54.798634+0000","last_undegraded":"2024-03-21T14:20:54.798634+0000","last_fullsized":"2024-03-21T14:20:54.798634+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:34.551577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:20:34.551577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:44.306785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":744,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797993+0000","last_change":"2024-03-21T14:20:24.821031+0000","last_active":"2024-03-21T14:20:54.797993+0000","last_peered":"2024-03-21T14:20:54.797993+0000","last_clean":"2024-03-21T14:20:54.797993+0000","last_became_active":"2024-03-21T14:20:24.819844+0000","last_became_peered":"2024-03-21T14:20:24.819844+0000","last_unstale":"2024-03-21T14:20:54.797993+0000","last_undegraded":"2024-03-21T14:20:54.797993+0000","last_fullsized":"2024-03-21T14:20:54.797993+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:15.665015+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.665015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":267,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797235+0000","last_change":"2024-03-21T14:19:15.688754+0000","last_active":"2024-03-21T14:20:54.797235+0000","last_peered":"2024-03-21T14:20:54.797235+0000","last_clean":"2024-03-21T14:20:54.797235+0000","last_became_active":"2024-03-21T14:19:15.688238+0000","last_became_peered":"2024-03-21T14:19:15.688238+0000","last_unstale":"2024-03-21T14:20:54.797235+0000","last_undegraded":"2024-03-21T14:20:54.797235+0000","last_fullsized":"2024-03-21T14:20:54.797235+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:11.920076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:11.920076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":704,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798193+0000","last_change":"2024-03-21T14:20:32.884624+0000","last_active":"2024-03-21T14:20:54.798193+0000","last_peered":"2024-03-21T14:20:54.798193+0000","last_clean":"2024-03-21T14:20:54.798193+0000","last_became_active":"2024-03-21T14:19:14.695943+0000","last_became_peered":"2024-03-21T14:19:14.695943+0000","last_unstale":"2024-03-21T14:20:54.798193+0000","last_undegraded":"2024-03-21T14:20:54.798193+0000","last_fullsized":"2024-03-21T14:20:54.798193+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:32.884514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:32.884514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:41.374001+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":803,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.816477+0000","last_change":"2024-03-21T14:18:29.201331+0000","last_active":"2024-03-21T14:20:53.816477+0000","last_peered":"2024-03-21T14:20:53.816477+0000","last_clean":"2024-03-21T14:20:53.816477+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:20:53.816477+0000","last_undegraded":"2024-03-21T14:20:53.816477+0000","last_fullsized":"2024-03-21T14:20:53.816477+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:18:29.201228+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.201228+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097594+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":742,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.796830+0000","last_change":"2024-03-21T14:18:44.046070+0000","last_active":"2024-03-21T14:20:54.796830+0000","last_peered":"2024-03-21T14:20:54.796830+0000","last_clean":"2024-03-21T14:20:54.796830+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:20:54.796830+0000","last_undegraded":"2024-03-21T14:20:54.796830+0000","last_fullsized":"2024-03-21T14:20:54.796830+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'11","last_scrub_stamp":"2024-03-21T14:18:44.045985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:44.045985+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":13,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099413+0000","last_change":"2024-03-21T14:20:59.110795+0000","last_active":"2024-03-21T14:21:00.099413+0000","last_peered":"2024-03-21T14:21:00.099413+0000","last_clean":"2024-03-21T14:21:00.099413+0000","last_became_active":"2024-03-21T14:20:59.110468+0000","last_became_peered":"2024-03-21T14:20:59.110468+0000","last_unstale":"2024-03-21T14:21:00.099413+0000","last_undegraded":"2024-03-21T14:21:00.099413+0000","last_fullsized":"2024-03-21T14:21:00.099413+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:09.204354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":826,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797653+0000","last_change":"2024-03-21T14:20:52.764101+0000","last_active":"2024-03-21T14:20:54.797653+0000","last_peered":"2024-03-21T14:20:54.797653+0000","last_clean":"2024-03-21T14:20:54.797653+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:20:54.797653+0000","last_undegraded":"2024-03-21T14:20:54.797653+0000","last_fullsized":"2024-03-21T14:20:54.797653+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:52.763988+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.763988+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:11.926808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":773,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398834+0000","last_change":"2024-03-21T14:20:29.685381+0000","last_active":"2024-03-21T14:21:00.398834+0000","last_peered":"2024-03-21T14:21:00.398834+0000","last_clean":"2024-03-21T14:21:00.398834+0000","last_became_active":"2024-03-21T14:19:14.684876+0000","last_became_peered":"2024-03-21T14:19:14.684876+0000","last_unstale":"2024-03-21T14:21:00.398834+0000","last_undegraded":"2024-03-21T14:21:00.398834+0000","last_fullsized":"2024-03-21T14:21:00.398834+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:29.685267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:27.609702+0000","last_clean_scrub_stamp":"2024-03-21T14:20:29.685267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:45.309330+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":640,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817322+0000","last_change":"2024-03-21T14:20:51.116248+0000","last_active":"2024-03-21T14:20:53.817322+0000","last_peered":"2024-03-21T14:20:53.817322+0000","last_clean":"2024-03-21T14:20:53.817322+0000","last_became_active":"2024-03-21T14:19:34.692258+0000","last_became_peered":"2024-03-21T14:19:34.692258+0000","last_unstale":"2024-03-21T14:20:53.817322+0000","last_undegraded":"2024-03-21T14:20:53.817322+0000","last_fullsized":"2024-03-21T14:20:53.817322+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:20:51.116139+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.116139+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:05.825547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":821,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098446+0000","last_change":"2024-03-21T14:17:56.127388+0000","last_active":"2024-03-21T14:21:00.098446+0000","last_peered":"2024-03-21T14:21:00.098446+0000","last_clean":"2024-03-21T14:21:00.098446+0000","last_became_active":"2024-03-21T14:15:59.607826+0000","last_became_peered":"2024-03-21T14:15:59.607826+0000","last_unstale":"2024-03-21T14:21:00.098446+0000","last_undegraded":"2024-03-21T14:21:00.098446+0000","last_fullsized":"2024-03-21T14:21:00.098446+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'28","last_scrub_stamp":"2024-03-21T14:17:56.127254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:56.127254+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":986,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797709+0000","last_change":"2024-03-21T14:18:38.115200+0000","last_active":"2024-03-21T14:20:54.797709+0000","last_peered":"2024-03-21T14:20:54.797709+0000","last_clean":"2024-03-21T14:20:54.797709+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:20:54.797709+0000","last_undegraded":"2024-03-21T14:20:54.797709+0000","last_fullsized":"2024-03-21T14:20:54.797709+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:18:38.115111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:38.115111+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":826,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098865+0000","last_change":"2024-03-21T14:19:14.682603+0000","last_active":"2024-03-21T14:21:00.098865+0000","last_peered":"2024-03-21T14:21:00.098865+0000","last_clean":"2024-03-21T14:21:00.098865+0000","last_became_active":"2024-03-21T14:19:14.682251+0000","last_became_peered":"2024-03-21T14:19:14.682251+0000","last_unstale":"2024-03-21T14:21:00.098865+0000","last_undegraded":"2024-03-21T14:21:00.098865+0000","last_fullsized":"2024-03-21T14:21:00.098865+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.954782+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.954782+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370246+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":773,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098713+0000","last_change":"2024-03-21T14:20:21.782339+0000","last_active":"2024-03-21T14:21:00.098713+0000","last_peered":"2024-03-21T14:21:00.098713+0000","last_clean":"2024-03-21T14:21:00.098713+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:21:00.098713+0000","last_undegraded":"2024-03-21T14:21:00.098713+0000","last_fullsized":"2024-03-21T14:21:00.098713+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:11.405823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:17:11.405823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370394+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":764,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798631+0000","last_change":"2024-03-21T14:17:02.767201+0000","last_active":"2024-03-21T14:20:54.798631+0000","last_peered":"2024-03-21T14:20:54.798631+0000","last_clean":"2024-03-21T14:20:54.798631+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:20:54.798631+0000","last_undegraded":"2024-03-21T14:20:54.798631+0000","last_fullsized":"2024-03-21T14:20:54.798631+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:02.767079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:17:02.767079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":65,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797082+0000","last_change":"2024-03-21T14:20:21.803142+0000","last_active":"2024-03-21T14:20:54.797082+0000","last_peered":"2024-03-21T14:20:54.797082+0000","last_clean":"2024-03-21T14:20:54.797082+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:20:54.797082+0000","last_undegraded":"2024-03-21T14:20:54.797082+0000","last_fullsized":"2024-03-21T14:20:54.797082+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:19:12.604917+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:19:12.604917+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":745,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098314+0000","last_change":"2024-03-21T14:17:14.254451+0000","last_active":"2024-03-21T14:21:00.098314+0000","last_peered":"2024-03-21T14:21:00.098314+0000","last_clean":"2024-03-21T14:21:00.098314+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:21:00.098314+0000","last_undegraded":"2024-03-21T14:21:00.098314+0000","last_fullsized":"2024-03-21T14:21:00.098314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:14.254334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:14.254334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370443+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":740,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798778+0000","last_change":"2024-03-21T14:20:36.503605+0000","last_active":"2024-03-21T14:20:54.798778+0000","last_peered":"2024-03-21T14:20:54.798778+0000","last_clean":"2024-03-21T14:20:54.798778+0000","last_became_active":"2024-03-21T14:19:14.692919+0000","last_became_peered":"2024-03-21T14:19:14.692919+0000","last_unstale":"2024-03-21T14:20:54.798778+0000","last_undegraded":"2024-03-21T14:20:54.798778+0000","last_fullsized":"2024-03-21T14:20:54.798778+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:36.503498+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:36.503498+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:04.933569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":781,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817011+0000","last_change":"2024-03-21T14:20:43.142589+0000","last_active":"2024-03-21T14:20:53.817011+0000","last_peered":"2024-03-21T14:20:53.817011+0000","last_clean":"2024-03-21T14:20:53.817011+0000","last_became_active":"2024-03-21T14:19:34.691121+0000","last_became_peered":"2024-03-21T14:19:34.691121+0000","last_unstale":"2024-03-21T14:20:53.817011+0000","last_undegraded":"2024-03-21T14:20:53.817011+0000","last_fullsized":"2024-03-21T14:20:53.817011+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:43.142495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:20:43.142495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:03.496861+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":983,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099548+0000","last_change":"2024-03-21T14:17:03.353514+0000","last_active":"2024-03-21T14:21:00.099548+0000","last_peered":"2024-03-21T14:21:00.099548+0000","last_clean":"2024-03-21T14:21:00.099548+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:21:00.099548+0000","last_undegraded":"2024-03-21T14:21:00.099548+0000","last_fullsized":"2024-03-21T14:21:00.099548+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:17:03.353394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.353394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370412+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":47,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797185+0000","last_change":"2024-03-21T14:20:21.783183+0000","last_active":"2024-03-21T14:20:54.797185+0000","last_peered":"2024-03-21T14:20:54.797185+0000","last_clean":"2024-03-21T14:20:54.797185+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:20:54.797185+0000","last_undegraded":"2024-03-21T14:20:54.797185+0000","last_fullsized":"2024-03-21T14:20:54.797185+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:53.136512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:53.136512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":811,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817394+0000","last_change":"2024-03-21T14:20:42.035288+0000","last_active":"2024-03-21T14:20:53.817394+0000","last_peered":"2024-03-21T14:20:53.817394+0000","last_clean":"2024-03-21T14:20:53.817394+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:20:53.817394+0000","last_undegraded":"2024-03-21T14:20:53.817394+0000","last_fullsized":"2024-03-21T14:20:53.817394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:20:42.035180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.035180+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:05.671907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798221+0000","last_change":"2024-03-21T14:18:28.880752+0000","last_active":"2024-03-21T14:20:54.798221+0000","last_peered":"2024-03-21T14:20:54.798221+0000","last_clean":"2024-03-21T14:20:54.798221+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:20:54.798221+0000","last_undegraded":"2024-03-21T14:20:54.798221+0000","last_fullsized":"2024-03-21T14:20:54.798221+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:28.880620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:28.880620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":748,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099462+0000","last_change":"2024-03-21T14:19:15.688745+0000","last_active":"2024-03-21T14:21:00.099462+0000","last_peered":"2024-03-21T14:21:00.099462+0000","last_clean":"2024-03-21T14:21:00.099462+0000","last_became_active":"2024-03-21T14:19:15.688278+0000","last_became_peered":"2024-03-21T14:19:15.688278+0000","last_unstale":"2024-03-21T14:21:00.099462+0000","last_undegraded":"2024-03-21T14:21:00.099462+0000","last_fullsized":"2024-03-21T14:21:00.099462+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:16.566202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:16.566202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370296+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":770,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098351+0000","last_change":"2024-03-21T14:20:48.538833+0000","last_active":"2024-03-21T14:21:00.098351+0000","last_peered":"2024-03-21T14:21:00.098351+0000","last_clean":"2024-03-21T14:21:00.098351+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:21:00.098351+0000","last_undegraded":"2024-03-21T14:21:00.098351+0000","last_fullsized":"2024-03-21T14:21:00.098351+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.538744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.538744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.083242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":806,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099414+0000","last_change":"2024-03-21T14:18:32.414098+0000","last_active":"2024-03-21T14:21:00.099414+0000","last_peered":"2024-03-21T14:21:00.099414+0000","last_clean":"2024-03-21T14:21:00.099414+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:21:00.099414+0000","last_undegraded":"2024-03-21T14:21:00.099414+0000","last_fullsized":"2024-03-21T14:21:00.099414+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:18:32.413975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:32.413975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370621+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-21T14:20:58.082826+0000","last_change":"2024-03-21T14:20:58.082826+0000","last_active":"2024-03-21T14:20:58.082826+0000","last_peered":"2024-03-21T14:20:58.082826+0000","last_clean":"2024-03-21T14:20:58.082826+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-21T14:20:58.082826+0000","last_undegraded":"2024-03-21T14:20:58.082826+0000","last_fullsized":"2024-03-21T14:20:58.082826+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":740,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797064+0000","last_change":"2024-03-21T14:20:37.552524+0000","last_active":"2024-03-21T14:20:54.797064+0000","last_peered":"2024-03-21T14:20:54.797064+0000","last_clean":"2024-03-21T14:20:54.797064+0000","last_became_active":"2024-03-21T14:19:14.700353+0000","last_became_peered":"2024-03-21T14:19:14.700353+0000","last_unstale":"2024-03-21T14:20:54.797064+0000","last_undegraded":"2024-03-21T14:20:54.797064+0000","last_fullsized":"2024-03-21T14:20:54.797064+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:37.552424+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.552424+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.901528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":559,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798978+0000","last_change":"2024-03-21T14:20:45.611947+0000","last_active":"2024-03-21T14:20:54.798978+0000","last_peered":"2024-03-21T14:20:54.798978+0000","last_clean":"2024-03-21T14:20:54.798978+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:20:54.798978+0000","last_undegraded":"2024-03-21T14:20:54.798978+0000","last_fullsized":"2024-03-21T14:20:54.798978+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.611844+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:06.146803+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":764,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:53.817224+0000","last_change":"2024-03-21T14:18:27.281305+0000","last_active":"2024-03-21T14:20:53.817224+0000","last_peered":"2024-03-21T14:20:53.817224+0000","last_clean":"2024-03-21T14:20:53.817224+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:20:53.817224+0000","last_undegraded":"2024-03-21T14:20:53.817224+0000","last_fullsized":"2024-03-21T14:20:53.817224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:27.281190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_clean_scrub_stamp":"2024-03-21T14:18:27.281190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.097716+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":51,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.798547+0000","last_change":"2024-03-21T14:20:19.703923+0000","last_active":"2024-03-21T14:20:54.798547+0000","last_peered":"2024-03-21T14:20:54.798547+0000","last_clean":"2024-03-21T14:20:54.798547+0000","last_became_active":"2024-03-21T14:20:19.703587+0000","last_became_peered":"2024-03-21T14:20:19.703587+0000","last_unstale":"2024-03-21T14:20:54.798547+0000","last_undegraded":"2024-03-21T14:20:54.798547+0000","last_fullsized":"2024-03-21T14:20:54.798547+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:16.247145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.247145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:20:58.569127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":807,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099124+0000","last_change":"2024-03-21T14:20:46.480561+0000","last_active":"2024-03-21T14:21:00.099124+0000","last_peered":"2024-03-21T14:21:00.099124+0000","last_clean":"2024-03-21T14:21:00.099124+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:21:00.099124+0000","last_undegraded":"2024-03-21T14:21:00.099124+0000","last_fullsized":"2024-03-21T14:21:00.099124+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:20:46.480457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.480457+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:06.690255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":13,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398654+0000","last_change":"2024-03-21T14:20:59.387035+0000","last_active":"2024-03-21T14:21:00.398654+0000","last_peered":"2024-03-21T14:21:00.398654+0000","last_clean":"2024-03-21T14:21:00.398654+0000","last_became_active":"2024-03-21T14:20:59.386715+0000","last_became_peered":"2024-03-21T14:20:59.386715+0000","last_unstale":"2024-03-21T14:21:00.398654+0000","last_undegraded":"2024-03-21T14:21:00.398654+0000","last_fullsized":"2024-03-21T14:21:00.398654+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:09.502262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1083,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.398402+0000","last_change":"2024-03-21T14:19:36.429638+0000","last_active":"2024-03-21T14:21:00.398402+0000","last_peered":"2024-03-21T14:21:00.398402+0000","last_clean":"2024-03-21T14:21:00.398402+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:21:00.398402+0000","last_undegraded":"2024-03-21T14:21:00.398402+0000","last_fullsized":"2024-03-21T14:21:00.398402+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:36.429507+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:36.429507+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":774,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.099614+0000","last_change":"2024-03-21T14:19:15.689377+0000","last_active":"2024-03-21T14:21:00.099614+0000","last_peered":"2024-03-21T14:21:00.099614+0000","last_clean":"2024-03-21T14:21:00.099614+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:21:00.099614+0000","last_undegraded":"2024-03-21T14:21:00.099614+0000","last_fullsized":"2024-03-21T14:21:00.099614+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:13.493205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:13.493205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370649+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":750,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797816+0000","last_change":"2024-03-21T14:20:49.535756+0000","last_active":"2024-03-21T14:20:54.797816+0000","last_peered":"2024-03-21T14:20:54.797816+0000","last_clean":"2024-03-21T14:20:54.797816+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:20:54.797816+0000","last_undegraded":"2024-03-21T14:20:54.797816+0000","last_fullsized":"2024-03-21T14:20:54.797816+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.535652+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:10.421931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":776,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-21T14:20:54.797884+0000","last_change":"2024-03-21T14:19:34.693436+0000","last_active":"2024-03-21T14:20:54.797884+0000","last_peered":"2024-03-21T14:20:54.797884+0000","last_clean":"2024-03-21T14:20:54.797884+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:20:54.797884+0000","last_undegraded":"2024-03-21T14:20:54.797884+0000","last_fullsized":"2024-03-21T14:20:54.797884+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:17.921956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:17.921956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":810,"reported_epoch":413,"state":"active+clean","last_fresh":"2024-03-21T14:21:00.098791+0000","last_change":"2024-03-21T14:18:34.406758+0000","last_active":"2024-03-21T14:21:00.098791+0000","last_peered":"2024-03-21T14:21:00.098791+0000","last_clean":"2024-03-21T14:21:00.098791+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:21:00.098791+0000","last_undegraded":"2024-03-21T14:21:00.098791+0000","last_fullsized":"2024-03-21T14:21:00.098791+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:18:34.406635+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:18:34.406635+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370558+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":9,"acting":9,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1038,"num_read_kb":1033,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":3592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103394,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810044,"kb_used_data":2796,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92935108,"statfs":{"total":95995035648,"available":95165550592,"internally_reserved":0,"allocated":2863104,"data_stored":1776108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":826603978},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":1507533520921,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575105,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781936,"kb_used_data":2464,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963216,"statfs":{"total":95995035648,"available":95194333184,"internally_reserved":0,"allocated":2523136,"data_stored":1427229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738630,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727780,"kb_used_data":2456,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93017372,"statfs":{"total":95995035648,"available":95249788928,"internally_reserved":0,"allocated":2514944,"data_stored":1425862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598291990,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1947076,"kb_used_data":1696572,"kb_used_omap":7,"kb_used_meta":250488,"kb_avail":91798076,"statfs":{"total":95995035648,"available":94001229824,"internally_reserved":0,"allocated":1737289728,"data_stored":1736229023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":256499844},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182645,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647860,"kb_used_data":3048,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097292,"statfs":{"total":95995035648,"available":95331627008,"internally_reserved":0,"allocated":3121152,"data_stored":2016603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019122,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845840,"kb_used_data":2492,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92899312,"statfs":{"total":95995035648,"available":95128895488,"internally_reserved":0,"allocated":2551808,"data_stored":1447629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692077,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810312,"kb_used_data":3068,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92934840,"statfs":{"total":95995035648,"available":95165276160,"internally_reserved":0,"allocated":3141632,"data_stored":2038758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:21:04.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:21:04.587 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:04.611 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:04.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:04.618 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:04 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:04.618 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:04 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/966161542' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:21:04.618 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:04 smithi003 ceph-mon[29576]: osdmap e414: 8 total, 8 up, 7 in 2024-03-21T14:21:04.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:04 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:04.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:04 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/966161542' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:21:04.618 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:04 smithi003 ceph-mon[40510]: osdmap e414: 8 total, 8 up, 7 in 2024-03-21T14:21:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:04 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:04 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/966161542' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:21:04.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:04 smithi118 ceph-mon[33862]: osdmap e414: 8 total, 8 up, 7 in 2024-03-21T14:21:04.713 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:04.736 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:04.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:04.862 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:04.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:04.987 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:04.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:05.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:05.112 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:05.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:05.238 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:05.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:05.337 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:05.342 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:05.369 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:05.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:05.494 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:05.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[29576]: pgmap v877: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:05.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:21:05.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:21:05.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:21:05.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:05.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[40510]: pgmap v877: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:05.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:21:05.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:21:05.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:21:05.569 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:05 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:05.579 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:05 smithi118 ceph-mon[33862]: pgmap v877: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:05.579 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:05 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:21:05.579 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:05 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:21:05.579 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:05 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:21:05.579 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:05 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:05.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:05.603 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:21:05.603 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:21:05.604 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:21:05.604 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:21:05.604 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:21:05.620 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:05.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:05.745 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:05.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:05.871 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:05.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:05.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:05.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:05.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:06.098 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:06.122 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:06.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:06.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:06.350 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:06.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T14:21:06.374 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:06.476 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:06.499 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:06.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:06.601 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:06.604 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:21:06.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:06.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:06.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:06.740 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:06 smithi003 ceph-mon[29576]: pgmap v878: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 840 B/s rd, 0 op/s 2024-03-21T14:21:06.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:06 smithi003 ceph-mon[29576]: pgmap v879: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1004 B/s rd, 0 op/s 2024-03-21T14:21:06.741 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:21:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:21:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:21:06.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:06 smithi003 ceph-mon[40510]: pgmap v878: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 840 B/s rd, 0 op/s 2024-03-21T14:21:06.741 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:06 smithi003 ceph-mon[40510]: pgmap v879: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1004 B/s rd, 0 op/s 2024-03-21T14:21:06.749 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:06.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:06.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:06.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:06.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:21:06.971 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:06 smithi118 ceph-mon[33862]: pgmap v878: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 840 B/s rd, 0 op/s 2024-03-21T14:21:06.971 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:06 smithi118 ceph-mon[33862]: pgmap v879: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 1004 B/s rd, 0 op/s 2024-03-21T14:21:06.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:07.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:07.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:07.127 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:07.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:07.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:07.252 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:07.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:07.354 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:07.367 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:21:07.378 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:07.390 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:21:07.414 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:21:07.434 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:21:07.454 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:21:07.473 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:21:07.480 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:07.493 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:21:07.503 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:07.514 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:21:07.515 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:21:07.516 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:21:07.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:07.628 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:07.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:21:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:21:07.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:07.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:07.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:07.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:07 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2098204152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:07.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:07 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2098204152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:07.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:07.879 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:07.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:07.980 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:08.005 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:08.007 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:07 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2098204152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:08.085 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:08.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:08.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:08.234 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:08.258 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:08.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:08.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:08.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:08.514 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:21:08.514 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:21:08.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:08.636 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:08.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:08.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:08.864 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:08.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[29576]: pgmap v880: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 726 B/s rd, 0 op/s 2024-03-21T14:21:08.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:08.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:21:08.873 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:21:08.888 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[40510]: pgmap v880: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 726 B/s rd, 0 op/s 2024-03-21T14:21:08.888 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:08.888 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:21:08.888 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:08 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:21:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:08.889 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:08.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:08.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:09.013 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:09.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:09.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:08 smithi118 ceph-mon[33862]: pgmap v880: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 726 B/s rd, 0 op/s 2024-03-21T14:21:09.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:09.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:08 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:21:09.110 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:08 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:21:09.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:09.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:21:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:09.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:09.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:09.265 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:09.367 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:09.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:09.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:09.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:09.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:09.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:09.769 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:09.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:09.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:09.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:09.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:09 smithi003 ceph-mon[29576]: osdmap e415: 8 total, 8 up, 7 in 2024-03-21T14:21:10.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:09 smithi003 ceph-mon[29576]: osdmap e416: 8 total, 8 up, 7 in 2024-03-21T14:21:10.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:09 smithi003 ceph-mon[40510]: osdmap e415: 8 total, 8 up, 7 in 2024-03-21T14:21:10.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:09 smithi003 ceph-mon[40510]: osdmap e416: 8 total, 8 up, 7 in 2024-03-21T14:21:10.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:10.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:09 smithi118 ceph-mon[33862]: osdmap e415: 8 total, 8 up, 7 in 2024-03-21T14:21:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:09 smithi118 ceph-mon[33862]: osdmap e416: 8 total, 8 up, 7 in 2024-03-21T14:21:10.246 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:10.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:10.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:10.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:10.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.521 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:10.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:10.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:10.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:21:10.649 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:10.735 INFO:teuthology.orchestra.run.smithi003.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-21T14:21:10.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:10 smithi003 ceph-mon[29576]: pgmap v882: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 870 B/s rd, 0 op/s 2024-03-21T14:21:10.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:10 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1245093769' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:21:10.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:10 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:21:10.751 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:10.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:10 smithi003 ceph-mon[40510]: pgmap v882: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 870 B/s rd, 0 op/s 2024-03-21T14:21:10.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:10 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1245093769' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:21:10.752 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:10 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:21:10.778 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:10.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:10.879 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:10.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:10.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:21:10.976 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:21:11.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.037 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:11.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:11.138 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:11.165 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:10 smithi118 ceph-mon[33862]: pgmap v882: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 870 B/s rd, 0 op/s 2024-03-21T14:21:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:10 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1245093769' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:21:11.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:10 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-21T14:21:11.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:11.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:11.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:11.421 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:11.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:11.550 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:11.652 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:11.679 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:11.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:11.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.807 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:11.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:11.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:11.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:11.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:11.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:11.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:11 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:21:11.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:11 smithi003 ceph-mon[29576]: osdmap e417: 8 total, 8 up, 7 in 2024-03-21T14:21:11.936 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:11 smithi003 ceph-mon[29576]: from='client.15966 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:21:11.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:11 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:21:11.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:11 smithi003 ceph-mon[40510]: osdmap e417: 8 total, 8 up, 7 in 2024-03-21T14:21:11.936 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:11 smithi003 ceph-mon[40510]: from='client.15966 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:21:12.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:12.160 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:11 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-21T14:21:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:11 smithi118 ceph-mon[33862]: osdmap e417: 8 total, 8 up, 7 in 2024-03-21T14:21:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:11 smithi118 ceph-mon[33862]: from='client.15966 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:21:12.188 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:12.289 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:12.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.319 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:12.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:12.447 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:12.549 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:12.577 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:12.661 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":884,"stamp":"2024-03-21T14:21:10.726186+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2178,"num_read_kb":1930,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":166,"num_bytes_recovered":3868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2297,"ondisk_log_size":2297,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6475060,"kb_used_data":1617124,"kb_used_omap":17,"kb_used_meta":4857838,"kb_avail":649741004,"statfs":{"total":671965249536,"available":665334788096,"internally_reserved":0,"allocated":1655934976,"data_stored":1648193491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":17623,"internal_metadata":4974426921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.533071"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":750,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369674+0000","last_change":"2024-03-21T14:18:38.441703+0000","last_active":"2024-03-21T14:21:04.369674+0000","last_peered":"2024-03-21T14:21:04.369674+0000","last_clean":"2024-03-21T14:21:04.369674+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:21:04.369674+0000","last_undegraded":"2024-03-21T14:21:04.369674+0000","last_fullsized":"2024-03-21T14:21:04.369674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:38.441623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:38.441623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370573+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":771,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313050+0000","last_change":"2024-03-21T14:19:09.223055+0000","last_active":"2024-03-21T14:21:03.313050+0000","last_peered":"2024-03-21T14:21:03.313050+0000","last_clean":"2024-03-21T14:21:03.313050+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:21:03.313050+0000","last_undegraded":"2024-03-21T14:21:03.313050+0000","last_fullsized":"2024-03-21T14:21:03.313050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:09.222911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:19:09.222911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113213+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":326,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650427+0000","last_change":"2024-03-21T14:20:42.558129+0000","last_active":"2024-03-21T14:21:04.650427+0000","last_peered":"2024-03-21T14:21:04.650427+0000","last_clean":"2024-03-21T14:21:04.650427+0000","last_became_active":"2024-03-21T14:19:34.695760+0000","last_became_peered":"2024-03-21T14:19:34.695760+0000","last_unstale":"2024-03-21T14:21:04.650427+0000","last_undegraded":"2024-03-21T14:21:04.650427+0000","last_fullsized":"2024-03-21T14:21:04.650427+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.558048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.558048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:58.393037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":816,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369917+0000","last_change":"2024-03-21T14:20:41.426448+0000","last_active":"2024-03-21T14:21:04.369917+0000","last_peered":"2024-03-21T14:21:04.369917+0000","last_clean":"2024-03-21T14:21:04.369917+0000","last_became_active":"2024-03-21T14:20:21.782187+0000","last_became_peered":"2024-03-21T14:20:21.782187+0000","last_unstale":"2024-03-21T14:21:04.369917+0000","last_undegraded":"2024-03-21T14:21:04.369917+0000","last_fullsized":"2024-03-21T14:21:04.369917+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:20:41.426344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:41.426344+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.597745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":55,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828526+0000","last_change":"2024-03-21T14:20:21.782736+0000","last_active":"2024-03-21T14:21:03.828526+0000","last_peered":"2024-03-21T14:21:03.828526+0000","last_clean":"2024-03-21T14:21:03.828526+0000","last_became_active":"2024-03-21T14:20:21.782401+0000","last_became_peered":"2024-03-21T14:20:21.782401+0000","last_unstale":"2024-03-21T14:21:03.828526+0000","last_undegraded":"2024-03-21T14:21:03.828526+0000","last_fullsized":"2024-03-21T14:21:03.828526+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:15.807973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.807973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813629+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":769,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369814+0000","last_change":"2024-03-21T14:20:52.477097+0000","last_active":"2024-03-21T14:21:04.369814+0000","last_peered":"2024-03-21T14:21:04.369814+0000","last_clean":"2024-03-21T14:21:04.369814+0000","last_became_active":"2024-03-21T14:19:14.689459+0000","last_became_peered":"2024-03-21T14:19:14.689459+0000","last_unstale":"2024-03-21T14:21:04.369814+0000","last_undegraded":"2024-03-21T14:21:04.369814+0000","last_fullsized":"2024-03-21T14:21:04.369814+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:52.477009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.477009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:00.750211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":780,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829120+0000","last_change":"2024-03-21T14:20:37.939217+0000","last_active":"2024-03-21T14:21:03.829120+0000","last_peered":"2024-03-21T14:21:03.829120+0000","last_clean":"2024-03-21T14:21:03.829120+0000","last_became_active":"2024-03-21T14:19:14.695703+0000","last_became_peered":"2024-03-21T14:19:14.695703+0000","last_unstale":"2024-03-21T14:21:03.829120+0000","last_undegraded":"2024-03-21T14:21:03.829120+0000","last_fullsized":"2024-03-21T14:21:03.829120+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:37.939080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.939080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:47.422281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":818,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828880+0000","last_change":"2024-03-21T14:20:40.919011+0000","last_active":"2024-03-21T14:21:03.828880+0000","last_peered":"2024-03-21T14:21:03.828880+0000","last_clean":"2024-03-21T14:21:03.828880+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:21:03.828880+0000","last_undegraded":"2024-03-21T14:21:03.828880+0000","last_fullsized":"2024-03-21T14:21:03.828880+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:20:40.918895+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:20:40.918895+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:07.224584+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":631,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.312716+0000","last_change":"2024-03-21T14:19:34.691274+0000","last_active":"2024-03-21T14:21:03.312716+0000","last_peered":"2024-03-21T14:21:03.312716+0000","last_clean":"2024-03-21T14:21:03.312716+0000","last_became_active":"2024-03-21T14:19:34.690300+0000","last_became_peered":"2024-03-21T14:19:34.690300+0000","last_unstale":"2024-03-21T14:21:03.312716+0000","last_undegraded":"2024-03-21T14:21:03.312716+0000","last_fullsized":"2024-03-21T14:21:03.312716+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:23.464551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:23.464551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113322+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":770,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.312194+0000","last_change":"2024-03-21T14:20:29.536936+0000","last_active":"2024-03-21T14:21:03.312194+0000","last_peered":"2024-03-21T14:21:03.312194+0000","last_clean":"2024-03-21T14:21:03.312194+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:21:03.312194+0000","last_undegraded":"2024-03-21T14:21:03.312194+0000","last_fullsized":"2024-03-21T14:21:03.312194+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:29.536820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:20:29.536820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:29.770498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":570,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370586+0000","last_change":"2024-03-21T14:17:49.153687+0000","last_active":"2024-03-21T14:21:04.370586+0000","last_peered":"2024-03-21T14:21:04.370586+0000","last_clean":"2024-03-21T14:21:04.370586+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:21:04.370586+0000","last_undegraded":"2024-03-21T14:21:04.370586+0000","last_fullsized":"2024-03-21T14:21:04.370586+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:49.153572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:17:49.153572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":775,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370149+0000","last_change":"2024-03-21T14:17:21.179086+0000","last_active":"2024-03-21T14:21:04.370149+0000","last_peered":"2024-03-21T14:21:04.370149+0000","last_clean":"2024-03-21T14:21:04.370149+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:21:04.370149+0000","last_undegraded":"2024-03-21T14:21:04.370149+0000","last_fullsized":"2024-03-21T14:21:04.370149+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:21.178967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:21.178967+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370459+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":720,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313247+0000","last_change":"2024-03-21T14:20:21.776714+0000","last_active":"2024-03-21T14:21:03.313247+0000","last_peered":"2024-03-21T14:21:03.313247+0000","last_clean":"2024-03-21T14:21:03.313247+0000","last_became_active":"2024-03-21T14:20:21.776360+0000","last_became_peered":"2024-03-21T14:20:21.776360+0000","last_unstale":"2024-03-21T14:21:03.313247+0000","last_undegraded":"2024-03-21T14:21:03.313247+0000","last_fullsized":"2024-03-21T14:21:03.313247+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:18.436540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:18.436540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113308+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":771,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650112+0000","last_change":"2024-03-21T14:19:45.482626+0000","last_active":"2024-03-21T14:21:04.650112+0000","last_peered":"2024-03-21T14:21:04.650112+0000","last_clean":"2024-03-21T14:21:04.650112+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:21:04.650112+0000","last_undegraded":"2024-03-21T14:21:04.650112+0000","last_fullsized":"2024-03-21T14:21:04.650112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:45.482513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:19:45.482513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:10.547844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":236,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370162+0000","last_change":"2024-03-21T14:17:09.374978+0000","last_active":"2024-03-21T14:21:04.370162+0000","last_peered":"2024-03-21T14:21:04.370162+0000","last_clean":"2024-03-21T14:21:04.370162+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:21:04.370162+0000","last_undegraded":"2024-03-21T14:21:04.370162+0000","last_fullsized":"2024-03-21T14:21:04.370162+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:09.374855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:09.374855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370329+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":771,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370238+0000","last_change":"2024-03-21T14:18:04.289351+0000","last_active":"2024-03-21T14:21:04.370238+0000","last_peered":"2024-03-21T14:21:04.370238+0000","last_clean":"2024-03-21T14:21:04.370238+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:21:04.370238+0000","last_undegraded":"2024-03-21T14:21:04.370238+0000","last_fullsized":"2024-03-21T14:21:04.370238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:18:04.289228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:04.289228+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370524+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":750,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370707+0000","last_change":"2024-03-21T14:17:06.425215+0000","last_active":"2024-03-21T14:21:04.370707+0000","last_peered":"2024-03-21T14:21:04.370707+0000","last_clean":"2024-03-21T14:21:04.370707+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:21:04.370707+0000","last_undegraded":"2024-03-21T14:21:04.370707+0000","last_fullsized":"2024-03-21T14:21:04.370707+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:06.425104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:06.425104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":763,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314638+0000","last_change":"2024-03-21T14:18:34.187073+0000","last_active":"2024-03-21T14:21:03.314638+0000","last_peered":"2024-03-21T14:21:03.314638+0000","last_clean":"2024-03-21T14:21:03.314638+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:21:03.314638+0000","last_undegraded":"2024-03-21T14:21:03.314638+0000","last_fullsized":"2024-03-21T14:21:03.314638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:34.186897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:34.186897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":621,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370202+0000","last_change":"2024-03-21T14:19:34.686397+0000","last_active":"2024-03-21T14:21:04.370202+0000","last_peered":"2024-03-21T14:21:04.370202+0000","last_clean":"2024-03-21T14:21:04.370202+0000","last_became_active":"2024-03-21T14:19:34.686106+0000","last_became_peered":"2024-03-21T14:19:34.686106+0000","last_unstale":"2024-03-21T14:21:04.370202+0000","last_undegraded":"2024-03-21T14:21:04.370202+0000","last_fullsized":"2024-03-21T14:21:04.370202+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:33.422232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:18:33.422232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370604+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"236'27","reported_seq":669,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650603+0000","last_change":"2024-03-21T14:19:14.701974+0000","last_active":"2024-03-21T14:21:04.650603+0000","last_peered":"2024-03-21T14:21:04.650603+0000","last_clean":"2024-03-21T14:21:04.650603+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:21:04.650603+0000","last_undegraded":"2024-03-21T14:21:04.650603+0000","last_fullsized":"2024-03-21T14:21:04.650603+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:14:48.508532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:14:48.508532+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":766,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829362+0000","last_change":"2024-03-21T14:17:05.017847+0000","last_active":"2024-03-21T14:21:03.829362+0000","last_peered":"2024-03-21T14:21:03.829362+0000","last_clean":"2024-03-21T14:21:03.829362+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:21:03.829362+0000","last_undegraded":"2024-03-21T14:21:03.829362+0000","last_fullsized":"2024-03-21T14:21:03.829362+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.017731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.017731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813301+0000","scrub_duration":208,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":747,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370337+0000","last_change":"2024-03-21T14:19:14.691492+0000","last_active":"2024-03-21T14:21:04.370337+0000","last_peered":"2024-03-21T14:21:04.370337+0000","last_clean":"2024-03-21T14:21:04.370337+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:21:04.370337+0000","last_undegraded":"2024-03-21T14:21:04.370337+0000","last_fullsized":"2024-03-21T14:21:04.370337+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:00.679529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.679529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":763,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370208+0000","last_change":"2024-03-21T14:20:31.407178+0000","last_active":"2024-03-21T14:21:04.370208+0000","last_peered":"2024-03-21T14:21:04.370208+0000","last_clean":"2024-03-21T14:21:04.370208+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:21:04.370208+0000","last_undegraded":"2024-03-21T14:21:04.370208+0000","last_fullsized":"2024-03-21T14:21:04.370208+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.407065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.407065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:43.751635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":799,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829034+0000","last_change":"2024-03-21T14:18:03.889554+0000","last_active":"2024-03-21T14:21:03.829034+0000","last_peered":"2024-03-21T14:21:03.829034+0000","last_clean":"2024-03-21T14:21:03.829034+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:21:03.829034+0000","last_undegraded":"2024-03-21T14:21:03.829034+0000","last_fullsized":"2024-03-21T14:21:03.829034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:18:03.889428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:03.889428+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813429+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":270,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650248+0000","last_change":"2024-03-21T14:19:14.687146+0000","last_active":"2024-03-21T14:21:04.650248+0000","last_peered":"2024-03-21T14:21:04.650248+0000","last_clean":"2024-03-21T14:21:04.650248+0000","last_became_active":"2024-03-21T14:19:14.686523+0000","last_became_peered":"2024-03-21T14:19:14.686523+0000","last_unstale":"2024-03-21T14:21:04.650248+0000","last_undegraded":"2024-03-21T14:21:04.650248+0000","last_fullsized":"2024-03-21T14:21:04.650248+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:14:00.547775+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"360'14","reported_seq":369,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370762+0000","last_change":"2024-03-21T14:19:15.820967+0000","last_active":"2024-03-21T14:21:04.370762+0000","last_peered":"2024-03-21T14:21:04.370762+0000","last_clean":"2024-03-21T14:21:04.370762+0000","last_became_active":"2024-03-21T14:19:15.688331+0000","last_became_peered":"2024-03-21T14:19:15.688331+0000","last_unstale":"2024-03-21T14:21:04.370762+0000","last_undegraded":"2024-03-21T14:21:04.370762+0000","last_fullsized":"2024-03-21T14:21:04.370762+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"298'10","last_scrub_stamp":"2024-03-21T14:16:59.343213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:59.343213+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":788,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313335+0000","last_change":"2024-03-21T14:20:37.151817+0000","last_active":"2024-03-21T14:21:03.313335+0000","last_peered":"2024-03-21T14:21:03.313335+0000","last_clean":"2024-03-21T14:21:03.313335+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:21:03.313335+0000","last_undegraded":"2024-03-21T14:21:03.313335+0000","last_fullsized":"2024-03-21T14:21:03.313335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:20:37.151721+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.151721+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:49.468465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":485,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650845+0000","last_change":"2024-03-21T14:20:21.799582+0000","last_active":"2024-03-21T14:21:04.650845+0000","last_peered":"2024-03-21T14:21:04.650845+0000","last_clean":"2024-03-21T14:21:04.650845+0000","last_became_active":"2024-03-21T14:20:21.790051+0000","last_became_peered":"2024-03-21T14:20:21.790051+0000","last_unstale":"2024-03-21T14:21:04.650845+0000","last_undegraded":"2024-03-21T14:21:04.650845+0000","last_fullsized":"2024-03-21T14:21:04.650845+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:19:21.606220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:21.606220+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":759,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314250+0000","last_change":"2024-03-21T14:17:58.261975+0000","last_active":"2024-03-21T14:21:03.314250+0000","last_peered":"2024-03-21T14:21:03.314250+0000","last_clean":"2024-03-21T14:21:03.314250+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:21:03.314250+0000","last_undegraded":"2024-03-21T14:21:03.314250+0000","last_fullsized":"2024-03-21T14:21:03.314250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:58.261849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:58.261849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113033+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":427,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650748+0000","last_change":"2024-03-21T14:20:49.417668+0000","last_active":"2024-03-21T14:21:04.650748+0000","last_peered":"2024-03-21T14:21:04.650748+0000","last_clean":"2024-03-21T14:21:04.650748+0000","last_became_active":"2024-03-21T14:19:14.675463+0000","last_became_peered":"2024-03-21T14:19:14.675463+0000","last_unstale":"2024-03-21T14:21:04.650748+0000","last_undegraded":"2024-03-21T14:21:04.650748+0000","last_fullsized":"2024-03-21T14:21:04.650748+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:49.417593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.417593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:15.841246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":792,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828527+0000","last_change":"2024-03-21T14:20:45.936416+0000","last_active":"2024-03-21T14:21:03.828527+0000","last_peered":"2024-03-21T14:21:03.828527+0000","last_clean":"2024-03-21T14:21:03.828527+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:21:03.828527+0000","last_undegraded":"2024-03-21T14:21:03.828527+0000","last_fullsized":"2024-03-21T14:21:03.828527+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.936318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.936318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:14.932353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":503,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314712+0000","last_change":"2024-03-21T14:20:46.036522+0000","last_active":"2024-03-21T14:21:03.314712+0000","last_peered":"2024-03-21T14:21:03.314712+0000","last_clean":"2024-03-21T14:21:03.314712+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:21:03.314712+0000","last_undegraded":"2024-03-21T14:21:03.314712+0000","last_fullsized":"2024-03-21T14:21:03.314712+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.036377+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:08.476100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":125,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650362+0000","last_change":"2024-03-21T14:19:14.680267+0000","last_active":"2024-03-21T14:21:04.650362+0000","last_peered":"2024-03-21T14:21:04.650362+0000","last_clean":"2024-03-21T14:21:04.650362+0000","last_became_active":"2024-03-21T14:19:14.679790+0000","last_became_peered":"2024-03-21T14:19:14.679790+0000","last_unstale":"2024-03-21T14:21:04.650362+0000","last_undegraded":"2024-03-21T14:21:04.650362+0000","last_fullsized":"2024-03-21T14:21:04.650362+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:49.859204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:49.859204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":444,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369811+0000","last_change":"2024-03-21T14:18:42.513377+0000","last_active":"2024-03-21T14:21:04.369811+0000","last_peered":"2024-03-21T14:21:04.369811+0000","last_clean":"2024-03-21T14:21:04.369811+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:21:04.369811+0000","last_undegraded":"2024-03-21T14:21:04.369811+0000","last_fullsized":"2024-03-21T14:21:04.369811+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.513167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370635+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":238,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829526+0000","last_change":"2024-03-21T14:16:56.847010+0000","last_active":"2024-03-21T14:21:03.829526+0000","last_peered":"2024-03-21T14:21:03.829526+0000","last_clean":"2024-03-21T14:21:03.829526+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:21:03.829526+0000","last_undegraded":"2024-03-21T14:21:03.829526+0000","last_fullsized":"2024-03-21T14:21:03.829526+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:56.846916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:56.846916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"275'10","reported_seq":801,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828653+0000","last_change":"2024-03-21T14:18:45.023742+0000","last_active":"2024-03-21T14:21:03.828653+0000","last_peered":"2024-03-21T14:21:03.828653+0000","last_clean":"2024-03-21T14:21:03.828653+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:21:03.828653+0000","last_undegraded":"2024-03-21T14:21:03.828653+0000","last_fullsized":"2024-03-21T14:21:03.828653+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'10","last_scrub_stamp":"2024-03-21T14:18:45.023596+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:18:45.023596+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813528+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":783,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829101+0000","last_change":"2024-03-21T14:19:34.693613+0000","last_active":"2024-03-21T14:21:03.829101+0000","last_peered":"2024-03-21T14:21:03.829101+0000","last_clean":"2024-03-21T14:21:03.829101+0000","last_became_active":"2024-03-21T14:19:34.693279+0000","last_became_peered":"2024-03-21T14:19:34.693279+0000","last_unstale":"2024-03-21T14:21:03.829101+0000","last_undegraded":"2024-03-21T14:21:03.829101+0000","last_fullsized":"2024-03-21T14:21:03.829101+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:07.939188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:17:07.939188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813316+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":793,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649612+0000","last_change":"2024-03-21T14:20:30.520958+0000","last_active":"2024-03-21T14:21:04.649612+0000","last_peered":"2024-03-21T14:21:04.649612+0000","last_clean":"2024-03-21T14:21:04.649612+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:21:04.649612+0000","last_undegraded":"2024-03-21T14:21:04.649612+0000","last_fullsized":"2024-03-21T14:21:04.649612+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:30.520827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.520827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:42.586304+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":241,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314653+0000","last_change":"2024-03-21T14:19:42.258049+0000","last_active":"2024-03-21T14:21:03.314653+0000","last_peered":"2024-03-21T14:21:03.314653+0000","last_clean":"2024-03-21T14:21:03.314653+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:21:03.314653+0000","last_undegraded":"2024-03-21T14:21:03.314653+0000","last_fullsized":"2024-03-21T14:21:03.314653+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:19:42.257908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:19:42.257908+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:09.908704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":829,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828854+0000","last_change":"2024-03-21T14:18:42.061939+0000","last_active":"2024-03-21T14:21:03.828854+0000","last_peered":"2024-03-21T14:21:03.828854+0000","last_clean":"2024-03-21T14:21:03.828854+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:21:03.828854+0000","last_undegraded":"2024-03-21T14:21:03.828854+0000","last_fullsized":"2024-03-21T14:21:03.828854+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:18:42.061791+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.061791+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813545+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":248,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.312592+0000","last_change":"2024-03-21T14:19:50.405106+0000","last_active":"2024-03-21T14:21:03.312592+0000","last_peered":"2024-03-21T14:21:03.312592+0000","last_clean":"2024-03-21T14:21:03.312592+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:21:03.312592+0000","last_undegraded":"2024-03-21T14:21:03.312592+0000","last_fullsized":"2024-03-21T14:21:03.312592+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:50.404999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:50.404999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113355+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":763,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828972+0000","last_change":"2024-03-21T14:17:19.983621+0000","last_active":"2024-03-21T14:21:03.828972+0000","last_peered":"2024-03-21T14:21:03.828972+0000","last_clean":"2024-03-21T14:21:03.828972+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:21:03.828972+0000","last_undegraded":"2024-03-21T14:21:03.828972+0000","last_fullsized":"2024-03-21T14:21:03.828972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:19.983497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:17:19.983497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":787,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650484+0000","last_change":"2024-03-21T14:20:45.456427+0000","last_active":"2024-03-21T14:21:04.650484+0000","last_peered":"2024-03-21T14:21:04.650484+0000","last_clean":"2024-03-21T14:21:04.650484+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:21:04.650484+0000","last_undegraded":"2024-03-21T14:21:04.650484+0000","last_fullsized":"2024-03-21T14:21:04.650484+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.456309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.456309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.470305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":817,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313860+0000","last_change":"2024-03-21T14:19:25.467111+0000","last_active":"2024-03-21T14:21:03.313860+0000","last_peered":"2024-03-21T14:21:03.313860+0000","last_clean":"2024-03-21T14:21:03.313860+0000","last_became_active":"2024-03-21T14:19:14.682980+0000","last_became_peered":"2024-03-21T14:19:14.682980+0000","last_unstale":"2024-03-21T14:21:03.313860+0000","last_undegraded":"2024-03-21T14:21:03.313860+0000","last_fullsized":"2024-03-21T14:21:03.313860+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:19:25.465595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:25.465595+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113339+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":63,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828609+0000","last_change":"2024-03-21T14:20:42.950594+0000","last_active":"2024-03-21T14:21:03.828609+0000","last_peered":"2024-03-21T14:21:03.828609+0000","last_clean":"2024-03-21T14:21:03.828609+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:21:03.828609+0000","last_undegraded":"2024-03-21T14:21:03.828609+0000","last_fullsized":"2024-03-21T14:21:03.828609+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.950500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.950500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.148493+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":783,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313755+0000","last_change":"2024-03-21T14:20:21.778599+0000","last_active":"2024-03-21T14:21:03.313755+0000","last_peered":"2024-03-21T14:21:03.313755+0000","last_clean":"2024-03-21T14:21:03.313755+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:21:03.313755+0000","last_undegraded":"2024-03-21T14:21:03.313755+0000","last_fullsized":"2024-03-21T14:21:03.313755+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:57.200762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:57.200762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113067+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":777,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370224+0000","last_change":"2024-03-21T14:19:14.691071+0000","last_active":"2024-03-21T14:21:04.370224+0000","last_peered":"2024-03-21T14:21:04.370224+0000","last_clean":"2024-03-21T14:21:04.370224+0000","last_became_active":"2024-03-21T14:19:14.690097+0000","last_became_peered":"2024-03-21T14:19:14.690097+0000","last_unstale":"2024-03-21T14:21:04.370224+0000","last_undegraded":"2024-03-21T14:21:04.370224+0000","last_fullsized":"2024-03-21T14:21:04.370224+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:47.020235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:14:47.020235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":838,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370278+0000","last_change":"2024-03-21T14:19:15.685722+0000","last_active":"2024-03-21T14:21:04.370278+0000","last_peered":"2024-03-21T14:21:04.370278+0000","last_clean":"2024-03-21T14:21:04.370278+0000","last_became_active":"2024-03-21T14:19:15.685220+0000","last_became_peered":"2024-03-21T14:19:15.685220+0000","last_unstale":"2024-03-21T14:21:04.370278+0000","last_undegraded":"2024-03-21T14:21:04.370278+0000","last_fullsized":"2024-03-21T14:21:04.370278+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:16:13.416403+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:16:13.416403+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":644,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650646+0000","last_change":"2024-03-21T14:20:21.784715+0000","last_active":"2024-03-21T14:21:04.650646+0000","last_peered":"2024-03-21T14:21:04.650646+0000","last_clean":"2024-03-21T14:21:04.650646+0000","last_became_active":"2024-03-21T14:20:21.784416+0000","last_became_peered":"2024-03-21T14:20:21.784416+0000","last_unstale":"2024-03-21T14:21:04.650646+0000","last_undegraded":"2024-03-21T14:21:04.650646+0000","last_fullsized":"2024-03-21T14:21:04.650646+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:49.467034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:49.467034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:06.681753+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":967,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314494+0000","last_change":"2024-03-21T14:18:37.151442+0000","last_active":"2024-03-21T14:21:03.314494+0000","last_peered":"2024-03-21T14:21:03.314494+0000","last_clean":"2024-03-21T14:21:03.314494+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:21:03.314494+0000","last_undegraded":"2024-03-21T14:21:03.314494+0000","last_fullsized":"2024-03-21T14:21:03.314494+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'10","last_scrub_stamp":"2024-03-21T14:18:37.151302+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:18:37.151302+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113152+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":781,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650144+0000","last_change":"2024-03-21T14:19:22.612161+0000","last_active":"2024-03-21T14:21:04.650144+0000","last_peered":"2024-03-21T14:21:04.650144+0000","last_clean":"2024-03-21T14:21:04.650144+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:21:04.650144+0000","last_undegraded":"2024-03-21T14:21:04.650144+0000","last_fullsized":"2024-03-21T14:21:04.650144+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:22.612041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:19:22.612041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":794,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829196+0000","last_change":"2024-03-21T14:16:12.543045+0000","last_active":"2024-03-21T14:21:03.829196+0000","last_peered":"2024-03-21T14:21:03.829196+0000","last_clean":"2024-03-21T14:21:03.829196+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:21:03.829196+0000","last_undegraded":"2024-03-21T14:21:03.829196+0000","last_fullsized":"2024-03-21T14:21:03.829196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:16:12.542909+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:16:12.542909+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813233+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":751,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370676+0000","last_change":"2024-03-21T14:20:38.602527+0000","last_active":"2024-03-21T14:21:04.370676+0000","last_peered":"2024-03-21T14:21:04.370676+0000","last_clean":"2024-03-21T14:21:04.370676+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:21:04.370676+0000","last_undegraded":"2024-03-21T14:21:04.370676+0000","last_fullsized":"2024-03-21T14:21:04.370676+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:38.602405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:20:38.602405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:02.464900+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":76,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829272+0000","last_change":"2024-03-21T14:20:44.917571+0000","last_active":"2024-03-21T14:21:03.829272+0000","last_peered":"2024-03-21T14:21:03.829272+0000","last_clean":"2024-03-21T14:21:03.829272+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:21:03.829272+0000","last_undegraded":"2024-03-21T14:21:03.829272+0000","last_fullsized":"2024-03-21T14:21:03.829272+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:20:44.917480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.917480+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:55.408449+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":55,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828449+0000","last_change":"2024-03-21T14:20:21.784789+0000","last_active":"2024-03-21T14:21:03.828449+0000","last_peered":"2024-03-21T14:21:03.828449+0000","last_clean":"2024-03-21T14:21:03.828449+0000","last_became_active":"2024-03-21T14:20:21.784349+0000","last_became_peered":"2024-03-21T14:20:21.784349+0000","last_unstale":"2024-03-21T14:21:03.828449+0000","last_undegraded":"2024-03-21T14:21:03.828449+0000","last_fullsized":"2024-03-21T14:21:03.828449+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:40.140340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:18:40.140340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813494+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"338'19","reported_seq":655,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650655+0000","last_change":"2024-03-21T14:19:43.545025+0000","last_active":"2024-03-21T14:21:04.650655+0000","last_peered":"2024-03-21T14:21:04.650655+0000","last_clean":"2024-03-21T14:21:04.650655+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:21:04.650655+0000","last_undegraded":"2024-03-21T14:21:04.650655+0000","last_fullsized":"2024-03-21T14:21:04.650655+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'19","last_scrub_stamp":"2024-03-21T14:19:43.544784+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:19:43.544784+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":600,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650885+0000","last_change":"2024-03-21T14:19:14.699974+0000","last_active":"2024-03-21T14:21:04.650885+0000","last_peered":"2024-03-21T14:21:04.650885+0000","last_clean":"2024-03-21T14:21:04.650885+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:21:04.650885+0000","last_undegraded":"2024-03-21T14:21:04.650885+0000","last_fullsized":"2024-03-21T14:21:04.650885+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:43.502435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:43.502435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":55,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829761+0000","last_change":"2024-03-21T14:20:21.787792+0000","last_active":"2024-03-21T14:21:03.829761+0000","last_peered":"2024-03-21T14:21:03.829761+0000","last_clean":"2024-03-21T14:21:03.829761+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:21:03.829761+0000","last_undegraded":"2024-03-21T14:21:03.829761+0000","last_fullsized":"2024-03-21T14:21:03.829761+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.399758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:08.885507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":749,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370041+0000","last_change":"2024-03-21T14:17:05.374072+0000","last_active":"2024-03-21T14:21:04.370041+0000","last_peered":"2024-03-21T14:21:04.370041+0000","last_clean":"2024-03-21T14:21:04.370041+0000","last_became_active":"2024-03-21T14:16:34.616172+0000","last_became_peered":"2024-03-21T14:16:34.616172+0000","last_unstale":"2024-03-21T14:21:04.370041+0000","last_undegraded":"2024-03-21T14:21:04.370041+0000","last_fullsized":"2024-03-21T14:21:04.370041+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.373895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.373895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370428+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"236'15","reported_seq":460,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829243+0000","last_change":"2024-03-21T14:17:16.885851+0000","last_active":"2024-03-21T14:21:03.829243+0000","last_peered":"2024-03-21T14:21:03.829243+0000","last_clean":"2024-03-21T14:21:03.829243+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:21:03.829243+0000","last_undegraded":"2024-03-21T14:21:03.829243+0000","last_fullsized":"2024-03-21T14:21:03.829243+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:16.885709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.885709+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813381+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":781,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649850+0000","last_change":"2024-03-21T14:20:34.570950+0000","last_active":"2024-03-21T14:21:04.649850+0000","last_peered":"2024-03-21T14:21:04.649850+0000","last_clean":"2024-03-21T14:21:04.649850+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:21:04.649850+0000","last_undegraded":"2024-03-21T14:21:04.649850+0000","last_fullsized":"2024-03-21T14:21:04.649850+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:34.570712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:34.570712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:43.732723+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":237,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829233+0000","last_change":"2024-03-21T14:17:03.781272+0000","last_active":"2024-03-21T14:21:03.829233+0000","last_peered":"2024-03-21T14:21:03.829233+0000","last_clean":"2024-03-21T14:21:03.829233+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:21:03.829233+0000","last_undegraded":"2024-03-21T14:21:03.829233+0000","last_fullsized":"2024-03-21T14:21:03.829233+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:03.781169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.781169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813284+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":422,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828416+0000","last_change":"2024-03-21T14:19:34.691021+0000","last_active":"2024-03-21T14:21:03.828416+0000","last_peered":"2024-03-21T14:21:03.828416+0000","last_clean":"2024-03-21T14:21:03.828416+0000","last_became_active":"2024-03-21T14:19:34.690247+0000","last_became_peered":"2024-03-21T14:19:34.690247+0000","last_unstale":"2024-03-21T14:21:03.828416+0000","last_undegraded":"2024-03-21T14:21:03.828416+0000","last_fullsized":"2024-03-21T14:21:03.828416+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:07.829823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:19:07.829823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":63,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829013+0000","last_change":"2024-03-21T14:20:21.796765+0000","last_active":"2024-03-21T14:21:03.829013+0000","last_peered":"2024-03-21T14:21:03.829013+0000","last_clean":"2024-03-21T14:21:03.829013+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:21:03.829013+0000","last_undegraded":"2024-03-21T14:21:03.829013+0000","last_fullsized":"2024-03-21T14:21:03.829013+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:19:32.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:32.302574+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:06.774275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314301+0000","last_change":"2024-03-21T14:20:59.387014+0000","last_active":"2024-03-21T14:21:03.314301+0000","last_peered":"2024-03-21T14:21:03.314301+0000","last_clean":"2024-03-21T14:21:03.314301+0000","last_became_active":"2024-03-21T14:20:59.386663+0000","last_became_peered":"2024-03-21T14:20:59.386663+0000","last_unstale":"2024-03-21T14:21:03.314301+0000","last_undegraded":"2024-03-21T14:21:03.314301+0000","last_fullsized":"2024-03-21T14:21:03.314301+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:09.657642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":645,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313342+0000","last_change":"2024-03-21T14:19:11.368761+0000","last_active":"2024-03-21T14:21:03.313342+0000","last_peered":"2024-03-21T14:21:03.313342+0000","last_clean":"2024-03-21T14:21:03.313342+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:21:03.313342+0000","last_undegraded":"2024-03-21T14:21:03.313342+0000","last_fullsized":"2024-03-21T14:21:03.313342+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:11.368572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:19:11.368572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113276+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":818,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650947+0000","last_change":"2024-03-21T14:19:42.441128+0000","last_active":"2024-03-21T14:21:04.650947+0000","last_peered":"2024-03-21T14:21:04.650947+0000","last_clean":"2024-03-21T14:21:04.650947+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:21:04.650947+0000","last_undegraded":"2024-03-21T14:21:04.650947+0000","last_fullsized":"2024-03-21T14:21:04.650947+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_clean_scrub_stamp":"2024-03-21T14:19:42.440997+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":700,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650151+0000","last_change":"2024-03-21T14:20:39.417922+0000","last_active":"2024-03-21T14:21:04.650151+0000","last_peered":"2024-03-21T14:21:04.650151+0000","last_clean":"2024-03-21T14:21:04.650151+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:21:04.650151+0000","last_undegraded":"2024-03-21T14:21:04.650151+0000","last_fullsized":"2024-03-21T14:21:04.650151+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:20:39.417781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:39.417781+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:43.468791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"372'10","reported_seq":1014,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649624+0000","last_change":"2024-03-21T14:19:47.479886+0000","last_active":"2024-03-21T14:21:04.649624+0000","last_peered":"2024-03-21T14:21:04.649624+0000","last_clean":"2024-03-21T14:21:04.649624+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:21:04.649624+0000","last_undegraded":"2024-03-21T14:21:04.649624+0000","last_fullsized":"2024-03-21T14:21:04.649624+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.479770+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.651083+0000","last_change":"2024-03-21T14:20:59.242772+0000","last_active":"2024-03-21T14:21:04.651083+0000","last_peered":"2024-03-21T14:21:04.651083+0000","last_clean":"2024-03-21T14:21:04.651083+0000","last_became_active":"2024-03-21T14:20:59.241791+0000","last_became_peered":"2024-03-21T14:20:59.241791+0000","last_unstale":"2024-03-21T14:21:04.651083+0000","last_undegraded":"2024-03-21T14:21:04.651083+0000","last_fullsized":"2024-03-21T14:21:04.651083+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:58.369144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":505,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828730+0000","last_change":"2024-03-21T14:18:26.997703+0000","last_active":"2024-03-21T14:21:03.828730+0000","last_peered":"2024-03-21T14:21:03.828730+0000","last_clean":"2024-03-21T14:21:03.828730+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:21:03.828730+0000","last_undegraded":"2024-03-21T14:21:03.828730+0000","last_fullsized":"2024-03-21T14:21:03.828730+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:26.997615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:26.997615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813478+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":796,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650742+0000","last_change":"2024-03-21T14:20:31.503763+0000","last_active":"2024-03-21T14:21:04.650742+0000","last_peered":"2024-03-21T14:21:04.650742+0000","last_clean":"2024-03-21T14:21:04.650742+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:21:04.650742+0000","last_undegraded":"2024-03-21T14:21:04.650742+0000","last_fullsized":"2024-03-21T14:21:04.650742+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.503687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.503687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:35.612376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":845,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370147+0000","last_change":"2024-03-21T14:20:21.797847+0000","last_active":"2024-03-21T14:21:04.370147+0000","last_peered":"2024-03-21T14:21:04.370147+0000","last_clean":"2024-03-21T14:21:04.370147+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:21:04.370147+0000","last_undegraded":"2024-03-21T14:21:04.370147+0000","last_fullsized":"2024-03-21T14:21:04.370147+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:18:29.432719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.432719+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370590+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":612,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650012+0000","last_change":"2024-03-21T14:19:14.699748+0000","last_active":"2024-03-21T14:21:04.650012+0000","last_peered":"2024-03-21T14:21:04.650012+0000","last_clean":"2024-03-21T14:21:04.650012+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:21:04.650012+0000","last_undegraded":"2024-03-21T14:21:04.650012+0000","last_fullsized":"2024-03-21T14:21:04.650012+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:39.425147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:14:39.425147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829253+0000","last_change":"2024-03-21T14:20:59.387129+0000","last_active":"2024-03-21T14:21:03.829253+0000","last_peered":"2024-03-21T14:21:03.829253+0000","last_clean":"2024-03-21T14:21:03.829253+0000","last_became_active":"2024-03-21T14:20:59.386704+0000","last_became_peered":"2024-03-21T14:20:59.386704+0000","last_unstale":"2024-03-21T14:21:03.829253+0000","last_undegraded":"2024-03-21T14:21:03.829253+0000","last_fullsized":"2024-03-21T14:21:03.829253+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:22.379363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":133,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649987+0000","last_change":"2024-03-21T14:20:44.543695+0000","last_active":"2024-03-21T14:21:04.649987+0000","last_peered":"2024-03-21T14:21:04.649987+0000","last_clean":"2024-03-21T14:21:04.649987+0000","last_became_active":"2024-03-21T14:19:14.700785+0000","last_became_peered":"2024-03-21T14:19:14.700785+0000","last_unstale":"2024-03-21T14:21:04.649987+0000","last_undegraded":"2024-03-21T14:21:04.649987+0000","last_fullsized":"2024-03-21T14:21:04.649987+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:44.543611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.543611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:50.060454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":573,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650796+0000","last_change":"2024-03-21T14:20:51.545141+0000","last_active":"2024-03-21T14:21:04.650796+0000","last_peered":"2024-03-21T14:21:04.650796+0000","last_clean":"2024-03-21T14:21:04.650796+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:21:04.650796+0000","last_undegraded":"2024-03-21T14:21:04.650796+0000","last_fullsized":"2024-03-21T14:21:04.650796+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:51.545057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.545057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:54.778605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":750,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829346+0000","last_change":"2024-03-21T14:17:54.944166+0000","last_active":"2024-03-21T14:21:03.829346+0000","last_peered":"2024-03-21T14:21:03.829346+0000","last_clean":"2024-03-21T14:21:03.829346+0000","last_became_active":"2024-03-21T14:16:34.618652+0000","last_became_peered":"2024-03-21T14:16:34.618652+0000","last_unstale":"2024-03-21T14:21:03.829346+0000","last_undegraded":"2024-03-21T14:21:03.829346+0000","last_fullsized":"2024-03-21T14:21:03.829346+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:54.944088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:54.944088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":729,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313618+0000","last_change":"2024-03-21T14:20:49.145974+0000","last_active":"2024-03-21T14:21:03.313618+0000","last_peered":"2024-03-21T14:21:03.313618+0000","last_clean":"2024-03-21T14:21:03.313618+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:21:03.313618+0000","last_undegraded":"2024-03-21T14:21:03.313618+0000","last_fullsized":"2024-03-21T14:21:03.313618+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.145792+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:53.951598+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650454+0000","last_change":"2024-03-21T14:20:59.109491+0000","last_active":"2024-03-21T14:21:04.650454+0000","last_peered":"2024-03-21T14:21:04.650454+0000","last_clean":"2024-03-21T14:21:04.650454+0000","last_became_active":"2024-03-21T14:20:59.109188+0000","last_became_peered":"2024-03-21T14:20:59.109188+0000","last_unstale":"2024-03-21T14:21:04.650454+0000","last_undegraded":"2024-03-21T14:21:04.650454+0000","last_fullsized":"2024-03-21T14:21:04.650454+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:20.048059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":132,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649931+0000","last_change":"2024-03-21T14:19:34.691261+0000","last_active":"2024-03-21T14:21:04.649931+0000","last_peered":"2024-03-21T14:21:04.649931+0000","last_clean":"2024-03-21T14:21:04.649931+0000","last_became_active":"2024-03-21T14:19:34.690841+0000","last_became_peered":"2024-03-21T14:19:34.690841+0000","last_unstale":"2024-03-21T14:21:04.649931+0000","last_undegraded":"2024-03-21T14:21:04.649931+0000","last_fullsized":"2024-03-21T14:21:04.649931+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.865488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.865488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":612,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650661+0000","last_change":"2024-03-21T14:19:14.692130+0000","last_active":"2024-03-21T14:21:04.650661+0000","last_peered":"2024-03-21T14:21:04.650661+0000","last_clean":"2024-03-21T14:21:04.650661+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:21:04.650661+0000","last_undegraded":"2024-03-21T14:21:04.650661+0000","last_fullsized":"2024-03-21T14:21:04.650661+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.532653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.532653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":572,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829062+0000","last_change":"2024-03-21T14:18:40.092328+0000","last_active":"2024-03-21T14:21:03.829062+0000","last_peered":"2024-03-21T14:21:03.829062+0000","last_clean":"2024-03-21T14:21:03.829062+0000","last_became_active":"2024-03-21T14:16:21.865349+0000","last_became_peered":"2024-03-21T14:16:21.865349+0000","last_unstale":"2024-03-21T14:21:03.829062+0000","last_undegraded":"2024-03-21T14:21:03.829062+0000","last_fullsized":"2024-03-21T14:21:03.829062+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:40.092182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:18:40.092182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":267,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650953+0000","last_change":"2024-03-21T14:19:14.692833+0000","last_active":"2024-03-21T14:21:04.650953+0000","last_peered":"2024-03-21T14:21:04.650953+0000","last_clean":"2024-03-21T14:21:04.650953+0000","last_became_active":"2024-03-21T14:19:14.692456+0000","last_became_peered":"2024-03-21T14:19:14.692456+0000","last_unstale":"2024-03-21T14:21:04.650953+0000","last_undegraded":"2024-03-21T14:21:04.650953+0000","last_fullsized":"2024-03-21T14:21:04.650953+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:13:50.159386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:13:50.159386+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828476+0000","last_change":"2024-03-21T14:20:59.241783+0000","last_active":"2024-03-21T14:21:03.828476+0000","last_peered":"2024-03-21T14:21:03.828476+0000","last_clean":"2024-03-21T14:21:03.828476+0000","last_became_active":"2024-03-21T14:20:59.241477+0000","last_became_peered":"2024-03-21T14:20:59.241477+0000","last_unstale":"2024-03-21T14:21:03.828476+0000","last_undegraded":"2024-03-21T14:21:03.828476+0000","last_fullsized":"2024-03-21T14:21:03.828476+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.067004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":748,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650159+0000","last_change":"2024-03-21T14:20:35.508783+0000","last_active":"2024-03-21T14:21:04.650159+0000","last_peered":"2024-03-21T14:21:04.650159+0000","last_clean":"2024-03-21T14:21:04.650159+0000","last_became_active":"2024-03-21T14:19:14.689666+0000","last_became_peered":"2024-03-21T14:19:14.689666+0000","last_unstale":"2024-03-21T14:21:04.650159+0000","last_undegraded":"2024-03-21T14:21:04.650159+0000","last_fullsized":"2024-03-21T14:21:04.650159+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:35.508660+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:35.508660+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.775681+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":760,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829343+0000","last_change":"2024-03-21T14:20:50.816435+0000","last_active":"2024-03-21T14:21:03.829343+0000","last_peered":"2024-03-21T14:21:03.829343+0000","last_clean":"2024-03-21T14:21:03.829343+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:21:03.829343+0000","last_undegraded":"2024-03-21T14:21:03.829343+0000","last_fullsized":"2024-03-21T14:21:03.829343+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:50.816334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:50.816334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:03.549011+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":272,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313465+0000","last_change":"2024-03-21T14:19:14.319436+0000","last_active":"2024-03-21T14:21:03.313465+0000","last_peered":"2024-03-21T14:21:03.313465+0000","last_clean":"2024-03-21T14:21:03.313465+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:21:03.313465+0000","last_undegraded":"2024-03-21T14:21:03.313465+0000","last_fullsized":"2024-03-21T14:21:03.313465+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:19:14.319346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:19:14.319346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":761,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370110+0000","last_change":"2024-03-21T14:16:57.382377+0000","last_active":"2024-03-21T14:21:04.370110+0000","last_peered":"2024-03-21T14:21:04.370110+0000","last_clean":"2024-03-21T14:21:04.370110+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:21:04.370110+0000","last_undegraded":"2024-03-21T14:21:04.370110+0000","last_fullsized":"2024-03-21T14:21:04.370110+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.382291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.382291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"360'54","reported_seq":688,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650374+0000","last_change":"2024-03-21T14:20:31.540383+0000","last_active":"2024-03-21T14:21:04.650374+0000","last_peered":"2024-03-21T14:21:04.650374+0000","last_clean":"2024-03-21T14:21:04.650374+0000","last_became_active":"2024-03-21T14:19:14.693616+0000","last_became_peered":"2024-03-21T14:19:14.693616+0000","last_unstale":"2024-03-21T14:21:04.650374+0000","last_undegraded":"2024-03-21T14:21:04.650374+0000","last_fullsized":"2024-03-21T14:21:04.650374+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'54","last_scrub_stamp":"2024-03-21T14:20:31.540283+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.540283+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:40.298858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":638,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314660+0000","last_change":"2024-03-21T14:19:34.691493+0000","last_active":"2024-03-21T14:21:03.314660+0000","last_peered":"2024-03-21T14:21:03.314660+0000","last_clean":"2024-03-21T14:21:03.314660+0000","last_became_active":"2024-03-21T14:19:34.690636+0000","last_became_peered":"2024-03-21T14:19:34.690636+0000","last_unstale":"2024-03-21T14:21:03.314660+0000","last_undegraded":"2024-03-21T14:21:03.314660+0000","last_fullsized":"2024-03-21T14:21:03.314660+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:01.122304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:01.122304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113051+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828533+0000","last_change":"2024-03-21T14:20:59.387006+0000","last_active":"2024-03-21T14:21:03.828533+0000","last_peered":"2024-03-21T14:21:03.828533+0000","last_clean":"2024-03-21T14:21:03.828533+0000","last_became_active":"2024-03-21T14:20:59.386643+0000","last_became_peered":"2024-03-21T14:20:59.386643+0000","last_unstale":"2024-03-21T14:21:03.828533+0000","last_undegraded":"2024-03-21T14:21:03.828533+0000","last_fullsized":"2024-03-21T14:21:03.828533+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:00.888523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":986,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369952+0000","last_change":"2024-03-21T14:16:58.345839+0000","last_active":"2024-03-21T14:21:04.369952+0000","last_peered":"2024-03-21T14:21:04.369952+0000","last_clean":"2024-03-21T14:21:04.369952+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:21:04.369952+0000","last_undegraded":"2024-03-21T14:21:04.369952+0000","last_fullsized":"2024-03-21T14:21:04.369952+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:16:58.345739+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:16:58.345739+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":775,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650649+0000","last_change":"2024-03-21T14:20:51.457667+0000","last_active":"2024-03-21T14:21:04.650649+0000","last_peered":"2024-03-21T14:21:04.650649+0000","last_clean":"2024-03-21T14:21:04.650649+0000","last_became_active":"2024-03-21T14:19:14.685026+0000","last_became_peered":"2024-03-21T14:19:14.685026+0000","last_unstale":"2024-03-21T14:21:04.650649+0000","last_undegraded":"2024-03-21T14:21:04.650649+0000","last_fullsized":"2024-03-21T14:21:04.650649+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:51.457549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.457549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:18.394890+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":780,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.312989+0000","last_change":"2024-03-21T14:19:17.321024+0000","last_active":"2024-03-21T14:21:03.312989+0000","last_peered":"2024-03-21T14:21:03.312989+0000","last_clean":"2024-03-21T14:21:03.312989+0000","last_became_active":"2024-03-21T14:19:14.697007+0000","last_became_peered":"2024-03-21T14:19:14.697007+0000","last_unstale":"2024-03-21T14:21:03.312989+0000","last_undegraded":"2024-03-21T14:21:03.312989+0000","last_fullsized":"2024-03-21T14:21:03.312989+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:17.320902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:19:17.320902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113229+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":615,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649891+0000","last_change":"2024-03-21T14:20:48.551092+0000","last_active":"2024-03-21T14:21:04.649891+0000","last_peered":"2024-03-21T14:21:04.649891+0000","last_clean":"2024-03-21T14:21:04.649891+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:21:04.649891+0000","last_undegraded":"2024-03-21T14:21:04.649891+0000","last_fullsized":"2024-03-21T14:21:04.649891+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.550975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.550975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:10.129850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":858,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314525+0000","last_change":"2024-03-21T14:19:34.717581+0000","last_active":"2024-03-21T14:21:03.314525+0000","last_peered":"2024-03-21T14:21:03.314525+0000","last_clean":"2024-03-21T14:21:03.314525+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:21:03.314525+0000","last_undegraded":"2024-03-21T14:21:03.314525+0000","last_fullsized":"2024-03-21T14:21:03.314525+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:18:41.125165+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-21T14:03:39.830542+0000","last_clean_scrub_stamp":"2024-03-21T14:18:41.125165+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113136+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":590,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.651302+0000","last_change":"2024-03-21T14:19:14.683429+0000","last_active":"2024-03-21T14:21:04.651302+0000","last_peered":"2024-03-21T14:21:04.651302+0000","last_clean":"2024-03-21T14:21:04.651302+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:21:04.651302+0000","last_undegraded":"2024-03-21T14:21:04.651302+0000","last_fullsized":"2024-03-21T14:21:04.651302+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:07.244640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:14:07.244640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828576+0000","last_change":"2024-03-21T14:20:59.387064+0000","last_active":"2024-03-21T14:21:03.828576+0000","last_peered":"2024-03-21T14:21:03.828576+0000","last_clean":"2024-03-21T14:21:03.828576+0000","last_became_active":"2024-03-21T14:20:59.386748+0000","last_became_peered":"2024-03-21T14:20:59.386748+0000","last_unstale":"2024-03-21T14:21:03.828576+0000","last_undegraded":"2024-03-21T14:21:03.828576+0000","last_fullsized":"2024-03-21T14:21:03.828576+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:02.455934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":237,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829848+0000","last_change":"2024-03-21T14:16:55.947536+0000","last_active":"2024-03-21T14:21:03.829848+0000","last_peered":"2024-03-21T14:21:03.829848+0000","last_clean":"2024-03-21T14:21:03.829848+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:21:03.829848+0000","last_undegraded":"2024-03-21T14:21:03.829848+0000","last_fullsized":"2024-03-21T14:21:03.829848+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:55.947438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:16:55.947438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813269+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1086,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649920+0000","last_change":"2024-03-21T14:19:40.473390+0000","last_active":"2024-03-21T14:21:04.649920+0000","last_peered":"2024-03-21T14:21:04.649920+0000","last_clean":"2024-03-21T14:21:04.649920+0000","last_became_active":"2024-03-21T14:19:37.727165+0000","last_became_peered":"2024-03-21T14:19:37.727165+0000","last_unstale":"2024-03-21T14:21:04.649920+0000","last_undegraded":"2024-03-21T14:21:04.649920+0000","last_fullsized":"2024-03-21T14:21:04.649920+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:40.473280+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:40.473280+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":786,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828587+0000","last_change":"2024-03-21T14:20:49.815275+0000","last_active":"2024-03-21T14:21:03.828587+0000","last_peered":"2024-03-21T14:21:03.828587+0000","last_clean":"2024-03-21T14:21:03.828587+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:21:03.828587+0000","last_undegraded":"2024-03-21T14:21:03.828587+0000","last_fullsized":"2024-03-21T14:21:03.828587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:49.815144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.815144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:11.062071+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":625,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370580+0000","last_change":"2024-03-21T14:17:59.176788+0000","last_active":"2024-03-21T14:21:04.370580+0000","last_peered":"2024-03-21T14:21:04.370580+0000","last_clean":"2024-03-21T14:21:04.370580+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:21:04.370580+0000","last_undegraded":"2024-03-21T14:21:04.370580+0000","last_fullsized":"2024-03-21T14:21:04.370580+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'29","last_scrub_stamp":"2024-03-21T14:17:59.176541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:59.176541+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":132,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650981+0000","last_change":"2024-03-21T14:20:33.512686+0000","last_active":"2024-03-21T14:21:04.650981+0000","last_peered":"2024-03-21T14:21:04.650981+0000","last_clean":"2024-03-21T14:21:04.650981+0000","last_became_active":"2024-03-21T14:19:14.690145+0000","last_became_peered":"2024-03-21T14:19:14.690145+0000","last_unstale":"2024-03-21T14:21:04.650981+0000","last_undegraded":"2024-03-21T14:21:04.650981+0000","last_fullsized":"2024-03-21T14:21:04.650981+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:33.512571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:33.512571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:00.208164+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828391+0000","last_change":"2024-03-21T14:20:59.239970+0000","last_active":"2024-03-21T14:21:03.828391+0000","last_peered":"2024-03-21T14:21:03.828391+0000","last_clean":"2024-03-21T14:21:03.828391+0000","last_became_active":"2024-03-21T14:20:59.239659+0000","last_became_peered":"2024-03-21T14:20:59.239659+0000","last_unstale":"2024-03-21T14:21:03.828391+0000","last_undegraded":"2024-03-21T14:21:03.828391+0000","last_fullsized":"2024-03-21T14:21:03.828391+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.241379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":842,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313629+0000","last_change":"2024-03-21T14:18:43.149509+0000","last_active":"2024-03-21T14:21:03.313629+0000","last_peered":"2024-03-21T14:21:03.313629+0000","last_clean":"2024-03-21T14:21:03.313629+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:21:03.313629+0000","last_undegraded":"2024-03-21T14:21:03.313629+0000","last_fullsized":"2024-03-21T14:21:03.313629+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:18:43.148036+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:18:43.148036+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113184+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":786,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314148+0000","last_change":"2024-03-21T14:20:48.131092+0000","last_active":"2024-03-21T14:21:03.314148+0000","last_peered":"2024-03-21T14:21:03.314148+0000","last_clean":"2024-03-21T14:21:03.314148+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:21:03.314148+0000","last_undegraded":"2024-03-21T14:21:03.314148+0000","last_fullsized":"2024-03-21T14:21:03.314148+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.130989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.130989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:04.073822+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":571,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370036+0000","last_change":"2024-03-21T14:20:21.778652+0000","last_active":"2024-03-21T14:21:04.370036+0000","last_peered":"2024-03-21T14:21:04.370036+0000","last_clean":"2024-03-21T14:21:04.370036+0000","last_became_active":"2024-03-21T14:20:21.778326+0000","last_became_peered":"2024-03-21T14:20:21.778326+0000","last_unstale":"2024-03-21T14:21:04.370036+0000","last_undegraded":"2024-03-21T14:21:04.370036+0000","last_fullsized":"2024-03-21T14:21:04.370036+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:50.207374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:50.207374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370492+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1797,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:06.282531+0000","last_change":"2024-03-21T14:18:35.154897+0000","last_active":"2024-03-21T14:21:06.282531+0000","last_peered":"2024-03-21T14:21:06.282531+0000","last_clean":"2024-03-21T14:21:06.282531+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:21:06.282531+0000","last_undegraded":"2024-03-21T14:21:06.282531+0000","last_fullsized":"2024-03-21T14:21:06.282531+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:18:35.154808+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:35.154808+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:11.129974+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1048,"num_read_kb":1043,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":715,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.315022+0000","last_change":"2024-03-21T14:19:34.715158+0000","last_active":"2024-03-21T14:21:03.315022+0000","last_peered":"2024-03-21T14:21:03.315022+0000","last_clean":"2024-03-21T14:21:03.315022+0000","last_became_active":"2024-03-21T14:19:34.692301+0000","last_became_peered":"2024-03-21T14:19:34.692301+0000","last_unstale":"2024-03-21T14:21:03.315022+0000","last_undegraded":"2024-03-21T14:21:03.315022+0000","last_fullsized":"2024-03-21T14:21:03.315022+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:19:24.502253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:24.502253+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113260+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":239,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.315217+0000","last_change":"2024-03-21T14:18:04.170207+0000","last_active":"2024-03-21T14:21:03.315217+0000","last_peered":"2024-03-21T14:21:03.315217+0000","last_clean":"2024-03-21T14:21:03.315217+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:21:03.315217+0000","last_undegraded":"2024-03-21T14:21:03.315217+0000","last_fullsized":"2024-03-21T14:21:03.315217+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:04.170084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:18:04.170084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113083+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314256+0000","last_change":"2024-03-21T14:20:59.110120+0000","last_active":"2024-03-21T14:21:03.314256+0000","last_peered":"2024-03-21T14:21:03.314256+0000","last_clean":"2024-03-21T14:21:03.314256+0000","last_became_active":"2024-03-21T14:20:59.109825+0000","last_became_peered":"2024-03-21T14:20:59.109825+0000","last_unstale":"2024-03-21T14:21:03.314256+0000","last_undegraded":"2024-03-21T14:21:03.314256+0000","last_fullsized":"2024-03-21T14:21:03.314256+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:05.092295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":578,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829011+0000","last_change":"2024-03-21T14:20:30.945601+0000","last_active":"2024-03-21T14:21:03.829011+0000","last_peered":"2024-03-21T14:21:03.829011+0000","last_clean":"2024-03-21T14:21:03.829011+0000","last_became_active":"2024-03-21T14:20:24.825637+0000","last_became_peered":"2024-03-21T14:20:24.825637+0000","last_unstale":"2024-03-21T14:21:03.829011+0000","last_undegraded":"2024-03-21T14:21:03.829011+0000","last_fullsized":"2024-03-21T14:21:03.829011+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:30.945471+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.945471+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:41.790389+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":775,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313150+0000","last_change":"2024-03-21T14:19:34.274417+0000","last_active":"2024-03-21T14:21:03.313150+0000","last_peered":"2024-03-21T14:21:03.313150+0000","last_clean":"2024-03-21T14:21:03.313150+0000","last_became_active":"2024-03-21T14:19:14.697998+0000","last_became_peered":"2024-03-21T14:19:14.697998+0000","last_unstale":"2024-03-21T14:21:03.313150+0000","last_undegraded":"2024-03-21T14:21:03.313150+0000","last_fullsized":"2024-03-21T14:21:03.313150+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:34.274201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:19:34.274201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:06.153531+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":781,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314439+0000","last_change":"2024-03-21T14:19:26.469534+0000","last_active":"2024-03-21T14:21:03.314439+0000","last_peered":"2024-03-21T14:21:03.314439+0000","last_clean":"2024-03-21T14:21:03.314439+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:21:03.314439+0000","last_undegraded":"2024-03-21T14:21:03.314439+0000","last_fullsized":"2024-03-21T14:21:03.314439+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:26.469255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:19:26.469255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113243+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":503,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.312386+0000","last_change":"2024-03-21T14:20:31.530370+0000","last_active":"2024-03-21T14:21:03.312386+0000","last_peered":"2024-03-21T14:21:03.312386+0000","last_clean":"2024-03-21T14:21:03.312386+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:21:03.312386+0000","last_undegraded":"2024-03-21T14:21:03.312386+0000","last_fullsized":"2024-03-21T14:21:03.312386+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:20:31.530270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.530270+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.391998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":136,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650266+0000","last_change":"2024-03-21T14:20:34.551705+0000","last_active":"2024-03-21T14:21:04.650266+0000","last_peered":"2024-03-21T14:21:04.650266+0000","last_clean":"2024-03-21T14:21:04.650266+0000","last_became_active":"2024-03-21T14:19:14.675538+0000","last_became_peered":"2024-03-21T14:19:14.675538+0000","last_unstale":"2024-03-21T14:21:04.650266+0000","last_undegraded":"2024-03-21T14:21:04.650266+0000","last_fullsized":"2024-03-21T14:21:04.650266+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:34.551577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:20:34.551577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:44.306785+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.651139+0000","last_change":"2024-03-21T14:20:59.111125+0000","last_active":"2024-03-21T14:21:04.651139+0000","last_peered":"2024-03-21T14:21:04.651139+0000","last_clean":"2024-03-21T14:21:04.651139+0000","last_became_active":"2024-03-21T14:20:59.110785+0000","last_became_peered":"2024-03-21T14:20:59.110785+0000","last_unstale":"2024-03-21T14:21:04.651139+0000","last_undegraded":"2024-03-21T14:21:04.651139+0000","last_fullsized":"2024-03-21T14:21:04.651139+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:05.117163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":752,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829249+0000","last_change":"2024-03-21T14:20:24.821031+0000","last_active":"2024-03-21T14:21:03.829249+0000","last_peered":"2024-03-21T14:21:03.829249+0000","last_clean":"2024-03-21T14:21:03.829249+0000","last_became_active":"2024-03-21T14:20:24.819844+0000","last_became_peered":"2024-03-21T14:20:24.819844+0000","last_unstale":"2024-03-21T14:21:03.829249+0000","last_undegraded":"2024-03-21T14:21:03.829249+0000","last_fullsized":"2024-03-21T14:21:03.829249+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:15.665015+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.665015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":275,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828283+0000","last_change":"2024-03-21T14:19:15.688754+0000","last_active":"2024-03-21T14:21:03.828283+0000","last_peered":"2024-03-21T14:21:03.828283+0000","last_clean":"2024-03-21T14:21:03.828283+0000","last_became_active":"2024-03-21T14:19:15.688238+0000","last_became_peered":"2024-03-21T14:19:15.688238+0000","last_unstale":"2024-03-21T14:21:03.828283+0000","last_undegraded":"2024-03-21T14:21:03.828283+0000","last_fullsized":"2024-03-21T14:21:03.828283+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:11.920076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:11.920076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813580+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":712,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828979+0000","last_change":"2024-03-21T14:20:32.884624+0000","last_active":"2024-03-21T14:21:03.828979+0000","last_peered":"2024-03-21T14:21:03.828979+0000","last_clean":"2024-03-21T14:21:03.828979+0000","last_became_active":"2024-03-21T14:19:14.695943+0000","last_became_peered":"2024-03-21T14:19:14.695943+0000","last_unstale":"2024-03-21T14:21:03.828979+0000","last_undegraded":"2024-03-21T14:21:03.828979+0000","last_fullsized":"2024-03-21T14:21:03.828979+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:32.884514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:32.884514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:41.374001+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":811,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313809+0000","last_change":"2024-03-21T14:18:29.201331+0000","last_active":"2024-03-21T14:21:03.313809+0000","last_peered":"2024-03-21T14:21:03.313809+0000","last_clean":"2024-03-21T14:21:03.313809+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:21:03.313809+0000","last_undegraded":"2024-03-21T14:21:03.313809+0000","last_fullsized":"2024-03-21T14:21:03.313809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:18:29.201228+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.201228+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":752,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828595+0000","last_change":"2024-03-21T14:18:44.046070+0000","last_active":"2024-03-21T14:21:03.828595+0000","last_peered":"2024-03-21T14:21:03.828595+0000","last_clean":"2024-03-21T14:21:03.828595+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:21:03.828595+0000","last_undegraded":"2024-03-21T14:21:03.828595+0000","last_fullsized":"2024-03-21T14:21:03.828595+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'11","last_scrub_stamp":"2024-03-21T14:18:44.045985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:44.045985+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370533+0000","last_change":"2024-03-21T14:20:59.110795+0000","last_active":"2024-03-21T14:21:04.370533+0000","last_peered":"2024-03-21T14:21:04.370533+0000","last_clean":"2024-03-21T14:21:04.370533+0000","last_became_active":"2024-03-21T14:20:59.110468+0000","last_became_peered":"2024-03-21T14:20:59.110468+0000","last_unstale":"2024-03-21T14:21:04.370533+0000","last_undegraded":"2024-03-21T14:21:04.370533+0000","last_fullsized":"2024-03-21T14:21:04.370533+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:09.204354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":834,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828921+0000","last_change":"2024-03-21T14:20:52.764101+0000","last_active":"2024-03-21T14:21:03.828921+0000","last_peered":"2024-03-21T14:21:03.828921+0000","last_clean":"2024-03-21T14:21:03.828921+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:21:03.828921+0000","last_undegraded":"2024-03-21T14:21:03.828921+0000","last_fullsized":"2024-03-21T14:21:03.828921+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:52.763988+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.763988+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:11.926808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":775,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650380+0000","last_change":"2024-03-21T14:20:29.685381+0000","last_active":"2024-03-21T14:21:04.650380+0000","last_peered":"2024-03-21T14:21:04.650380+0000","last_clean":"2024-03-21T14:21:04.650380+0000","last_became_active":"2024-03-21T14:19:14.684876+0000","last_became_peered":"2024-03-21T14:19:14.684876+0000","last_unstale":"2024-03-21T14:21:04.650380+0000","last_undegraded":"2024-03-21T14:21:04.650380+0000","last_fullsized":"2024-03-21T14:21:04.650380+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:29.685267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:27.609702+0000","last_clean_scrub_stamp":"2024-03-21T14:20:29.685267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:45.309330+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":648,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314842+0000","last_change":"2024-03-21T14:20:51.116248+0000","last_active":"2024-03-21T14:21:03.314842+0000","last_peered":"2024-03-21T14:21:03.314842+0000","last_clean":"2024-03-21T14:21:03.314842+0000","last_became_active":"2024-03-21T14:19:34.692258+0000","last_became_peered":"2024-03-21T14:19:34.692258+0000","last_unstale":"2024-03-21T14:21:03.314842+0000","last_undegraded":"2024-03-21T14:21:03.314842+0000","last_fullsized":"2024-03-21T14:21:03.314842+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:20:51.116139+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.116139+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:05.825547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":823,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369674+0000","last_change":"2024-03-21T14:17:56.127388+0000","last_active":"2024-03-21T14:21:04.369674+0000","last_peered":"2024-03-21T14:21:04.369674+0000","last_clean":"2024-03-21T14:21:04.369674+0000","last_became_active":"2024-03-21T14:15:59.607826+0000","last_became_peered":"2024-03-21T14:15:59.607826+0000","last_unstale":"2024-03-21T14:21:04.369674+0000","last_undegraded":"2024-03-21T14:21:04.369674+0000","last_fullsized":"2024-03-21T14:21:04.369674+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'28","last_scrub_stamp":"2024-03-21T14:17:56.127254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:56.127254+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370477+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":996,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828808+0000","last_change":"2024-03-21T14:18:38.115200+0000","last_active":"2024-03-21T14:21:03.828808+0000","last_peered":"2024-03-21T14:21:03.828808+0000","last_clean":"2024-03-21T14:21:03.828808+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:21:03.828808+0000","last_undegraded":"2024-03-21T14:21:03.828808+0000","last_fullsized":"2024-03-21T14:21:03.828808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:18:38.115111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:38.115111+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813445+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314102+0000","last_change":"2024-03-21T14:20:59.110456+0000","last_active":"2024-03-21T14:21:03.314102+0000","last_peered":"2024-03-21T14:21:03.314102+0000","last_clean":"2024-03-21T14:21:03.314102+0000","last_became_active":"2024-03-21T14:20:59.109593+0000","last_became_peered":"2024-03-21T14:20:59.109593+0000","last_unstale":"2024-03-21T14:21:03.314102+0000","last_undegraded":"2024-03-21T14:21:03.314102+0000","last_fullsized":"2024-03-21T14:21:03.314102+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:20.788956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":828,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370391+0000","last_change":"2024-03-21T14:19:14.682603+0000","last_active":"2024-03-21T14:21:04.370391+0000","last_peered":"2024-03-21T14:21:04.370391+0000","last_clean":"2024-03-21T14:21:04.370391+0000","last_became_active":"2024-03-21T14:19:14.682251+0000","last_became_peered":"2024-03-21T14:19:14.682251+0000","last_unstale":"2024-03-21T14:21:04.370391+0000","last_undegraded":"2024-03-21T14:21:04.370391+0000","last_fullsized":"2024-03-21T14:21:04.370391+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:14:35.954782+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:14:35.954782+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":775,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370137+0000","last_change":"2024-03-21T14:20:21.782339+0000","last_active":"2024-03-21T14:21:04.370137+0000","last_peered":"2024-03-21T14:21:04.370137+0000","last_clean":"2024-03-21T14:21:04.370137+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:21:04.370137+0000","last_undegraded":"2024-03-21T14:21:04.370137+0000","last_fullsized":"2024-03-21T14:21:04.370137+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:11.405823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:17:11.405823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370394+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":772,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829616+0000","last_change":"2024-03-21T14:17:02.767201+0000","last_active":"2024-03-21T14:21:03.829616+0000","last_peered":"2024-03-21T14:21:03.829616+0000","last_clean":"2024-03-21T14:21:03.829616+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:21:03.829616+0000","last_undegraded":"2024-03-21T14:21:03.829616+0000","last_fullsized":"2024-03-21T14:21:03.829616+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:02.767079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:17:02.767079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813332+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":73,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828169+0000","last_change":"2024-03-21T14:20:21.803142+0000","last_active":"2024-03-21T14:21:03.828169+0000","last_peered":"2024-03-21T14:21:03.828169+0000","last_clean":"2024-03-21T14:21:03.828169+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:21:03.828169+0000","last_undegraded":"2024-03-21T14:21:03.828169+0000","last_fullsized":"2024-03-21T14:21:03.828169+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:19:12.604917+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:19:12.604917+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813614+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":747,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369869+0000","last_change":"2024-03-21T14:17:14.254451+0000","last_active":"2024-03-21T14:21:04.369869+0000","last_peered":"2024-03-21T14:21:04.369869+0000","last_clean":"2024-03-21T14:21:04.369869+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:21:04.369869+0000","last_undegraded":"2024-03-21T14:21:04.369869+0000","last_fullsized":"2024-03-21T14:21:04.369869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:14.254334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:14.254334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370443+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.313087+0000","last_change":"2024-03-21T14:20:59.109540+0000","last_active":"2024-03-21T14:21:03.313087+0000","last_peered":"2024-03-21T14:21:03.313087+0000","last_clean":"2024-03-21T14:21:03.313087+0000","last_became_active":"2024-03-21T14:20:59.109191+0000","last_became_peered":"2024-03-21T14:20:59.109191+0000","last_unstale":"2024-03-21T14:21:03.313087+0000","last_undegraded":"2024-03-21T14:21:03.313087+0000","last_fullsized":"2024-03-21T14:21:03.313087+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:00.799027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":748,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650561+0000","last_change":"2024-03-21T14:20:36.503605+0000","last_active":"2024-03-21T14:21:04.650561+0000","last_peered":"2024-03-21T14:21:04.650561+0000","last_clean":"2024-03-21T14:21:04.650561+0000","last_became_active":"2024-03-21T14:19:14.692919+0000","last_became_peered":"2024-03-21T14:19:14.692919+0000","last_unstale":"2024-03-21T14:21:04.650561+0000","last_undegraded":"2024-03-21T14:21:04.650561+0000","last_fullsized":"2024-03-21T14:21:04.650561+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:36.503498+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:36.503498+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:04.933569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":789,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314143+0000","last_change":"2024-03-21T14:20:43.142589+0000","last_active":"2024-03-21T14:21:03.314143+0000","last_peered":"2024-03-21T14:21:03.314143+0000","last_clean":"2024-03-21T14:21:03.314143+0000","last_became_active":"2024-03-21T14:19:34.691121+0000","last_became_peered":"2024-03-21T14:19:34.691121+0000","last_unstale":"2024-03-21T14:21:03.314143+0000","last_undegraded":"2024-03-21T14:21:03.314143+0000","last_fullsized":"2024-03-21T14:21:03.314143+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:43.142495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:20:43.142495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:03.496861+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":986,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370720+0000","last_change":"2024-03-21T14:17:03.353514+0000","last_active":"2024-03-21T14:21:04.370720+0000","last_peered":"2024-03-21T14:21:04.370720+0000","last_clean":"2024-03-21T14:21:04.370720+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:21:04.370720+0000","last_undegraded":"2024-03-21T14:21:04.370720+0000","last_fullsized":"2024-03-21T14:21:04.370720+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:17:03.353394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.353394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828263+0000","last_change":"2024-03-21T14:20:21.783183+0000","last_active":"2024-03-21T14:21:03.828263+0000","last_peered":"2024-03-21T14:21:03.828263+0000","last_clean":"2024-03-21T14:21:03.828263+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:21:03.828263+0000","last_undegraded":"2024-03-21T14:21:03.828263+0000","last_fullsized":"2024-03-21T14:21:03.828263+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:53.136512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:53.136512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813396+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":819,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314646+0000","last_change":"2024-03-21T14:20:42.035288+0000","last_active":"2024-03-21T14:21:03.314646+0000","last_peered":"2024-03-21T14:21:03.314646+0000","last_clean":"2024-03-21T14:21:03.314646+0000","last_became_active":"2024-03-21T14:02:35.452770+0000","last_became_peered":"2024-03-21T14:02:35.452770+0000","last_unstale":"2024-03-21T14:21:03.314646+0000","last_undegraded":"2024-03-21T14:21:03.314646+0000","last_fullsized":"2024-03-21T14:21:03.314646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:20:42.035180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.035180+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:05.671907+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828274+0000","last_change":"2024-03-21T14:20:59.240921+0000","last_active":"2024-03-21T14:21:03.828274+0000","last_peered":"2024-03-21T14:21:03.828274+0000","last_clean":"2024-03-21T14:21:03.828274+0000","last_became_active":"2024-03-21T14:20:59.240641+0000","last_became_peered":"2024-03-21T14:20:59.240641+0000","last_unstale":"2024-03-21T14:21:03.828274+0000","last_undegraded":"2024-03-21T14:21:03.828274+0000","last_fullsized":"2024-03-21T14:21:03.828274+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:06.150245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":766,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.828821+0000","last_change":"2024-03-21T14:18:28.880752+0000","last_active":"2024-03-21T14:21:03.828821+0000","last_peered":"2024-03-21T14:21:03.828821+0000","last_clean":"2024-03-21T14:21:03.828821+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:21:03.828821+0000","last_undegraded":"2024-03-21T14:21:03.828821+0000","last_fullsized":"2024-03-21T14:21:03.828821+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:28.880620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:28.880620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813461+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":750,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370650+0000","last_change":"2024-03-21T14:19:15.688745+0000","last_active":"2024-03-21T14:21:04.370650+0000","last_peered":"2024-03-21T14:21:04.370650+0000","last_clean":"2024-03-21T14:21:04.370650+0000","last_became_active":"2024-03-21T14:19:15.688278+0000","last_became_peered":"2024-03-21T14:19:15.688278+0000","last_unstale":"2024-03-21T14:21:04.370650+0000","last_undegraded":"2024-03-21T14:21:04.370650+0000","last_fullsized":"2024-03-21T14:21:04.370650+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:16.566202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:16:16.566202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":772,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.369901+0000","last_change":"2024-03-21T14:20:48.538833+0000","last_active":"2024-03-21T14:21:04.369901+0000","last_peered":"2024-03-21T14:21:04.369901+0000","last_clean":"2024-03-21T14:21:04.369901+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:21:04.369901+0000","last_undegraded":"2024-03-21T14:21:04.369901+0000","last_fullsized":"2024-03-21T14:21:04.369901+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.538744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.538744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.083242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":808,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370615+0000","last_change":"2024-03-21T14:18:32.414098+0000","last_active":"2024-03-21T14:21:04.370615+0000","last_peered":"2024-03-21T14:21:04.370615+0000","last_clean":"2024-03-21T14:21:04.370615+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:21:04.370615+0000","last_undegraded":"2024-03-21T14:21:04.370615+0000","last_fullsized":"2024-03-21T14:21:04.370615+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:18:32.413975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:32.413975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649986+0000","last_change":"2024-03-21T14:20:59.239976+0000","last_active":"2024-03-21T14:21:04.649986+0000","last_peered":"2024-03-21T14:21:04.649986+0000","last_clean":"2024-03-21T14:21:04.649986+0000","last_became_active":"2024-03-21T14:20:59.239396+0000","last_became_peered":"2024-03-21T14:20:59.239396+0000","last_unstale":"2024-03-21T14:21:04.649986+0000","last_undegraded":"2024-03-21T14:21:04.649986+0000","last_fullsized":"2024-03-21T14:21:04.649986+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:59.991754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":748,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649834+0000","last_change":"2024-03-21T14:20:37.552524+0000","last_active":"2024-03-21T14:21:04.649834+0000","last_peered":"2024-03-21T14:21:04.649834+0000","last_clean":"2024-03-21T14:21:04.649834+0000","last_became_active":"2024-03-21T14:19:14.700353+0000","last_became_peered":"2024-03-21T14:19:14.700353+0000","last_unstale":"2024-03-21T14:21:04.649834+0000","last_undegraded":"2024-03-21T14:21:04.649834+0000","last_fullsized":"2024-03-21T14:21:04.649834+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:37.552424+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.552424+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:51.901528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":567,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650595+0000","last_change":"2024-03-21T14:20:45.611947+0000","last_active":"2024-03-21T14:21:04.650595+0000","last_peered":"2024-03-21T14:21:04.650595+0000","last_clean":"2024-03-21T14:21:04.650595+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:21:04.650595+0000","last_undegraded":"2024-03-21T14:21:04.650595+0000","last_fullsized":"2024-03-21T14:21:04.650595+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.611844+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:06.146803+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":772,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.314520+0000","last_change":"2024-03-21T14:18:27.281305+0000","last_active":"2024-03-21T14:21:03.314520+0000","last_peered":"2024-03-21T14:21:03.314520+0000","last_clean":"2024-03-21T14:21:03.314520+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:21:03.314520+0000","last_undegraded":"2024-03-21T14:21:03.314520+0000","last_fullsized":"2024-03-21T14:21:03.314520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:27.281190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:54.426178+0000","last_clean_scrub_stamp":"2024-03-21T14:18:27.281190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:05.113168+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":59,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.650175+0000","last_change":"2024-03-21T14:20:19.703923+0000","last_active":"2024-03-21T14:21:04.650175+0000","last_peered":"2024-03-21T14:21:04.650175+0000","last_clean":"2024-03-21T14:21:04.650175+0000","last_became_active":"2024-03-21T14:20:19.703587+0000","last_became_peered":"2024-03-21T14:20:19.703587+0000","last_unstale":"2024-03-21T14:21:04.650175+0000","last_undegraded":"2024-03-21T14:21:04.650175+0000","last_fullsized":"2024-03-21T14:21:04.650175+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:16.247145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.247145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":809,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370681+0000","last_change":"2024-03-21T14:20:46.480561+0000","last_active":"2024-03-21T14:21:04.370681+0000","last_peered":"2024-03-21T14:21:04.370681+0000","last_clean":"2024-03-21T14:21:04.370681+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:21:04.370681+0000","last_undegraded":"2024-03-21T14:21:04.370681+0000","last_fullsized":"2024-03-21T14:21:04.370681+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:20:46.480457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.480457+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:06.690255+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":15,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649991+0000","last_change":"2024-03-21T14:20:59.387035+0000","last_active":"2024-03-21T14:21:04.649991+0000","last_peered":"2024-03-21T14:21:04.649991+0000","last_clean":"2024-03-21T14:21:04.649991+0000","last_became_active":"2024-03-21T14:20:59.386715+0000","last_became_peered":"2024-03-21T14:20:59.386715+0000","last_unstale":"2024-03-21T14:21:04.649991+0000","last_undegraded":"2024-03-21T14:21:04.649991+0000","last_fullsized":"2024-03-21T14:21:04.649991+0000","mapping_epoch":411,"log_start":"0'0","ondisk_log_start":"0'0","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:09.502262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1085,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.649943+0000","last_change":"2024-03-21T14:19:36.429638+0000","last_active":"2024-03-21T14:21:04.649943+0000","last_peered":"2024-03-21T14:21:04.649943+0000","last_clean":"2024-03-21T14:21:04.649943+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:21:04.649943+0000","last_undegraded":"2024-03-21T14:21:04.649943+0000","last_fullsized":"2024-03-21T14:21:04.649943+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:36.429507+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:36.429507+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":776,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370795+0000","last_change":"2024-03-21T14:19:15.689377+0000","last_active":"2024-03-21T14:21:04.370795+0000","last_peered":"2024-03-21T14:21:04.370795+0000","last_clean":"2024-03-21T14:21:04.370795+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:21:04.370795+0000","last_undegraded":"2024-03-21T14:21:04.370795+0000","last_fullsized":"2024-03-21T14:21:04.370795+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:13.493205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:13.493205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":760,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.651208+0000","last_change":"2024-03-21T14:20:49.535756+0000","last_active":"2024-03-21T14:21:04.651208+0000","last_peered":"2024-03-21T14:21:04.651208+0000","last_clean":"2024-03-21T14:21:04.651208+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:21:04.651208+0000","last_undegraded":"2024-03-21T14:21:04.651208+0000","last_fullsized":"2024-03-21T14:21:04.651208+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.535652+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:10.421931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":784,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:03.829036+0000","last_change":"2024-03-21T14:19:34.693436+0000","last_active":"2024-03-21T14:21:03.829036+0000","last_peered":"2024-03-21T14:21:03.829036+0000","last_clean":"2024-03-21T14:21:03.829036+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:21:03.829036+0000","last_undegraded":"2024-03-21T14:21:03.829036+0000","last_fullsized":"2024-03-21T14:21:03.829036+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:17.921956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:17.921956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:07.813363+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":812,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-21T14:21:04.370331+0000","last_change":"2024-03-21T14:18:34.406758+0000","last_active":"2024-03-21T14:21:04.370331+0000","last_peered":"2024-03-21T14:21:04.370331+0000","last_clean":"2024-03-21T14:21:04.370331+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:21:04.370331+0000","last_undegraded":"2024-03-21T14:21:04.370331+0000","last_fullsized":"2024-03-21T14:21:04.370331+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:18:34.406635+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:18:34.406635+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:21:04.370558+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1048,"num_read_kb":1043,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":157,"num_bytes_recovered":3592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":11216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103395,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810052,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92935100,"statfs":{"total":95995035648,"available":95165542400,"internally_reserved":0,"allocated":2875392,"data_stored":1783125,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":826603978},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":1507533520923,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575107,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":781980,"kb_used_data":2512,"kb_used_omap":1,"kb_used_meta":779454,"kb_avail":92963172,"statfs":{"total":95995035648,"available":95194288128,"internally_reserved":0,"allocated":2572288,"data_stored":1456036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":798161349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738632,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":727828,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":725310,"kb_avail":93017324,"statfs":{"total":95995035648,"available":95249739776,"internally_reserved":0,"allocated":2564096,"data_stored":1454669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":742717893},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598291992,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1851124,"kb_used_data":1600620,"kb_used_omap":7,"kb_used_meta":250488,"kb_avail":91894028,"statfs":{"total":95995035648,"available":94099484672,"internally_reserved":0,"allocated":1639034880,"data_stored":1637953830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":256499844},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182647,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":647908,"kb_used_data":3096,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93097244,"statfs":{"total":95995035648,"available":95331577856,"internally_reserved":0,"allocated":3170304,"data_stored":2045410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019123,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":845844,"kb_used_data":2504,"kb_used_omap":1,"kb_used_meta":843326,"kb_avail":92899308,"statfs":{"total":95995035648,"available":95128891392,"internally_reserved":0,"allocated":2564096,"data_stored":1454646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":863566277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692078,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":810324,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92934828,"statfs":{"total":95995035648,"available":95165263872,"internally_reserved":0,"allocated":3153920,"data_stored":2045775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:21:12.667 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T14:21:12.668 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:21:12.668 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-21T14:21:12.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd in 6 2024-03-21T14:21:12.671 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:12.671 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:12.678 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:12.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:12.751 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:21:12.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:12.834 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:12.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:12.962 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:12.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[29576]: pgmap v885: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[29576]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:21:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2361008123' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:21:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:21:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[40510]: pgmap v885: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[40510]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:21:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2361008123' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:21:13.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:12 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:21:13.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:13.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:13.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:13.120 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:12 smithi118 ceph-mon[33862]: pgmap v885: 158 pgs: 158 active+clean; 582 KiB data, 6.2 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:13.121 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:12 smithi118 ceph-mon[33862]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-21T14:21:13.121 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2361008123' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:21:13.121 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:12 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:21:13.190 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.216 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:13.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:13.280 DEBUG:teuthology.orchestra.run.smithi003:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-21T14:21:13.282 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-21T14:21:13.282 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-21T14:21:13.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:13.442 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:13.466 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:13.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:13.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:13.588 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:hints = 1 2024-03-21T14:21:13.588 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-21T14:21:13.588 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object prefix: benchmark_data_smithi003_188959 2024-03-21T14:21:13.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:21:13.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 0 0 0 0 0 0 - 0 2024-03-21T14:21:13.595 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:13.697 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:13.726 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:13.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:13.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:13.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:13.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:13 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:13 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:21:14.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:13 smithi003 ceph-mon[29576]: osdmap e418: 8 total, 8 up, 7 in 2024-03-21T14:21:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:13 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:13 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:21:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:13 smithi003 ceph-mon[40510]: osdmap e418: 8 total, 8 up, 7 in 2024-03-21T14:21:14.009 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:14.110 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:14.138 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:13 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:13 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:21:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:13 smithi118 ceph-mon[33862]: osdmap e418: 8 total, 8 up, 7 in 2024-03-21T14:21:14.240 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:14.269 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:14.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:14.399 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:14.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:14.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:14.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 1 15 2950 2935 183.427 183.438 0.00579167 0.00543138 2024-03-21T14:21:14.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:14.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:14.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:14.788 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:14.890 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:14.896 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:14 smithi003 ceph-mon[29576]: pgmap v887: 158 pgs: 158 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:21:14.896 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:14 smithi003 ceph-mon[40510]: pgmap v887: 158 pgs: 158 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:21:14.918 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:14.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:15.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:15.059 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:15.161 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:14 smithi118 ceph-mon[33862]: pgmap v887: 158 pgs: 158 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:21:15.189 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:15.291 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:15.318 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:15.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:15.446 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:15.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:15.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:15.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 2 15 5597 5582 174.425 165.438 0.00556358 0.00571756 2024-03-21T14:21:15.615 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:15.716 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:15.747 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:15.849 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:15.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:15.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:15.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.031 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:16.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:16.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:16.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:16.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:16.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:16.290 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:16.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:16.324 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:21:16.368 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:21:16.392 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:16.419 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:16.419 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:21:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:16.466 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:21:16.506 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:21:16.507 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:21:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:21:16.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:16.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:21:16.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:16.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 3 16 8074 8058 167.862 154.75 0.00829557 0.00594562 2024-03-21T14:21:16.590 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:21:16.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:21:16.637 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:21:16.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:16.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:16.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:16.807 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:16.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:16.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:16.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:16.978 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:17.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:17.088 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[29576]: pgmap v888: 158 pgs: 158 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2496188003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[40510]: pgmap v888: 158 pgs: 158 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:21:17.089 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:16 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2496188003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:16 smithi118 ceph-mon[33862]: pgmap v888: 158 pgs: 158 active+clean; 582 KiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:16 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:16 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:16 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:16 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:21:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:16 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2496188003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:17.189 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:17.215 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:17.251 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:21:17.251 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:21:17.251 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:9575 11.6 11:6cf95a03:::benchmark_data_smithi003_188959_object9574:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.246161+0000", 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0040653809999999999, 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041019170000000001, 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 9575 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.252 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246161+0000", 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246161+0000", 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246162+0000", 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2580000000000001e-06 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.253 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246182+0000", 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0088000000000001e-05 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246198+0000", 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5781e-05 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246206+0000", 2024-03-21T14:21:17.254 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8490000000000006e-06 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246255+0000", 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9274000000000003e-05 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246371+0000", 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116344 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.246504+0000", 2024-03-21T14:21:17.255 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132387 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.247484+0000", 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00097977999999999997 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:9576 11.6 11:63738b39:::benchmark_data_smithi003_188959_object9575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.248144+0000", 2024-03-21T14:21:17.256 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0020816900000000002, 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022092790000000002, 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 9576 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248144+0000", 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248144+0000", 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248146+0000", 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.322e-06 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.258 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248185+0000", 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8952000000000002e-05 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248203+0000", 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8731000000000001e-05 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248283+0000", 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9062999999999998e-05 2024-03-21T14:21:17.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248525+0000", 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024221400000000001 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249104+0000", 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00057906799999999997 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:21:17.260 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249247+0000", 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014337400000000001 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:9578 11.d 11:b7750554:::benchmark_data_smithi003_188959_object9577:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.248996+0000", 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001229909, 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001434846, 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:17.261 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 9578 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248996+0000", 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248996+0000", 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.248997+0000", 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1119999999999999e-06 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249024+0000", 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6333e-05 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249035+0000", 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1637000000000001e-05 2024-03-21T14:21:17.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249044+0000", 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4759999999999998e-06 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249232+0000", 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018818200000000001 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249360+0000", 2024-03-21T14:21:17.264 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000127506 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,4", 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249520+0000", 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015999300000000001 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:9582 11.6 11.64e2b746 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.249065+0000", 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.001161521, 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014388739999999999, 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 9582 2024-03-21T14:21:17.266 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249065+0000", 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249065+0000", 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249065+0000", 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7299999999999996e-07 2024-03-21T14:21:17.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249112+0000", 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6897999999999997e-05 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249120+0000", 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3329999999999997e-06 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249128+0000", 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.807e-06 2024-03-21T14:21:17.268 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.25811.0:9575 11.6 e418/411)", 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.249863+0000", 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.00036269699999999999, 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00069136100000000004, 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.269 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249863+0000", 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249863+0000", 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.270 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249864+0000", 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6000000000000002e-07 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249869+0000", 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3800000000000004e-06 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249879+0000", 2024-03-21T14:21:17.271 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9790000000000005e-06 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249886+0000", 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.883e-06 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:9577 11.1 e418/411 11:88bc2ec2:::benchmark_data_smithi003_188959_object9576:head v 418'582, mlcod=418'580)", 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.249944+0000", 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.000282019, 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00066708999999999996, 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.272 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249944+0000", 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249944+0000", 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.273 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249945+0000", 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2399999999999998e-07 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249967+0000", 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2229999999999999e-05 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249975+0000", 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2230000000000007e-06 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.274 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.249981+0000", 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0519999999999998e-06 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250029+0000", 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8273999999999999e-05 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250050+0000", 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0437000000000001e-05 2024-03-21T14:21:17.275 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.25811.0:9578 11.d e418/411)", 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:17.250571+0000", 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 4294967295.9996552, 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102023, 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "queued for pg", 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250571+0000", 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250571+0000", 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250572+0000", 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0399999999999995e-07 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250580+0000", 2024-03-21T14:21:17.277 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4500000000000004e-06 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250589+0000", 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2469999999999997e-06 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:17.250596+0000", 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.6109999999999999e-06 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.278 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:17.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.279 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:17.279 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:21:17.279 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 7 2024-03-21T14:21:17.279 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:21:17.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:17.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:17.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:17.457 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:17.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:17.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:17.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:17.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 4 16 10271 10255 160.221 137.312 0.00302998 0.0061649 2024-03-21T14:21:17.614 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:17.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:21:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:21:17.717 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:17.746 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:17.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:17.876 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:17.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:17.978 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:18.006 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:18.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:18.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:18.136 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:18.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:18.238 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:18.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:18.380 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:18.420 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:18.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:18.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:18.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 5 16 12598 12582 157.263 145.438 0.00315869 0.0063504 2024-03-21T14:21:18.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:18 smithi118 ceph-mon[33862]: pgmap v889: 158 pgs: 158 active+clean; 108 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 683 B/s rd, 14 MiB/s wr, 230 op/s 2024-03-21T14:21:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:18 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:18.694 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:18.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[29576]: pgmap v889: 158 pgs: 158 active+clean; 108 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 683 B/s rd, 14 MiB/s wr, 230 op/s 2024-03-21T14:21:18.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:18.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:18.709 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:18.710 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[40510]: pgmap v889: 158 pgs: 158 active+clean; 108 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 683 B/s rd, 14 MiB/s wr, 230 op/s 2024-03-21T14:21:18.710 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:18.710 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:18.710 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:18.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:18.823 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:18.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:18.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:18.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:18.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:19.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:19.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:19.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:19.217 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:19.245 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:19.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: 1.9 scrub starts 2024-03-21T14:21:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: 1.9 scrub ok 2024-03-21T14:21:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:21:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:21:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-03-21T14:21:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]: dispatch 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]: dispatch 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: 1.9 scrub starts 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: 1.9 scrub ok 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]: dispatch 2024-03-21T14:21:19.384 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-03-21T14:21:19.385 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:19 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]: dispatch 2024-03-21T14:21:19.391 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:19.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:19.433 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T14:21:19.493 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:19.494 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:19.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:19.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:19.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 6 16 14669 14653 152.623 129.438 0.00150783 0.00641609 2024-03-21T14:21:19.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:19.657 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: 1.9 scrub starts 2024-03-21T14:21:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: 1.9 scrub ok 2024-03-21T14:21:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:21:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:21:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-03-21T14:21:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]: dispatch 2024-03-21T14:21:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]: dispatch 2024-03-21T14:21:19.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:19 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]: dispatch 2024-03-21T14:21:19.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:19.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:19.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:19.908 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:19.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:20.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:20.032 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:20.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:20.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:21:20.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:20.181 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:20.283 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:20.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:20.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:20.432 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:21:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:21:20.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: pgmap v890: 158 pgs: 158 active+clean; 628 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 6.1c scrub starts 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 6.1c scrub ok 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 1.5 scrub starts 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 1.5 scrub ok 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]': finished 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: osdmap e419: 8 total, 8 up, 7 in 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[29576]: osdmap e420: 8 total, 8 up, 7 in 2024-03-21T14:21:20.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: pgmap v890: 158 pgs: 158 active+clean; 628 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 6.1c scrub starts 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 6.1c scrub ok 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 1.5 scrub starts 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 1.5 scrub ok 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]': finished 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: osdmap e419: 8 total, 8 up, 7 in 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:21:20.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:20 smithi003 ceph-mon[40510]: osdmap e420: 8 total, 8 up, 7 in 2024-03-21T14:21:20.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:20.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:20.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 7 16 14669 14653 130.819 0 - 0.00641609 2024-03-21T14:21:20.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:21:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:21:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: pgmap v890: 158 pgs: 158 active+clean; 628 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-21T14:21:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 6.1c scrub starts 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 6.1c scrub ok 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 1.5 scrub starts 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 1.5 scrub ok 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [3, 2]}]': finished 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.f", "id": [1, 4]}]': finished 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: osdmap e419: 8 total, 8 up, 7 in 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:21:20.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:20 smithi118 ceph-mon[33862]: osdmap e420: 8 total, 8 up, 7 in 2024-03-21T14:21:20.683 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:20.765 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:21:20.784 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:20.811 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:20.913 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:20.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:20.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:20.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:21.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:21.103 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:21.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:21.121 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:21.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:21.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.234 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:21.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:21.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:21.385 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:21.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:21.514 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:21.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 8 16 16176 16160 126.24 47.0938 0.0072379 0.00791285 2024-03-21T14:21:21.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:21.642 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:21.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:21.786 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[40510]: osdmap e421: 8 total, 8 up, 7 in 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:21:21.804 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:21:21.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:21:21.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:21:21.805 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:21 smithi003 ceph-mon[29576]: osdmap e421: 8 total, 8 up, 7 in 2024-03-21T14:21:21.887 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:21.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:22.018 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:22.046 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:22.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:22.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:21 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:21:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:21 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:21:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:21 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:21:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:21 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:21:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:21 smithi118 ceph-mon[33862]: osdmap e421: 8 total, 8 up, 7 in 2024-03-21T14:21:22.189 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:22.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:22.319 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:22.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:22.421 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:22.445 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:22.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:22.586 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:22.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 9 16 18170 18154 126.059 124.625 0.00625474 0.00792519 2024-03-21T14:21:22.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:22.734 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:22.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:22.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:22.861 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:22.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:22.962 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:22.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:22.989 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:22.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:22 smithi003 ceph-mon[29576]: pgmap v894: 158 pgs: 158 active+clean; 628 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-03-21T14:21:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:22 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:21:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:22 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:21:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:22 smithi003 ceph-mon[40510]: pgmap v894: 158 pgs: 158 active+clean; 628 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-03-21T14:21:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:22 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:21:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:22 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:21:23.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:22 smithi118 ceph-mon[33862]: pgmap v894: 158 pgs: 158 active+clean; 628 MiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-03-21T14:21:23.082 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:22 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:21:23.083 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:22 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:21:23.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:23.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:23.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:23.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:23.253 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:23.355 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:23.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:23.480 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:23.504 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:23.589 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 10 16 19920 19904 124.39 109.375 0.0109943 0.00803245 2024-03-21T14:21:23.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:23.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:23.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:23.767 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:23.869 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:23.900 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:23.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:24.002 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:24.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.044 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:24.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:24.144 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:24.187 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:24.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:24.317 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:24.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:24.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:24.462 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:24.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:24.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:24.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 11 16 21618 21602 122.728 106.125 0.010569 0.00814136 2024-03-21T14:21:24.591 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:24.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:24.693 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:24.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:24.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:24.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:24.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:24.973 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:24.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[29576]: pgmap v895: 158 pgs: 1 active+recovering+remapped, 157 active+clean; 1.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 168 MiB/s wr, 2.70k op/s; 765/54288 objects misplaced (1.409%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T14:21:24.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:21:24.988 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[29576]: osdmap e422: 8 total, 8 up, 7 in 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[40510]: pgmap v895: 158 pgs: 1 active+recovering+remapped, 157 active+clean; 1.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 168 MiB/s wr, 2.70k op/s; 765/54288 objects misplaced (1.409%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[40510]: osdmap e422: 8 total, 8 up, 7 in 2024-03-21T14:21:24.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:24 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:21:25.000 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:25.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:25.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:25.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:24 smithi118 ceph-mon[33862]: pgmap v895: 158 pgs: 1 active+recovering+remapped, 157 active+clean; 1.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 168 MiB/s wr, 2.70k op/s; 765/54288 objects misplaced (1.409%); 1.5 MiB/s, 24 objects/s recovering 2024-03-21T14:21:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:24 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:21:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:24 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:21:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:24 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:21:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:24 smithi118 ceph-mon[33862]: osdmap e422: 8 total, 8 up, 7 in 2024-03-21T14:21:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:24 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:21:25.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:25.271 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:25.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:25.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:25.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:25.557 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:25.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:25.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 12 16 23096 23080 120.198 92.375 0.00795385 0.00831203 2024-03-21T14:21:25.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:25.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.698 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:25.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:25.737 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:25 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a[29572]: 2024-03-21T14:21:25.460+0000 7f8851090700 -1 mon.a@0(leader).osd e422 definitely_dead 0 2024-03-21T14:21:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:marked in osd.6. 2024-03-21T14:21:25.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:25.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:25.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:25.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:25.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:26.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3366437' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-21T14:21:26.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:25 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3366437' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-21T14:21:26.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:26.089 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:25 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3366437' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-21T14:21:26.191 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:26.214 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:26.316 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:26.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:26.441 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:26.568 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:26.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 13 16 23558 23542 113.172 28.875 0.00178668 0.00831018 2024-03-21T14:21:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.592 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:26.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:26.693 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.717 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:26.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:26.750 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:21:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:21:26.818 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:26.841 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:26.943 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:26.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:26.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-mon[29576]: pgmap v897: 158 pgs: 1 active+recovering+remapped, 157 active+clean; 1.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 874 B/s rd, 84 MiB/s wr, 1.34k op/s; 765/54288 objects misplaced (1.409%); 1.5 MiB/s, 25 objects/s recovering 2024-03-21T14:21:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3366437' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-21T14:21:27.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-mon[29576]: osdmap e423: 8 total, 8 up, 8 in 2024-03-21T14:21:27.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-mon[40510]: pgmap v897: 158 pgs: 1 active+recovering+remapped, 157 active+clean; 1.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 874 B/s rd, 84 MiB/s wr, 1.34k op/s; 765/54288 objects misplaced (1.409%); 1.5 MiB/s, 25 objects/s recovering 2024-03-21T14:21:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3366437' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-21T14:21:27.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:26 smithi003 ceph-mon[40510]: osdmap e423: 8 total, 8 up, 8 in 2024-03-21T14:21:27.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:27.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:27.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:26 smithi118 ceph-mon[33862]: pgmap v897: 158 pgs: 1 active+recovering+remapped, 157 active+clean; 1.1 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 874 B/s rd, 84 MiB/s wr, 1.34k op/s; 765/54288 objects misplaced (1.409%); 1.5 MiB/s, 25 objects/s recovering 2024-03-21T14:21:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3366437' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-21T14:21:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:26 smithi118 ceph-mon[33862]: osdmap e423: 8 total, 8 up, 8 in 2024-03-21T14:21:27.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:27.216 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:27.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:27.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:27.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:27.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:27.467 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:27.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:27.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:27.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 14 16 23558 23542 105.089 0 - 0.00831018 2024-03-21T14:21:27.593 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:27.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:27.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:21:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:21:27.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:27.719 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:27.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:27.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:27.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:27.973 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:27.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:27 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:21:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:27 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:21:28.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:27 smithi003 ceph-mon[29576]: osdmap e424: 8 total, 8 up, 8 in 2024-03-21T14:21:28.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:27 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:21:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:27 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:21:28.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:27 smithi003 ceph-mon[40510]: osdmap e424: 8 total, 8 up, 8 in 2024-03-21T14:21:28.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:28.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:28.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:27 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:21:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:27 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:21:28.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:27 smithi118 ceph-mon[33862]: osdmap e424: 8 total, 8 up, 8 in 2024-03-21T14:21:28.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:28.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:28.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:28.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:28.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:28.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:28.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:28.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 15 16 23965 23949 99.7784 12.7188 0.00209034 0.00984855 2024-03-21T14:21:28.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:28.632 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:28.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:28.758 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:28.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:28.884 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:28.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:28.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:28.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:29.027 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:29.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:29.158 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:28 smithi118 ceph-mon[33862]: pgmap v900: 158 pgs: 1 remapped+peering, 1 active+recovering+remapped, 156 active+clean; 1.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s; 765/57735 objects misplaced (1.325%); 1.5 MiB/s, 26 objects/s recovering 2024-03-21T14:21:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:28 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:21:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:28 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:21:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:28 smithi118 ceph-mon[33862]: osdmap e425: 8 total, 8 up, 8 in 2024-03-21T14:21:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[29576]: pgmap v900: 158 pgs: 1 remapped+peering, 1 active+recovering+remapped, 156 active+clean; 1.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s; 765/57735 objects misplaced (1.325%); 1.5 MiB/s, 26 objects/s recovering 2024-03-21T14:21:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:21:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:21:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[29576]: osdmap e425: 8 total, 8 up, 8 in 2024-03-21T14:21:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[40510]: pgmap v900: 158 pgs: 1 remapped+peering, 1 active+recovering+remapped, 156 active+clean; 1.2 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 94 MiB/s wr, 1.50k op/s; 765/57735 objects misplaced (1.325%); 1.5 MiB/s, 26 objects/s recovering 2024-03-21T14:21:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:21:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:21:29.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:28 smithi003 ceph-mon[40510]: osdmap e425: 8 total, 8 up, 8 in 2024-03-21T14:21:29.260 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:29.292 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:29.394 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:29.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:29.565 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:29.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:29.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 16 15 25197 25182 98.3583 77.0625 0.0152538 0.0101599 2024-03-21T14:21:29.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:29.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:29.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:29.825 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:29.927 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:29.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:29.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:30.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:30.100 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:30.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:29 smithi118 ceph-mon[33862]: osdmap e426: 8 total, 8 up, 8 in 2024-03-21T14:21:30.202 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:30.241 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:29 smithi003 ceph-mon[29576]: osdmap e426: 8 total, 8 up, 8 in 2024-03-21T14:21:30.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:29 smithi003 ceph-mon[40510]: osdmap e426: 8 total, 8 up, 8 in 2024-03-21T14:21:30.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:30.376 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:30.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:30.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:30.516 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:30.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 17 16 26762 26746 98.3221 97.75 0.00646837 0.0101641 2024-03-21T14:21:30.619 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:30.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.649 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:30.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:30.750 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:30.779 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:30.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:30.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:30.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:31.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:31.052 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:31.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:31.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:30 smithi118 ceph-mon[33862]: pgmap v903: 158 pgs: 1 activating+undersized+remapped, 1 remapped+peering, 1 active+remapped, 155 active+clean; 1.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 1456/71232 objects degraded (2.044%); 12 MiB/s, 194 objects/s recovering 2024-03-21T14:21:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:30 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:21:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:30 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:21:31.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:31.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:30 smithi003 ceph-mon[29576]: pgmap v903: 158 pgs: 1 activating+undersized+remapped, 1 remapped+peering, 1 active+remapped, 155 active+clean; 1.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 1456/71232 objects degraded (2.044%); 12 MiB/s, 194 objects/s recovering 2024-03-21T14:21:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:30 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:21:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:30 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:21:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:30 smithi003 ceph-mon[40510]: pgmap v903: 158 pgs: 1 activating+undersized+remapped, 1 remapped+peering, 1 active+remapped, 155 active+clean; 1.4 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.41k op/s; 1456/71232 objects degraded (2.044%); 12 MiB/s, 194 objects/s recovering 2024-03-21T14:21:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:30 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:21:31.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:30 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:21:31.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.334 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:31.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:31.435 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:31.466 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:31.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:31.567 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:31.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 18 15 28445 28430 98.7064 105.25 0.0076168 0.0101243 2024-03-21T14:21:31.600 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:31.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:31.702 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:31.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:31.833 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:31.865 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:31.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:31.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:31.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:31.999 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:32.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:32.142 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:32.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:32.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:32.274 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:32.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:32.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:32.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:32.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:32.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:32.588 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-21T14:21:32.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 19 16 30138 30122 99.0767 105.75 0.00733188 0.0100863 2024-03-21T14:21:32.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:32.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:32.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:32.800 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:32.840 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:32.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:32.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:32.982 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:32.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:33.046 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:32 smithi003 ceph-mon[29576]: pgmap v904: 158 pgs: 1 activating+undersized+remapped, 1 remapped+peering, 1 active+remapped, 155 active+clean; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 940 B/s rd, 65 MiB/s wr, 1.04k op/s; 1456/71232 objects degraded (2.044%); 8.8 MiB/s, 142 objects/s recovering 2024-03-21T14:21:33.046 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:33.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:32 smithi003 ceph-mon[40510]: pgmap v904: 158 pgs: 1 activating+undersized+remapped, 1 remapped+peering, 1 active+remapped, 155 active+clean; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 940 B/s rd, 65 MiB/s wr, 1.04k op/s; 1456/71232 objects degraded (2.044%); 8.8 MiB/s, 142 objects/s recovering 2024-03-21T14:21:33.049 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:33.084 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:33.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.138 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:33.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:32 smithi118 ceph-mon[33862]: pgmap v904: 158 pgs: 1 activating+undersized+remapped, 1 remapped+peering, 1 active+remapped, 155 active+clean; 1.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 940 B/s rd, 65 MiB/s wr, 1.04k op/s; 1456/71232 objects degraded (2.044%); 8.8 MiB/s, 142 objects/s recovering 2024-03-21T14:21:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:33.238 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:33.266 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:33.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:33.406 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:33.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:33.547 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:33.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:33.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:21:33.591582+0000 min lat: 0.00125552 max lat: 2.43115 avg lat: 0.0100903 2024-03-21T14:21:33.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:21:33.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 20 16 31711 31695 99.038 98.3125 0.00742865 0.0100903 2024-03-21T14:21:33.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:33.679 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:33.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:33.811 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:33.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:33.942 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:33.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:33.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:33 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:21:33.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:33 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:21:33.964 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:33 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:21:33.964 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:33 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:21:33.965 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:33 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:21:33.965 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:33 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:21:34.044 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:34.069 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:34.087 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:33 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:21:34.087 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:33 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:21:34.087 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:33 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:21:34.171 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:34.195 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:34.296 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:34.320 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:34.345 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:21:34.345 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:21:34.345 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:21:34.366 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:21:34.387 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:21:34.409 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:21:34.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:34.431 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:21:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:34.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:34.451 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:21:34.453 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:21:34.551 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:34.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:34.590 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 21 16 32215 32199 95.8217 31.5 0.00230018 0.0100681 2024-03-21T14:21:34.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:34.702 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:34.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:34.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:34.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:34.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:34.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:34.997 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:21:34.997 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:21:35.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: pgmap v905: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 1479/90252 objects misplaced (1.639%); 21 MiB/s, 334 objects/s recovering 2024-03-21T14:21:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:21:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: osdmap e427: 8 total, 8 up, 8 in 2024-03-21T14:21:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:21:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:21:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1855099924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:35.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:34 smithi118 ceph-mon[33862]: osdmap e428: 8 total, 8 up, 8 in 2024-03-21T14:21:35.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: pgmap v905: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 1479/90252 objects misplaced (1.639%); 21 MiB/s, 334 objects/s recovering 2024-03-21T14:21:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:21:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: osdmap e427: 8 total, 8 up, 8 in 2024-03-21T14:21:35.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1855099924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[29576]: osdmap e428: 8 total, 8 up, 8 in 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: pgmap v905: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.6 KiB/s rd, 116 MiB/s wr, 1.86k op/s; 1479/90252 objects misplaced (1.639%); 21 MiB/s, 334 objects/s recovering 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: osdmap e427: 8 total, 8 up, 8 in 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1855099924' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:35.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:34 smithi003 ceph-mon[40510]: osdmap e428: 8 total, 8 up, 8 in 2024-03-21T14:21:35.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:35.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:35.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:35.390 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:35.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:35.530 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:35.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 22 16 33528 33512 95.196 82.0625 0.0131048 0.0104967 2024-03-21T14:21:35.632 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:35.673 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:35.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:35.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:35.950 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:35.987 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:35.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:36.089 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.121 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:36.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:35 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:21:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:35 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:21:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:35 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:21:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:35 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:21:36.195 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:21:36.195 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:21:36.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:21:36.196 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:21:36.196 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:21:36.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:21:36.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:21:36.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:35 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:21:36.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:36.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.267 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:36.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:36.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:36.408 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:36.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:36.538 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:36.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 23 16 35171 35155 95.5213 102.688 0.00650401 0.0104618 2024-03-21T14:21:36.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T14:21:36.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.709 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:36.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:21:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:21:36.811 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:36.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.837 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:36.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:36.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:36.996 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:36.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:37.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:37.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.124 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:37.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:36 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:21:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:36 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:21:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:36 smithi118 ceph-mon[33862]: pgmap v908: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 1.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 794 B/s rd, 62 MiB/s wr, 985 op/s; 1479/90252 objects misplaced (1.639%); 13 MiB/s, 214 objects/s recovering 2024-03-21T14:21:37.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:21:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:21:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-mon[29576]: pgmap v908: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 1.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 794 B/s rd, 62 MiB/s wr, 985 op/s; 1479/90252 objects misplaced (1.639%); 13 MiB/s, 214 objects/s recovering 2024-03-21T14:21:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:21:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:21:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:36 smithi003 ceph-mon[40510]: pgmap v908: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 1.8 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 794 B/s rd, 62 MiB/s wr, 985 op/s; 1479/90252 objects misplaced (1.639%); 13 MiB/s, 214 objects/s recovering 2024-03-21T14:21:37.271 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:37.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:37.373 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:37.402 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:37.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:37.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:37.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:37.589 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:21:37.589 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:21:37.589 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-21T14:21:37.590 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T14:21:37.590 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T14:21:37.590 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-21T14:21:37.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 24 16 36665 36649 95.4315 93.375 0.00389008 0.010473 2024-03-21T14:21:37.650 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:37.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:37.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:21:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:21:37.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:37.807 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:37.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:37.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:21:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:21:37.909 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:37.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:37.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:37.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:38.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.090 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:38.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:38.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:37 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:21:38.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:37 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:21:38.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:37 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:21:38.134 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:37 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:21:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:37 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:21:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:37 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:21:38.192 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:38.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:38.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:38.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:38.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:38.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:38.490 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:38.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 25 16 37982 37966 94.9065 82.3125 0.019723 0.0105307 2024-03-21T14:21:38.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:38.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:38.730 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:38.762 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:38.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:38.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:38.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:38.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:38.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:38.991 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:21:39.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[29576]: pgmap v909: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s; 1022/95274 objects misplaced (1.073%); 14 MiB/s, 229 objects/s recovering 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[40510]: pgmap v909: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s; 1022/95274 objects misplaced (1.073%); 14 MiB/s, 229 objects/s recovering 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:21:39.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:38 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:21:39.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:39.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:39.141 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:39.146 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:38 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:21:39.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:38 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:21:39.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:38 smithi118 ceph-mon[33862]: pgmap v909: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 1.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s; 1022/95274 objects misplaced (1.073%); 14 MiB/s, 229 objects/s recovering 2024-03-21T14:21:39.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:39.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:38 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:21:39.147 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:38 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:21:39.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:39.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:39.286 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:39.328 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:39.429 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:39.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:39.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:39.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 26 16 39529 39513 94.9747 96.6875 0.00880349 0.0105233 2024-03-21T14:21:39.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:39.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:39.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:39.752 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:39.763 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:39.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:39.768 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:39.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:39.894 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:39.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:39.996 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:40.036 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:40.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:40.137 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:40.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:39 smithi118 ceph-mon[33862]: osdmap e429: 8 total, 8 up, 8 in 2024-03-21T14:21:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:39 smithi118 ceph-mon[33862]: osdmap e430: 8 total, 8 up, 8 in 2024-03-21T14:21:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:39 smithi003 ceph-mon[29576]: osdmap e429: 8 total, 8 up, 8 in 2024-03-21T14:21:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:39 smithi003 ceph-mon[29576]: osdmap e430: 8 total, 8 up, 8 in 2024-03-21T14:21:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:39 smithi003 ceph-mon[40510]: osdmap e429: 8 total, 8 up, 8 in 2024-03-21T14:21:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:39 smithi003 ceph-mon[40510]: osdmap e430: 8 total, 8 up, 8 in 2024-03-21T14:21:40.269 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:21:40.269 INFO:teuthology.orchestra.run.smithi118.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-21T14:21:40.269 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:21:40.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:40.309 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:40.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:40.410 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:40.440 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:40.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:40.572 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:40.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 27 16 40736 40720 94.2508 75.4375 0.0065253 0.010604 2024-03-21T14:21:40.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:40.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:40.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:40.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:40.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:40.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:40.985 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:40.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:40.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:40 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:21:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:40 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:21:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:40 smithi003 ceph-mon[29576]: pgmap v911: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 382/109668 objects misplaced (0.348%); 11 MiB/s, 182 objects/s recovering 2024-03-21T14:21:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:40 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:21:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:40 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:21:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:40 smithi003 ceph-mon[40510]: pgmap v911: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 382/109668 objects misplaced (0.348%); 11 MiB/s, 182 objects/s recovering 2024-03-21T14:21:41.087 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:41.113 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:40 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:21:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:40 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:21:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:40 smithi118 ceph-mon[33862]: pgmap v911: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 382/109668 objects misplaced (0.348%); 11 MiB/s, 182 objects/s recovering 2024-03-21T14:21:41.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.240 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:41.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:41.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.365 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:41.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:41.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:41.493 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:41.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 28 16 41383 41367 92.3288 40.4375 0.00212474 0.0106029 2024-03-21T14:21:41.595 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:41.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:41.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:41.720 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:41.745 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:41.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:41.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:41.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:41.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:41.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:41.972 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:21:42.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:42.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:42.144 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:41 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:21:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:41 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:21:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:41 smithi118 ceph-mon[33862]: osdmap e431: 8 total, 8 up, 8 in 2024-03-21T14:21:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:41 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:21:42.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:41 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:21:42.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:41 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3446065729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:21:42.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:21:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:21:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[29576]: osdmap e431: 8 total, 8 up, 8 in 2024-03-21T14:21:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:21:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:21:42.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3446065729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:21:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:21:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:21:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[40510]: osdmap e431: 8 total, 8 up, 8 in 2024-03-21T14:21:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:21:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:21:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:41 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3446065729' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:21:42.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:42.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:42.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:42.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:42.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:42.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 29 16 42258 42242 91.0306 54.6875 0.0574234 0.0109616 2024-03-21T14:21:42.651 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:42.680 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:42.782 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:42.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:42.922 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:42.946 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:42.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:43.047 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:43.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:43.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:21:43.175 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: pgmap v914: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 382/109668 objects misplaced (0.348%); 11 MiB/s, 182 objects/s recovering 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3446065729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: osdmap e432: 8 total, 8 up, 8 in 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:21:43.176 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: pgmap v914: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 382/109668 objects misplaced (0.348%); 11 MiB/s, 182 objects/s recovering 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3446065729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: osdmap e432: 8 total, 8 up, 8 in 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:21:43.177 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:42 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:21:43.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:21:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:21:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: pgmap v914: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s; 382/109668 objects misplaced (0.348%); 11 MiB/s, 182 objects/s recovering 2024-03-21T14:21:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:21:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3446065729' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: osdmap e432: 8 total, 8 up, 8 in 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:21:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:42 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:21:43.220 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:43.322 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:43.348 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:43.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:43.450 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:43.484 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:43.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:43.585 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:43.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 30 16 43775 43759 91.1564 94.8125 0.0119291 0.0109645 2024-03-21T14:21:43.624 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:43.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:43.770 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:43.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:43.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:43.909 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:43.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:43.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:44.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.053 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:44.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:44.154 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:44.194 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:44.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:44.296 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:44.335 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:44.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:44.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.474 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:44.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:44.575 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:44.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 31 16 45184 45168 91.0564 88.0625 0.00871126 0.0109763 2024-03-21T14:21:44.606 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:44.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:44.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:44.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:44.848 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:44.875 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:44.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:44.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:44.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:45.006 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:45.108 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:45.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[29576]: pgmap v916: 158 pgs: 158 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 5.5 MiB/s, 88 objects/s recovering 2024-03-21T14:21:45.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:21:45.128 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[40510]: pgmap v916: 158 pgs: 158 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 5.5 MiB/s, 88 objects/s recovering 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:21:45.129 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:44 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:21:45.149 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:44 smithi118 ceph-mon[33862]: pgmap v916: 158 pgs: 158 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 5.5 MiB/s, 88 objects/s recovering 2024-03-21T14:21:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:44 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:21:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:44 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:21:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:44 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:21:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:44 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:21:45.251 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:45.283 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:45.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:45.411 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:45.513 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:45.538 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:45.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 32 16 46674 46658 91.1208 93.125 0.00709831 0.0109675 2024-03-21T14:21:45.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:45.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:45.780 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:45.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:45.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:45.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:45.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:46.011 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:21:46.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:46.082 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:46.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:46.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:46.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:46.332 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:46.361 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:46.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:46.463 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:46.494 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:46.591 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 33 16 47901 47885 90.6832 76.6875 0.00977415 0.0110218 2024-03-21T14:21:46.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:46.641 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:46.743 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:21:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:21:46.771 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:46.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:46.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:46.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:47.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:47.038 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:47.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:47.180 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[29576]: pgmap v917: 158 pgs: 158 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 958 op/s; 4.0 MiB/s, 63 objects/s recovering 2024-03-21T14:21:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:21:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:21:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:21:47.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:21:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[40510]: pgmap v917: 158 pgs: 158 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 958 op/s; 4.0 MiB/s, 63 objects/s recovering 2024-03-21T14:21:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:21:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:21:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:21:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:46 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:21:47.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:47.325 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:47.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:47.411 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:46 smithi118 ceph-mon[33862]: pgmap v917: 158 pgs: 158 active+clean; 2.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 60 MiB/s wr, 958 op/s; 4.0 MiB/s, 63 objects/s recovering 2024-03-21T14:21:47.412 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:46 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:21:47.412 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:46 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:21:47.412 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:46 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:21:47.412 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:46 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:21:47.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:47.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:47.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:47.588 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:47.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:47.592 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 34 16 49227 49211 90.4533 82.875 0.0033665 0.0110494 2024-03-21T14:21:47.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:21:47.690 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:47.718 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:47.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:47.846 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:47.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:47.976 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:47.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:48.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:48.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:48.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:21:48.146 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:21:48.146 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-21T14:21:48.146 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.938484 2024-03-21T14:21:48.147 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 0.9384843187712272 2024-03-21T14:21:48.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:21:48.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:21:48.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:48.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:48.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:21:48.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:21:48.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:21:48.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:21:48.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:48.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:48.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:21:48.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:47 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:21:48.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:48.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:48.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:48.377 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:48.403 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:48.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:21:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:21:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:21:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:21:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:21:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:47 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:21:48.505 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:48.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:48.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:48.592 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 35 16 50565 50549 90.2562 83.625 0.00999134 0.0110737 2024-03-21T14:21:48.665 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:48.706 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:48.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:48.846 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:48.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:48.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:48.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:48.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:49.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:49.122 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:49.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[29576]: pgmap v918: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 683 B/s rd, 62 MiB/s wr, 997 op/s; 3.2 MiB/s, 50 objects/s recovering 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[29576]: 4.7 deep-scrub starts 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[29576]: 4.7 deep-scrub ok 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[40510]: pgmap v918: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 683 B/s rd, 62 MiB/s wr, 997 op/s; 3.2 MiB/s, 50 objects/s recovering 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[40510]: 4.7 deep-scrub starts 2024-03-21T14:21:49.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:48 smithi003 ceph-mon[40510]: 4.7 deep-scrub ok 2024-03-21T14:21:49.224 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:49.251 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:49.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:49.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:49.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:49.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:49.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:48 smithi118 ceph-mon[33862]: pgmap v918: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 683 B/s rd, 62 MiB/s wr, 997 op/s; 3.2 MiB/s, 50 objects/s recovering 2024-03-21T14:21:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:48 smithi118 ceph-mon[33862]: 4.7 deep-scrub starts 2024-03-21T14:21:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:48 smithi118 ceph-mon[33862]: 4.7 deep-scrub ok 2024-03-21T14:21:49.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:49.521 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:49.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:49.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:49.593 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 36 16 51910 51894 90.084 84.0625 0.0120271 0.0110952 2024-03-21T14:21:49.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:49.667 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:49.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:49.757 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:21:49.768 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:49.801 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:21:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:49.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:49.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:49.853 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:21:49.877 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:21:49.877 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250798 10.9 10:968a5cf1:::benchmark_data_smithi003_160672_object97422:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.736338+0000", 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 129.13630182099999, 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.053511334, 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.878 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250798 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736338+0000", 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736338+0000", 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.879 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736339+0000", 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.689e-06 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736344+0000", 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1250000000000001e-06 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736356+0000", 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1440000000000001e-05 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.880 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736372+0000", 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6345e-05 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736675+0000", 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030272300000000002 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736685+0000", 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6549999999999999e-06 2024-03-21T14:21:49.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.789762+0000", 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0530769639999999 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.789849+0000", 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7392999999999995e-05 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.882 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250817 10.9 10:988ed933:::benchmark_data_smithi003_160672_object97441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.741969+0000", 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 129.13067015199999, 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.056560226, 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250817 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.883 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741969+0000", 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741969+0000", 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741971+0000", 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.542e-06 2024-03-21T14:21:49.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741985+0000", 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3771000000000001e-05 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742001+0000", 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5928999999999999e-05 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.885 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742009+0000", 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9069999999999996e-06 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742065+0000", 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5371000000000002e-05 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742075+0000", 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8309999999999996e-06 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.886 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.790296+0000", 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.048221058 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.790474+0000", 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178025 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.790922+0000", 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000448007 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.887 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798262+0000", 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073399989999999998 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798381+0000", 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118768 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798419+0000", 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8210000000000002e-05 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.888 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798473+0000", 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4722999999999998e-05 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798530+0000", 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6085000000000003e-05 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.889 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250819 10.9 10:9b87e9d2:::benchmark_data_smithi003_160672_object97443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.743601+0000", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 129.12903839500001, 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0550646159999999, 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250819 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743601+0000", 2024-03-21T14:21:49.890 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743601+0000", 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743602+0000", 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.052e-06 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.891 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743610+0000", 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5719999999999997e-06 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743621+0000", 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1430000000000001e-05 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743630+0000", 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4919999999999993e-06 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.892 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743695+0000", 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5195000000000002e-05 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743703+0000", 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0290000000000005e-06 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.793749+0000", 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.050046029 2024-03-21T14:21:49.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.793880+0000", 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013149 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.794024+0000", 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014386600000000001 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798328+0000", 2024-03-21T14:21:49.894 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043040869999999998 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798542+0000", 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021386500000000001 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798580+0000", 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8172e-05 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798627+0000", 2024-03-21T14:21:49.895 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6836999999999999e-05 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798666+0000", 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8500000000000001e-05 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250826 10.9 10:96022229:::benchmark_data_smithi003_160672_object97450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.752788+0000", 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 129.11985175000001, 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0512797460000001, 2024-03-21T14:21:49.896 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250826 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752788+0000", 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.897 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752788+0000", 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752789+0000", 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4670000000000001e-06 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752794+0000", 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.498e-06 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.898 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752809+0000", 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5284000000000001e-05 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752818+0000", 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0850000000000003e-06 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752981+0000", 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163287 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.899 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752991+0000", 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6350000000000001e-06 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.796127+0000", 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0431358509999999 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.796244+0000", 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011756000000000001 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.900 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.796365+0000", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120338 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798676+0000", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023114519999999999 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798715+0000", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8964999999999997e-05 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.901 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.803956+0000", 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052408039999999999 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.804019+0000", 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3431999999999998e-05 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.804067+0000", 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8087999999999998e-05 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.902 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276756 10.a 10:5a01975e:::benchmark_data_smithi003_160672_object123380:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.951630+0000", 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.92100904999999, 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0536734160000001, 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276756 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.903 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951630+0000", 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951630+0000", 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.904 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951631+0000", 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9999999999999996e-07 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.905 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951636+0000", 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0880000000000002e-06 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951649+0000", 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2384e-05 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951657+0000", 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6789999999999994e-06 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.906 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.963113+0000", 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011455051000000001 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.963120+0000", 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2479999999999999e-06 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.966072+0000", 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029518650000000001 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.907 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.966083+0000", 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.097e-05 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005133+0000", 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.039050142 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.908 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005304+0000", 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171089 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276757 10.a 10:5609b524:::benchmark_data_smithi003_160672_object123381:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.951674+0000", 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.920965068, 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.05404067, 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.909 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276757 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951674+0000", 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951674+0000", 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951675+0000", 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8100000000000003e-07 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951678+0000", 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.881e-06 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951684+0000", 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5869999999999998e-06 2024-03-21T14:21:49.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951692+0000", 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3180000000000001e-06 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969235+0000", 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017543751 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969244+0000", 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1589999999999993e-06 2024-03-21T14:21:49.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005666+0000", 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.036422465 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005715+0000", 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9027999999999999e-05 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.913 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276763 10.a 10:55e751fa:::benchmark_data_smithi003_160672_object123387:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.92093961800001, 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0682812049999999, 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276763 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.914 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3000000000000001e-07 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951703+0000", 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2259999999999999e-06 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951708+0000", 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7610000000000004e-06 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951714+0000", 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2480000000000004e-06 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.916 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969268+0000", 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017554311 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969273+0000", 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6620000000000002e-06 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005800+0000", 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0365267819999999 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.917 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005888+0000", 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7812999999999994e-05 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.019908+0000", 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014019723 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.019981+0000", 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3249000000000003e-05 2024-03-21T14:21:49.918 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276784 10.a 10:58b60665:::benchmark_data_smithi003_160672_object123408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.971287+0000", 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.901352962, 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0421002779999999, 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.919 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276784 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971287+0000", 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971287+0000", 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.920 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971288+0000", 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5600000000000001e-06 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971297+0000", 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3209999999999994e-06 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971308+0000", 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0791000000000001e-05 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.921 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971317+0000", 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4980000000000006e-06 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971370+0000", 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3062999999999997e-05 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971382+0000", 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2031000000000001e-05 2024-03-21T14:21:49.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005911+0000", 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0345289339999999 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.006009+0000", 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8376999999999997e-05 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.006443+0000", 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043439599999999998 2024-03-21T14:21:49.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.011607+0000", 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051635889999999997 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013084+0000", 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014772990000000001 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013233+0000", 2024-03-21T14:21:49.924 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014816300000000001 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013291+0000", 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8937999999999997e-05 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013387+0000", 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5317999999999998e-05 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.925 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276785 10.a 10:563d6131:::benchmark_data_smithi003_160672_object123409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.971785+0000", 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.900854147, 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0526706690000001, 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276785 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.926 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971785+0000", 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971785+0000", 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971786+0000", 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9299999999999996e-07 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.927 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971791+0000", 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.831e-06 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971802+0000", 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0675e-05 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971810+0000", 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.943e-06 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.928 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971862+0000", 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1919999999999998e-05 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971870+0000", 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2279999999999998e-06 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013436+0000", 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0415666139999999 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.929 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013561+0000", 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012478000000000001 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013641+0000", 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0142999999999995e-05 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.019676+0000", 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060351420000000003 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.930 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020014+0000", 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033725200000000002 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024299+0000", 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042853709999999996 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024349+0000", 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0249e-05 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024456+0000", 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106628 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276810 10.a 10:595ca004:::benchmark_data_smithi003_160672_object123434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.996981+0000", 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.875658703, 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.027762557, 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.932 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276810 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996981+0000", 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996981+0000", 2024-03-21T14:21:49.933 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996982+0000", 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2300000000000001e-06 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996987+0000", 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4199999999999998e-06 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.934 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996998+0000", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1015e-05 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.997007+0000", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4689999999999997e-06 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.997196+0000", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000189192 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.997206+0000", 2024-03-21T14:21:49.935 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.995e-06 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.015289+0000", 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.018083045 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.015422+0000", 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013287600000000001 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.015557+0000", 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013471300000000001 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024466+0000", 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089097580000000003 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024517+0000", 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0138e-05 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024582+0000", 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5592999999999994e-05 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.937 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024627+0000", 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4487000000000003e-05 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024743+0000", 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116626 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276838 10.a 10:51795154:::benchmark_data_smithi003_160672_object123462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.018961+0000", 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.853678712, 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.006085809, 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276838 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.939 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018961+0000", 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018961+0000", 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018962+0000", 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2500000000000001e-06 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.940 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018967+0000", 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2159999999999997e-06 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018979+0000", 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2109e-05 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018988+0000", 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5599999999999994e-06 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.941 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.019049+0000", 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1373999999999999e-05 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.019059+0000", 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5719999999999992e-06 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.017466+0000", 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99840741600000005 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.942 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.017571+0000", 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105098 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.017686+0000", 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114448 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024473+0000", 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067870939999999996 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.943 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024548+0000", 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5094999999999998e-05 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024896+0000", 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034793900000000002 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024931+0000", 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5045000000000001e-05 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.944 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:49.946 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:21:49.946 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.946 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025047+0000", 2024-03-21T14:21:49.946 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115593 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276844 10.a 10:5e872913:::benchmark_data_smithi003_160672_object123468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.84678095, 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99941103799999997, 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.947 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276844 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.948 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3799999999999996e-07 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025864+0000", 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5329999999999996e-06 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025873+0000", 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6670000000000002e-06 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025880+0000", 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5059999999999998e-06 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025944+0000", 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4313000000000006e-05 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025953+0000", 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9809999999999992e-06 2024-03-21T14:21:49.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020032+0000", 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99407909500000002 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020142+0000", 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010915 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020275+0000", 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013345800000000001 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024789+0000", 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045142899999999998 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024851+0000", 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1876000000000002e-05 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025143+0000", 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029137500000000002 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.952 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025181+0000", 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8148999999999997e-05 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025270+0000", 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8807000000000005e-05 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276859 10.a 10:5e9d8ab9:::benchmark_data_smithi003_160672_object123483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.953 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.042891+0000", 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.829748061, 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99221122399999995, 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276859 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.954 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042891+0000", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042891+0000", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042892+0000", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2300000000000001e-07 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042895+0000", 2024-03-21T14:21:49.955 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8660000000000002e-06 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042902+0000", 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0040000000000002e-06 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042908+0000", 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0469999999999999e-06 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.043025+0000", 2024-03-21T14:21:49.956 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011754 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.043032+0000", 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1080000000000004e-06 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.027880+0000", 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98484717399999999 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.957 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.028056+0000", 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017604299999999999 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.028177+0000", 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120949 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034673+0000", 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064963210000000002 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.958 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034918+0000", 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024549199999999999 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034966+0000", 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7184000000000001e-05 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.035013+0000", 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7333000000000002e-05 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.959 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.035103+0000", 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9640000000000002e-05 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276865 10.a 10:5935ebfb:::benchmark_data_smithi003_160672_object123489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.046907+0000", 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 103.825732872, 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99829890899999996, 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.960 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276865 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046907+0000", 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046907+0000", 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.961 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046908+0000", 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2100000000000001e-06 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046913+0000", 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4550000000000003e-06 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046923+0000", 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2909999999999996e-06 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.962 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046939+0000", 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6055000000000001e-05 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046992+0000", 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3052e-05 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.047002+0000", 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9229999999999997e-06 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.963 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.030224+0000", 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98322199099999996 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.030332+0000", 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108626 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.030446+0000", 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011424799999999999 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034880+0000", 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004433233 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.035146+0000", 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026675700000000001 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.044794+0000", 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096480620000000007 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.965 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.045083+0000", 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028866800000000001 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.045206+0000", 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012233799999999999 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.966 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276950 10.3 10:c59a8dcb:::benchmark_data_smithi003_160672_object123574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:07.058797+0000", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 102.81384275800001, 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.016286622, 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276950 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058797+0000", 2024-03-21T14:21:49.967 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058797+0000", 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058798+0000", 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2619999999999999e-06 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.968 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058805+0000", 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2969999999999998e-06 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058820+0000", 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4528999999999999e-05 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058828+0000", 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5650000000000002e-06 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.969 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058897+0000", 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8546000000000004e-05 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058906+0000", 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2809999999999997e-06 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.006938+0000", 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94803164699999998 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.970 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.007121+0000", 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018341200000000001 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.007251+0000", 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013005800000000001 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009281+0000", 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020301920000000001 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.971 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.013085+0000", 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003803374 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.074976+0000", 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.061891310999999997 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.075032+0000", 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5872999999999999e-05 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.972 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.075083+0000", 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1275e-05 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276968 10.3 10:cda049ff:::benchmark_data_smithi003_160672_object123592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:07.071761+0000", 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 102.80087869800001, 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.01005374, 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.973 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276968 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071761+0000", 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071761+0000", 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071762+0000", 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.125e-06 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071767+0000", 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1279999999999999e-06 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071777+0000", 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7289999999999995e-06 2024-03-21T14:21:49.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071785+0000", 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9289999999999998e-06 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071855+0000", 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0745000000000001e-05 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071864+0000", 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7770000000000008e-06 2024-03-21T14:21:49.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009467+0000", 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93760276399999998 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009583+0000", 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116512 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009655+0000", 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1285999999999995e-05 2024-03-21T14:21:49.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.013140+0000", 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034857109999999998 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.013185+0000", 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4195e-05 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.081693+0000", 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.068508821999999997 2024-03-21T14:21:49.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.081763+0000", 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9262000000000002e-05 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.081815+0000", 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1755000000000002e-05 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23439 11.d 11:b7606483:::benchmark_data_smithi003_188959_object23438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.811936+0000", 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.060703926999999, 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3735996780000002, 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23439 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.980 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811936+0000", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811936+0000", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811937+0000", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3689999999999999e-06 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.981 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811981+0000", 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4029999999999997e-05 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811991+0000", 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0562e-05 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811998+0000", 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2249999999999997e-06 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.982 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.812063+0000", 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5409999999999999e-05 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.812076+0000", 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3271000000000001e-05 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759898+0000", 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94782209299999998 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.983 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759911+0000", 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2788e-05 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.179898+0000", 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4199869139999999 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.180053+0000", 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015448099999999999 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:21:49.984 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.180234+0000", 2024-03-21T14:21:49.985 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018103499999999999 2024-03-21T14:21:49.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.985 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184334+0000", 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041007659999999996 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184354+0000", 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9066e-05 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.986 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184409+0000", 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4957000000000003e-05 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184451+0000", 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2542e-05 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.185535+0000", 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010841690000000001 2024-03-21T14:21:49.987 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23501 11.d 11:b959049b:::benchmark_data_smithi003_188959_object23500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.847823+0000", 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 24.024816649000002, 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3399457090000002, 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23501 2024-03-21T14:21:49.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847823+0000", 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847823+0000", 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847824+0000", 2024-03-21T14:21:49.989 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2559999999999999e-06 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847847+0000", 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2459999999999998e-05 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847857+0000", 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0543000000000001e-05 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.990 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847864+0000", 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1690000000000003e-06 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847934+0000", 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9986000000000004e-05 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847944+0000", 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0173000000000001e-05 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.991 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759953+0000", 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91200873100000002 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759961+0000", 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.79e-06 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.182716+0000", 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.422755558 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.992 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.182884+0000", 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016775700000000001 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.183093+0000", 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020857499999999999 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186539+0000", 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034464610000000001 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.993 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186560+0000", 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0954999999999998e-05 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186611+0000", 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0311e-05 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186682+0000", 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1217999999999995e-05 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.187769+0000", 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010867660000000001 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23540 11.d 11:b1a32943:::benchmark_data_smithi003_188959_object23539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.877480+0000", 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 23.995159252000001, 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3302135609999999, 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:49.995 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23540 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877480+0000", 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877480+0000", 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877481+0000", 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-06 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877504+0000", 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2674e-05 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877518+0000", 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4299e-05 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.997 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877526+0000", 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6990000000000003e-06 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877589+0000", 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3091999999999998e-05 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877600+0000", 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0730999999999999e-05 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:49.998 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:49.999 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:49.999 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:49.999 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:21:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759982+0000", 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.882382377 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759989+0000", 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0210000000000002e-06 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:50.000 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.187818+0000", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.42782864 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.188000+0000", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018164200000000001 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.188187+0000", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000187484 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.192410+0000", 2024-03-21T14:21:50.001 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042227740000000003 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.196014+0000", 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036038210000000001 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.206195+0000", 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010181638 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.206251+0000", 2024-03-21T14:21:50.002 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5529000000000003e-05 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.207694+0000", 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001442902 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23556 11.d 11:b802ef67:::benchmark_data_smithi003_188959_object23555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.893032+0000", 2024-03-21T14:21:50.003 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 23.979607870999999, 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3156643900000002, 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23556 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893032+0000", 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.004 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893032+0000", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893033+0000", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.155e-06 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893051+0000", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8555000000000001e-05 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:21:50.005 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893066+0000", 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5079000000000001e-05 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893074+0000", 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4900000000000003e-06 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893128+0000", 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4055000000000003e-05 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.006 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893136+0000", 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.354e-06 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.760015+0000", 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86687844599999997 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.760023+0000", 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3459999999999994e-06 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.007 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.189354+0000", 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4293309910000001 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.189494+0000", 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013979899999999999 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.189657+0000", 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016322900000000001 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.008 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.196065+0000", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006407968 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.196114+0000", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8859000000000001e-05 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.207711+0000", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011597077000000001 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:21:50.009 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.207768+0000", 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7327999999999997e-05 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.208696+0000", 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092765899999999995 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:21:50.010 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:21:50.011 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:50.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:49 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:21:50.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:49 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:21:50.011 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1362826266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:50.012 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:50.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:49 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:21:50.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:49 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:21:50.012 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1362826266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:50.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:50.012 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:21:50.051 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:21:50.097 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:21:50.106 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:21:50.107 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:21:50.113 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:50.160 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:50.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:50.262 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:50.291 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:50.393 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:49 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:21:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:49 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:21:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1362826266' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:21:50.432 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:50.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:50.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:50.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:50.593 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 37 16 53084 53068 89.6322 73.375 0.00350496 0.0111493 2024-03-21T14:21:50.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.703 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:50.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:50.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:50.843 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:50.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:50.944 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:50.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:50.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:51.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:51.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:51.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:51.207 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:51.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: 4.16 deep-scrub starts 2024-03-21T14:21:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: 4.16 deep-scrub ok 2024-03-21T14:21:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: pgmap v919: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-21T14:21:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: 4.1c deep-scrub starts 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: 4.1c deep-scrub ok 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: 3.e deep-scrub starts 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[40510]: 3.e deep-scrub ok 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: 4.16 deep-scrub starts 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: 4.16 deep-scrub ok 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: pgmap v919: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: 4.1c deep-scrub starts 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: 4.1c deep-scrub ok 2024-03-21T14:21:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: 3.e deep-scrub starts 2024-03-21T14:21:51.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:50 smithi003 ceph-mon[29576]: 3.e deep-scrub ok 2024-03-21T14:21:51.353 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:51.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: 4.16 deep-scrub starts 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: 4.16 deep-scrub ok 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: pgmap v919: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.53k op/s; 2.9 MiB/s, 46 objects/s recovering 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: 4.1c deep-scrub starts 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: 4.1c deep-scrub ok 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: 3.e deep-scrub starts 2024-03-21T14:21:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:50 smithi118 ceph-mon[33862]: 3.e deep-scrub ok 2024-03-21T14:21:51.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:51.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:51.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:51.593 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 38 16 54457 54441 89.5315 85.8125 0.0103022 0.0111636 2024-03-21T14:21:51.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:51.662 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:51.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:51.764 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:51.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:51.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:51.951 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:51.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:52.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:52.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:52.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:52.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:52.217 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:21:52.217 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:21:52.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:21:52.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:21:52.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:21:52.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:21:52.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:21:52.218 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:51 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:51 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:51 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:51 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:51 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:21:52.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:52 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:21:52.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:52.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:52.347 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:52.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:21:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:21:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:21:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:21:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:21:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:21:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:21:52.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:51 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:21:52.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:52.522 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:52.593 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 39 16 55778 55762 89.3527 82.5625 0.0161342 0.0111863 2024-03-21T14:21:52.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:52.660 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:52.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:52.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:52.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:52.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:52.895 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:52.925 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:52.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:53.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:53.055 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:53.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:53.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:53 smithi003 ceph-mon[29576]: pgmap v920: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s; 2.4 MiB/s, 38 objects/s recovering 2024-03-21T14:21:53.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:53 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:21:53.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:53 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:21:53.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:53 smithi003 ceph-mon[40510]: pgmap v920: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s; 2.4 MiB/s, 38 objects/s recovering 2024-03-21T14:21:53.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:53 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:21:53.151 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:53 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:21:53.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:53.185 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:53.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:53.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-21T14:21:53.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:53.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:53.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:53 smithi118 ceph-mon[33862]: pgmap v920: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.27k op/s; 2.4 MiB/s, 38 objects/s recovering 2024-03-21T14:21:53.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:53 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:21:53.331 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:53 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:21:53.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:53.455 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:53.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:53.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:53.595 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:21:53.594076+0000 min lat: 0.00125552 max lat: 2.43115 avg lat: 0.0112095 2024-03-21T14:21:53.595 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:21:53.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 40 16 57085 57069 89.1607 81.6875 0.0299836 0.0112095 2024-03-21T14:21:53.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:53.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:53.720 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:53.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:21:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.775 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:53.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:53.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:53.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:53.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:54.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:54.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:54.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:54 smithi003 ceph-mon[29576]: 1.2 scrub starts 2024-03-21T14:21:54.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:54 smithi003 ceph-mon[29576]: 1.2 scrub ok 2024-03-21T14:21:54.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:54 smithi003 ceph-mon[40510]: 1.2 scrub starts 2024-03-21T14:21:54.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:54 smithi003 ceph-mon[40510]: 1.2 scrub ok 2024-03-21T14:21:54.158 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:54.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:54.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:54.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:54 smithi118 ceph-mon[33862]: 1.2 scrub starts 2024-03-21T14:21:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:54 smithi118 ceph-mon[33862]: 1.2 scrub ok 2024-03-21T14:21:54.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:54.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:54.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:54.593 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 41 16 58238 58222 88.7429 72.0625 0.013917 0.0112625 2024-03-21T14:21:54.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:54.719 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:54.744 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:54.846 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:54.871 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:54.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:54.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:54.973 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:55.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:55.114 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:55.139 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:55.241 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: pgmap v921: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-03-21T14:21:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 3.1c deep-scrub starts 2024-03-21T14:21:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 3.1c deep-scrub ok 2024-03-21T14:21:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 5.c deep-scrub starts 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[29576]: 5.c deep-scrub ok 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: pgmap v921: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 3.1c deep-scrub starts 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 3.1c deep-scrub ok 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:21:55.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:21:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 5.c deep-scrub starts 2024-03-21T14:21:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:55 smithi003 ceph-mon[40510]: 5.c deep-scrub ok 2024-03-21T14:21:55.270 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: pgmap v921: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.3 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 2.1 MiB/s, 33 objects/s recovering 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 3.1c deep-scrub starts 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 3.1c deep-scrub ok 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:21:55.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 5.c deep-scrub starts 2024-03-21T14:21:55.334 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:55 smithi118 ceph-mon[33862]: 5.c deep-scrub ok 2024-03-21T14:21:55.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:55.400 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:55.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:55.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:55.594 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 42 16 59278 59262 88.1767 65 0.0208157 0.0113352 2024-03-21T14:21:55.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:55.713 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:55.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:55.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:55.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:55.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:55.984 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:55.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:21:56.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:56.132 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:56.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:56.201 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:21:56.201 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:21:56.201 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/253374082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:21:56.201 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:21:56.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:21:56.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:21:56.202 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/253374082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:21:56.202 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:21:56.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:56.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:56.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:56.384 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:21:56.410 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:56.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:21:56.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:56 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:21:56.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:56 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:21:56.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/253374082' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:21:56.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:56 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:21:56.451 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:56.457 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:21:56.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:56.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:56.594 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 43 16 60236 60220 87.5178 59.875 0.00877353 0.0114197 2024-03-21T14:21:56.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:56.711 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:56.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:56.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:21:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:21:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:21:56.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:56.854 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:56.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:56.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:57.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.6 primary-affinity to 0.938484 (8615042) 2024-03-21T14:21:57.109 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:57.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[29576]: pgmap v922: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:21:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[29576]: osdmap e433: 8 total, 8 up, 8 in 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/914751405' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.9384843187712272}]: dispatch 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[40510]: pgmap v922: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[40510]: osdmap e433: 8 total, 8 up, 8 in 2024-03-21T14:21:57.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/914751405' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.9384843187712272}]: dispatch 2024-03-21T14:21:57.257 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:57.283 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:57.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:57.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:57 smithi118 ceph-mon[33862]: pgmap v922: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:21:57.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:57 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:21:57.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:57 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:21:57.421 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:57 smithi118 ceph-mon[33862]: osdmap e433: 8 total, 8 up, 8 in 2024-03-21T14:21:57.422 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/914751405' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.9384843187712272}]: dispatch 2024-03-21T14:21:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:57.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:57.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:21:57.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.579 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:57.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:21:57.594 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 44 16 61444 61428 87.2445 75.5 0.0127392 0.011456 2024-03-21T14:21:57.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:57.682 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:21:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:21:57.709 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:57.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:57.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:57.953 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:57.995 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:57.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:58.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:58.126 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:58.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:58.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:58.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/914751405' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.9384843187712272}]': finished 2024-03-21T14:21:58.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:58 smithi003 ceph-mon[29576]: osdmap e434: 8 total, 8 up, 8 in 2024-03-21T14:21:58.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/914751405' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.9384843187712272}]': finished 2024-03-21T14:21:58.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:58 smithi003 ceph-mon[40510]: osdmap e434: 8 total, 8 up, 8 in 2024-03-21T14:21:58.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.275 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:58.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:58.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:58.406 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/914751405' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 0.9384843187712272}]': finished 2024-03-21T14:21:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:58 smithi118 ceph-mon[33862]: osdmap e434: 8 total, 8 up, 8 in 2024-03-21T14:21:58.507 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:58.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:58.594 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 45 16 62747 62731 87.1153 81.4375 0.00724395 0.0114735 2024-03-21T14:21:58.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:58.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:21:58.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:58.838 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:58.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:58.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:21:58.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:21:58.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:58.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:58.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:21:59.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:21:59.121 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:21:59.224 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:21:59.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:59 smithi003 ceph-mon[29576]: pgmap v925: 158 pgs: 158 active+clean; 3.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-21T14:21:59.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:21:59 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:59.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:59 smithi003 ceph-mon[40510]: pgmap v925: 158 pgs: 158 active+clean; 3.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-21T14:21:59.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:21:59 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:59.262 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:59.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:21:59.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:59.391 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:59 smithi118 ceph-mon[33862]: pgmap v925: 158 pgs: 158 active+clean; 3.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 62 MiB/s wr, 997 op/s 2024-03-21T14:21:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:21:59 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:21:59.493 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:59.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:59.595 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 46 16 63935 63919 86.8355 74.25 0.0195643 0.0115099 2024-03-21T14:21:59.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:21:59.651 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:21:59.753 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:21:59.780 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:21:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:21:59.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:21:59.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:21:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:21:59.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:00.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:00.052 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:00.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:00.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:00.180 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:00.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:00.282 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:00.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:00.408 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:00 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:00 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:00 smithi118 ceph-mon[33862]: osdmap e435: 8 total, 8 up, 8 in 2024-03-21T14:22:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:00 smithi118 ceph-mon[33862]: osdmap e436: 8 total, 8 up, 8 in 2024-03-21T14:22:00.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:00.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:00.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[29576]: osdmap e435: 8 total, 8 up, 8 in 2024-03-21T14:22:00.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[29576]: osdmap e436: 8 total, 8 up, 8 in 2024-03-21T14:22:00.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:00.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:00.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[40510]: osdmap e435: 8 total, 8 up, 8 in 2024-03-21T14:22:00.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:00 smithi003 ceph-mon[40510]: osdmap e436: 8 total, 8 up, 8 in 2024-03-21T14:22:00.454 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:00.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:00.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:00.585 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:00.595 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 47 16 65156 65140 86.611 76.3125 0.0105548 0.0115407 2024-03-21T14:22:00.686 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:00.714 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:00.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:00.816 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:00.857 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:00.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:00.958 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:00.994 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:22:01.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:01.033 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:22:01.080 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:22:01.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:01.122 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:22:01.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:01.166 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:22:01.206 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:22:01.234 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:01.246 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:22:01.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:01 smithi003 ceph-mon[29576]: pgmap v927: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 943 op/s 2024-03-21T14:22:01.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:01 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1182001375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:01.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:01 smithi003 ceph-mon[40510]: pgmap v927: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 943 op/s 2024-03-21T14:22:01.247 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:01 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1182001375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:01.262 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:01.286 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:22:01.290 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:22:01.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:01.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:01 smithi118 ceph-mon[33862]: pgmap v927: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 59 MiB/s wr, 943 op/s 2024-03-21T14:22:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:01 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1182001375' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:01.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:01.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 48 16 66267 66251 86.2516 69.4375 0.00817183 0.0115883 2024-03-21T14:22:01.654 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:01.687 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:01.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:01.715 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:01.788 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:01.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:01.934 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:01.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:02.079 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:02.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:02.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:02.241 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:02.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:02.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:02.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:02.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:02.522 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:02.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:02.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 49 15 67490 67475 86.0525 76.5 0.0113782 0.0116154 2024-03-21T14:22:02.624 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:02.678 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:02.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:02.779 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:02.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:02.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:02.932 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:02.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:02.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:02.947 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:22:02.947 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:22:02.947 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T14:22:02.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T14:22:02.990 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:22:02.991 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:22:03.034 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:03.063 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:03.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:03.165 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.194 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:03.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:03.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:03.323 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:03.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:03 smithi118 ceph-mon[33862]: pgmap v929: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 988 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:03 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:22:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:03.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:03 smithi003 ceph-mon[29576]: pgmap v929: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 988 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:03 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:22:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:03 smithi003 ceph-mon[40510]: pgmap v929: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 988 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:03 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:22:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:03.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:03.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 50 16 68612 68596 85.7326 70.0625 0.00511325 0.0116591 2024-03-21T14:22:03.619 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:03.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:03.760 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:03.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:03.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:03.894 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:03.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:03.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:04.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.020 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:04.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:04.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:04.162 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:04.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:04.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:22:04.264 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:04.293 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:04.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:04.395 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:04.426 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:04.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:04.528 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:04.554 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:04.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:04.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 51 16 69677 69661 85.3566 66.5625 0.0171829 0.0117105 2024-03-21T14:22:04.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:04.697 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:04.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:04.821 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:04.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:04.963 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:04.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:04.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:05.065 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:05.093 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:05.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:05.194 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:05.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:05.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:05.319 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:05.346 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:05.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:05 smithi118 ceph-mon[33862]: pgmap v930: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T14:22:05.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.500 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:05.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:05.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:05 smithi003 ceph-mon[29576]: pgmap v930: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T14:22:05.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:05 smithi003 ceph-mon[40510]: pgmap v930: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-03-21T14:22:05.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 52 16 70725 70709 84.9746 65.5 0.00485397 0.0117613 2024-03-21T14:22:05.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:05.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:05.727 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:05.757 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:05.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:05.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:05.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:05.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:06.012 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:06.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.071 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:06.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:06.171 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:06.213 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:06.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:06.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:06.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:06 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:22:06.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:06.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:06.494 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:06 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:22:06.494 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:06 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:22:06.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:06.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 53 16 71606 71590 84.4102 55.0625 0.00594316 0.0118307 2024-03-21T14:22:06.596 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:06.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:06.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:22:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:22:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:22:06.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:06.882 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:06.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:06.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:06.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:07.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:07.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:07.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:07.175 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:07.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:07.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:07.306 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:07.408 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:07 smithi118 ceph-mon[33862]: pgmap v931: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T14:22:07.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:07.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:07 smithi003 ceph-mon[29576]: pgmap v931: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T14:22:07.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:07 smithi003 ceph-mon[40510]: pgmap v931: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-21T14:22:07.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:07.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:07.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 54 16 72666 72650 84.0738 66.25 0.0101909 0.0118889 2024-03-21T14:22:07.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:07.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:07.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:07.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:07.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:22:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:22:07.972 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:07.998 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:07.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:08.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:08.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:08.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:08.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:08.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:08.366 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:08.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:08.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:08.507 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:08.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:08.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 55 16 73742 73726 83.7678 67.25 0.00316617 0.0119317 2024-03-21T14:22:08.610 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:08.634 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:08.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:08.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:08.764 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:08.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:08.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:08.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:08.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:08.899 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:08.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:09.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:09.038 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:09.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:09.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:09.168 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:09.270 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:09.297 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:09.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:09.399 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.431 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:09.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:09 smithi118 ceph-mon[33862]: pgmap v932: 158 pgs: 158 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 629 B/s rd, 57 MiB/s wr, 911 op/s 2024-03-21T14:22:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:09 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:09 smithi003 ceph-mon[29576]: pgmap v932: 158 pgs: 158 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 629 B/s rd, 57 MiB/s wr, 911 op/s 2024-03-21T14:22:09.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:09 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:09.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:09 smithi003 ceph-mon[40510]: pgmap v932: 158 pgs: 158 active+clean; 4.2 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 629 B/s rd, 57 MiB/s wr, 911 op/s 2024-03-21T14:22:09.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:09 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:09.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:09.596 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 56 16 74687 74671 83.3266 59.0625 0.00956071 0.0119824 2024-03-21T14:22:09.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:09.684 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:09.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:09.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:09.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:09.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:09.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:09.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:09.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:09.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:10.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:10.099 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:10.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:10.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:10.345 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:10.374 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:10.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:10.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:10 smithi118 ceph-mon[33862]: pgmap v933: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:22:10.476 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:10 smithi003 ceph-mon[29576]: pgmap v933: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:22:10.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:10 smithi003 ceph-mon[40510]: pgmap v933: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-21T14:22:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.506 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:10.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:10.597 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 57 16 75682 75666 82.9557 62.1875 0.0120256 0.0120488 2024-03-21T14:22:10.607 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:10.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.650 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:10.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:10.751 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:10.780 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:10.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:10.918 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:10.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:11.020 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.063 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:11.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:11.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:11.209 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:11.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:11.338 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:11.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:11.440 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:11.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-21T14:22:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.532 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:11.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:11.597 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 58 16 76767 76751 82.6939 67.8125 0.00590594 0.0120877 2024-03-21T14:22:11.634 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:11.686 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:11.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:11.822 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:11.924 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:11.962 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:11.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:12.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:12.084 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.119 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:12.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:12.220 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:12.247 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:12.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:12.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:12.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.395 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:12.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[29576]: pgmap v934: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 891 B/s rd, 61 MiB/s wr, 980 op/s 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[29576]: pgmap v935: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 983 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[40510]: pgmap v934: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 891 B/s rd, 61 MiB/s wr, 980 op/s 2024-03-21T14:22:12.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:22:12.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:22:12.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[40510]: pgmap v935: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 983 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T14:22:12.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:12 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:22:12.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:12.524 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:12.597 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 59 16 77888 77872 82.4797 70.0625 0.0175498 0.0121187 2024-03-21T14:22:12.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:12.655 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:12 smithi118 ceph-mon[33862]: pgmap v934: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 891 B/s rd, 61 MiB/s wr, 980 op/s 2024-03-21T14:22:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:12 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:22:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:12 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:22:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:12 smithi118 ceph-mon[33862]: pgmap v935: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 983 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T14:22:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:12 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:22:12.757 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.799 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:12.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:12.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:12.940 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:12.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:13.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:13.074 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:13.176 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:13.205 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:13.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:13.307 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:13.332 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:13.434 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:13.471 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:13.572 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:13.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:22:13.598879+0000 min lat: 0.00125552 max lat: 2.43115 avg lat: 0.0121519 2024-03-21T14:22:13.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:22:13.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 60 16 78989 78973 82.2511 68.8125 0.0112916 0.0121519 2024-03-21T14:22:13.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:13.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:13.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:13.747 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:13.849 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:13.878 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:13.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:13.980 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:14.021 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:14.123 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:14.148 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:14.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:14.250 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:14.277 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:14.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:14.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.436 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:14.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:14.538 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:14.588 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:14.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:14.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 61 16 80070 80054 82.0102 67.5625 0.0164121 0.0121878 2024-03-21T14:22:14.690 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:14.717 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:22:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:14.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:14.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:14 smithi003 ceph-mon[29576]: pgmap v936: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 983 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-21T14:22:14.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3624325250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:14.823 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:14 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:14.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:14 smithi003 ceph-mon[40510]: pgmap v936: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 983 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-21T14:22:14.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:14 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3624325250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:14.824 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:14 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:14.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:14.889 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:14.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:14.991 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:15.017 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:15.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:15.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:15.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:15.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:14 smithi118 ceph-mon[33862]: pgmap v936: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 983 B/s rd, 63 MiB/s wr, 1.00k op/s 2024-03-21T14:22:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:14 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3624325250' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:14 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:15.223 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:15.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:15.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:15.422 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.463 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:15.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:15.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:15.594 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:15.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 62 15 81118 81103 81.7449 65.5625 0.0151373 0.0122279 2024-03-21T14:22:15.696 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:15.739 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:15.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:15.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:15.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:15.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:22:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:22:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[29576]: osdmap e437: 8 total, 8 up, 8 in 2024-03-21T14:22:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:22:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:22:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:15 smithi003 ceph-mon[40510]: osdmap e437: 8 total, 8 up, 8 in 2024-03-21T14:22:16.017 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:16.119 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:16.158 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:15 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:15 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:15 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:22:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:15 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:22:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:15 smithi118 ceph-mon[33862]: osdmap e437: 8 total, 8 up, 8 in 2024-03-21T14:22:16.261 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:16.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:16.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:16.420 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:16.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:16.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:16.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:16.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 63 16 82280 82264 81.599 72.5625 0.00798751 0.0122496 2024-03-21T14:22:16.664 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:16.694 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:22:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:22:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:22:16.796 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:16.845 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:16.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:16.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:16.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:16.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:17.006 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:22:17.046 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:22:17.046 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:16 smithi003 ceph-mon[29576]: pgmap v938: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T14:22:17.047 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:16 smithi003 ceph-mon[40510]: pgmap v938: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T14:22:17.077 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:17.087 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:22:17.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:17.129 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:22:17.171 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:22:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:16 smithi118 ceph-mon[33862]: pgmap v938: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-21T14:22:17.211 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:22:17.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:17.242 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:17.250 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:22:17.288 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:22:17.297 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:22:17.298 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:22:17.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:17.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.387 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:17.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:17.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:17.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:17.598 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 64 16 83375 83359 81.3933 68.4375 0.0165253 0.012281 2024-03-21T14:22:17.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.692 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:17.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:17.773 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:22:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:22:17.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:17.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:17.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:17.938 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:17.980 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:17.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:18.081 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:18.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:18.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:17 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3020195163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:18.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:18.213 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:18.254 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:17 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3020195163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:17 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3020195163' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:18.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:18.356 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:18.393 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:18.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:18.555 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:18.600 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 65 16 84501 84485 81.2215 70.375 0.00721936 0.012306 2024-03-21T14:22:18.657 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:18.686 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:18.788 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:18.921 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:18.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:18.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:18.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:18.989 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:18 smithi003 ceph-mon[29576]: pgmap v939: 158 pgs: 158 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-21T14:22:18.989 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:18.990 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:18 smithi003 ceph-mon[40510]: pgmap v939: 158 pgs: 158 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-21T14:22:19.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:19.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:19.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:18 smithi118 ceph-mon[33862]: pgmap v939: 158 pgs: 158 active+clean; 5.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s 2024-03-21T14:22:19.186 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:19.230 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:19.332 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:19.356 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:19.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.510 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:19.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:19.544 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:22:19.600 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 66 16 85644 85628 81.0732 71.4375 0.0109126 0.0123288 2024-03-21T14:22:19.611 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:19.639 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:19.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.806 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:19.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:19.907 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:19.931 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:19.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:19.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:19 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:20.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:19 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:19 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:20.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:19 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:20.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:20.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:20.117 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:22:20.118 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:22:20.118 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 438, 2024-03-21T14:22:20.118 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T14:22:20.118 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-03-21T13:56:54.530125+0000", 2024-03-21T14:22:20.118 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-03-21T14:22:19.780016+0000", 2024-03-21T14:22:20.118 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2024-03-21T14:19:13.730963+0000", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2024-03-21T14:21:25.732577+0000", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5806080, 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "nodeep-scrub", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "noscrub", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 20, 2024-03-21T14:22:20.119 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 11, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "luminous", 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "luminous", 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "squid", 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_crimson": false, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".mgr", 2024-03-21T14:22:20.120 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:00:12.925535+0000", 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.121 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 11, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 11, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 11, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 11, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 11, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "270", 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "268", 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:22:20.123 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.124 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.125 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_max": 32, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.126 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": {} 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.9099999666213989, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.9099999666213989, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.9099999666213989, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.9099999666213989, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "datapool", 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.373659+0000", 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.128 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 3, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 3, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 3, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 3, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 3, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.129 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 2, 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 68, 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.130 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.131 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.132 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 4.6500000953674316, 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 4.6500000953674316, 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.86000001430511475, 2024-03-21T14:22:20.133 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 4, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 4, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".rgw.root", 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.794831+0000", 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:22:20.134 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:22:20.135 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.136 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.136 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.136 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.136 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "66", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.137 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.138 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.139 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.190000057220459, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.190000057220459, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.5, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.5, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.140 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 4, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.log", 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:33.750394+0000", 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.141 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.142 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:22:20.143 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.144 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.145 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.146 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.190000057220459, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.190000057220459, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.5, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.5, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.147 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 5, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.control", 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:35.711498+0000", 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.148 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.149 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "70", 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.150 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.151 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.152 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5399999618530273, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5399999618530273, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.75, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.75, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.153 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 6, 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.meta", 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:37.763124+0000", 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:22:20.154 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:22:20.155 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "72", 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.156 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.157 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.159 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.9700000286102295, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.9700000286102295, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 11, 2024-03-21T14:22:20.160 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "unique_pool_4", 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:20:58.032622+0000", 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 16, 2024-03-21T14:22:20.161 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 16, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 16, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 16, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 16, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "417", 2024-03-21T14:22:20.162 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:22:20.163 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:22:20.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:22:20.165 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "rados": {} 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.630000114440918, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.630000114440918, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 3, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 3, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:22:20.166 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb", 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:22:20.167 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 8, 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 393, 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.169 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.170 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6801/930096433", 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6802/930096433", 2024-03-21T14:22:20.171 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6804/930096433", 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6803/930096433", 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d", 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.172 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 13, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 424, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2024-03-21T14:22:20.174 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.175 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6805/1526485482", 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6806/1526485482", 2024-03-21T14:22:20.176 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6808/1526485482", 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/1526485482", 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c", 2024-03-21T14:22:20.177 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 18, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 277, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 358, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 431, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 278, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.179 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.180 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6809/3011075847", 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6810/3011075847", 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6812/3011075847", 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6811/3011075847", 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.181 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c", 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 25, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 411, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.182 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.183 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6813/1918209225", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6814/1918209225", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6816/1918209225", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6815/1918209225", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.185 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5", 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 30, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 425, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.186 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6800", 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6801", 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:22:20.187 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6803", 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6802", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6800/296472230", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6801/296472230", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6803/296472230", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6802/296472230", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "bff24605-38ce-47ce-9142-4ced3766476e", 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0.0266571044921875, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 40, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 382, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.190 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6804", 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6805", 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:22:20.191 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6807", 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.192 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6806", 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6804/326385780", 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6805/326385780", 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6807/326385780", 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6806/326385780", 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.193 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "73a781e7-8660-4885-b976-2aeb8515109a", 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.2286529541015625, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0.9384765625, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 48, 2024-03-21T14:22:20.194 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 332, 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 351, 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 423, 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 333, 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6808", 2024-03-21T14:22:20.195 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6809", 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.196 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6811", 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6810", 2024-03-21T14:22:20.197 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6808/1863637416", 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6809/1863637416", 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6811/1863637416", 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6810/1863637416", 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.198 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0", 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0.6165008544921875, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 58, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 427, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:22:20.199 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6812", 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6813", 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:22:20.200 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6815", 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.201 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6814", 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6812/2128678671", 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6813/2128678671", 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6815/2128678671", 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6814/2128678671", 2024-03-21T14:22:20.202 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.203 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:22:19.351542+0000", 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:21:07.853065+0000", 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.204 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:21:37.949148+0000", 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 277 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.205 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:21:33.825521+0000", 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:21:59.001548+0000", 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:22:20.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:22:14.261001+0000", 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.207 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:21:23.546645+0000", 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 332 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:22:20.208 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:21:27.421987+0000", 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [], 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [ 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.a", 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.209 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.13", 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.4", 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:22:20.210 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 5 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.b", 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.212 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:20.213 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:19 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:20.213 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:19 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.f", 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.213 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.13", 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.2", 2024-03-21T14:22:20.214 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.4", 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 5, 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.215 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.a", 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.13", 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:22:20.216 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "11.f", 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 4 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:22:20.217 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_primaries": [], 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklist": { 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/45234419": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/384540368": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3883474825": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/1598547836": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3687155565": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3884352348": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2319227492": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2154949954": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/4162970769": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2292819669": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3411869253": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:22:20.218 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/520312891": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2267426691": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3211426112": "2024-03-22T13:57:56.576834+0000" 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "range_blocklist": {}, 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.219 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {}, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": { 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_enabled": false, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_bucket_count": 0, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "degraded_stretch_mode": 0, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "recovering_stretch_mode": 0, 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_bucket": 0 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:20.220 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:22:20.265 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:20.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:20.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.409 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:20.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:20.511 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:20.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:20.600 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 67 16 86585 86569 80.7409 58.8125 0.0156911 0.0123799 2024-03-21T14:22:20.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.690 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:20.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:20.791 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:20.820 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:20.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:20 smithi003 ceph-mon[29576]: pgmap v940: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-21T14:22:20.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:20 smithi003 ceph-mon[29576]: osdmap e438: 8 total, 8 up, 8 in 2024-03-21T14:22:20.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1217861227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:22:20.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:20 smithi003 ceph-mon[40510]: pgmap v940: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-21T14:22:20.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:20 smithi003 ceph-mon[40510]: osdmap e438: 8 total, 8 up, 8 in 2024-03-21T14:22:20.894 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1217861227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:22:20.922 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:20.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:20.947 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:20.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:21.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:21.077 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:21.179 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:20 smithi118 ceph-mon[33862]: pgmap v940: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-21T14:22:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:20 smithi118 ceph-mon[33862]: osdmap e438: 8 total, 8 up, 8 in 2024-03-21T14:22:21.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1217861227' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:22:21.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:21.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:21.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:21.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:21.487 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:21.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:21.589 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:21.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 68 16 87685 87669 80.5642 68.75 0.0140088 0.0124071 2024-03-21T14:22:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:21.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:21.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:21.784 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:21.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:21.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:21.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:22.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:22.056 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:22.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:22.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:22.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:22.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:22.286 INFO:tasks.thrashosds.thrasher:j is {'epoch': 438, 'fsid': 'c60e082a-e78a-11ee-95cd-87774f69a715', 'created': '2024-03-21T13:56:54.530125+0000', 'modified': '2024-03-21T14:22:19.780016+0000', 'last_up_change': '2024-03-21T14:19:13.730963+0000', 'last_in_change': '2024-03-21T14:21:25.732577+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T14:00:12.925535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '268', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T14:02:31.373659+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.650000095367432, 'score_stable': 4.650000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 4, 'raw_score_stable': 4, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T14:02:31.794831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T14:02:33.750394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T14:02:35.711498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T14:02:37.763124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-21T14:20:58.032622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '417', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'daf9bc3c-4e73-4379-afb6-7ee181b686fb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 393, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6801', 'nonce': 930096433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6802', 'nonce': 930096433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6804', 'nonce': 930096433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6803', 'nonce': 930096433}]}, 'public_addr': '172.21.15.3:6801/930096433', 'cluster_addr': '172.21.15.3:6802/930096433', 'heartbeat_back_addr': '172.21.15.3:6804/930096433', 'heartbeat_front_addr': '172.21.15.3:6803/930096433', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1652f24d-aa0e-471c-bf54-03814b1ee11d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 424, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6805', 'nonce': 1526485482}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6806', 'nonce': 1526485482}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6808', 'nonce': 1526485482}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6807', 'nonce': 1526485482}]}, 'public_addr': '172.21.15.3:6805/1526485482', 'cluster_addr': '172.21.15.3:6806/1526485482', 'heartbeat_back_addr': '172.21.15.3:6808/1526485482', 'heartbeat_front_addr': '172.21.15.3:6807/1526485482', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5e67d907-f8d1-4ecc-85d3-df88da5c364c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 277, 'up_from': 358, 'up_thru': 431, 'down_at': 278, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6809', 'nonce': 3011075847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6810', 'nonce': 3011075847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6812', 'nonce': 3011075847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6811', 'nonce': 3011075847}]}, 'public_addr': '172.21.15.3:6809/3011075847', 'cluster_addr': '172.21.15.3:6810/3011075847', 'heartbeat_back_addr': '172.21.15.3:6812/3011075847', 'heartbeat_front_addr': '172.21.15.3:6811/3011075847', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'ad8dabdf-fce5-4312-9e80-dbbb6ba2947c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 411, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6813', 'nonce': 1918209225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6814', 'nonce': 1918209225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6816', 'nonce': 1918209225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6815', 'nonce': 1918209225}]}, 'public_addr': '172.21.15.3:6813/1918209225', 'cluster_addr': '172.21.15.3:6814/1918209225', 'heartbeat_back_addr': '172.21.15.3:6816/1918209225', 'heartbeat_front_addr': '172.21.15.3:6815/1918209225', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b5d486b1-2cb6-46c7-8664-994af07ff7b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 425, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 296472230}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 296472230}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 296472230}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 296472230}]}, 'public_addr': '172.21.15.118:6800/296472230', 'cluster_addr': '172.21.15.118:6801/296472230', 'heartbeat_back_addr': '172.21.15.118:6803/296472230', 'heartbeat_front_addr': '172.21.15.118:6802/296472230', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bff24605-38ce-47ce-9142-4ced3766476e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0266571044921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 326385780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 326385780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 326385780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 326385780}]}, 'public_addr': '172.21.15.118:6804/326385780', 'cluster_addr': '172.21.15.118:6805/326385780', 'heartbeat_back_addr': '172.21.15.118:6807/326385780', 'heartbeat_front_addr': '172.21.15.118:6806/326385780', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '73a781e7-8660-4885-b976-2aeb8515109a', 'up': 1, 'in': 1, 'weight': 0.2286529541015625, 'primary_affinity': 0.9384765625, 'last_clean_begin': 48, 'last_clean_end': 332, 'up_from': 351, 'up_thru': 423, 'down_at': 333, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 1863637416}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 1863637416}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 1863637416}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 1863637416}]}, 'public_addr': '172.21.15.118:6808/1863637416', 'cluster_addr': '172.21.15.118:6809/1863637416', 'heartbeat_back_addr': '172.21.15.118:6811/1863637416', 'heartbeat_front_addr': '172.21.15.118:6810/1863637416', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3e95559b-79f6-40ba-9d98-d350bcdd60e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6165008544921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 427, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 2128678671}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 2128678671}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 2128678671}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 2128678671}]}, 'public_addr': '172.21.15.118:6812/2128678671', 'cluster_addr': '172.21.15.118:6813/2128678671', 'heartbeat_back_addr': '172.21.15.118:6815/2128678671', 'heartbeat_front_addr': '172.21.15.118:6814/2128678671', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:22:19.351542+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:21:07.853065+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:21:37.949148+0000', 'dead_epoch': 277}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:21:33.825521+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:21:59.001548+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:22:14.261001+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:21:23.546645+0000', 'dead_epoch': 332}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:21:27.421987+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.13', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.2', 'mappings': [{'from': 2, 'to': 0}]}, {'pgid': '6.4', 'mappings': [{'from': 5, 'to': 0}]}, {'pgid': '6.a', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '6.13', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '11.f', 'mappings': [{'from': 1, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.3:0/45234419': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/384540368': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3883474825': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/1598547836': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:6800/3687155565': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3884352348': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2319227492': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2154949954': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/4162970769': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/2292819669': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/3411869253': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/520312891': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2267426691': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/3211426112': '2024-03-22T13:57:56.576834+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T14:22:22.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:22:22.304 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:22.334 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:22.436 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:22.464 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:22.566 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:22.594 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:22.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 69 16 88762 88746 80.3717 67.3125 0.00956057 0.0124363 2024-03-21T14:22:22.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:22.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.752 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:22.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:22.756 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:22:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:22:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:22:22.756 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:22:22.756 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:22:22.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:22.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:22.901 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:22.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:23.002 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:23.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:22 smithi003 ceph-mon[29576]: pgmap v942: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 691 op/s 2024-03-21T14:22:23.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:22 smithi003 ceph-mon[40510]: pgmap v942: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 691 op/s 2024-03-21T14:22:23.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.029 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:23.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:23.130 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:23.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:22 smithi118 ceph-mon[33862]: pgmap v942: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 43 MiB/s wr, 691 op/s 2024-03-21T14:22:23.263 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:23.304 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:23.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:23.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:23.448 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:23.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:23.550 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:23.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-21T14:22:23.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 70 16 89830 89814 80.177 66.75 0.0112253 0.0124662 2024-03-21T14:22:23.606 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:23.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:23.680 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:23.708 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:23.734 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:23.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:22:23.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:23.880 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:23.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:23.982 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:24.009 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:24.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:24.111 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:24.163 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:24.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:24.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.305 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:24.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:24.406 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:24.435 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:24.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:24.537 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.583 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:24.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:24.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 71 16 90810 90794 79.9104 61.25 0.0105564 0.0125088 2024-03-21T14:22:24.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:24.731 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:24.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:24.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:24.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:24.973 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:25.003 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:25.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:24 smithi003 ceph-mon[29576]: pgmap v943: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-21T14:22:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:24 smithi003 ceph-mon[40510]: pgmap v943: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-21T14:22:25.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.148 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:25.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:24 smithi118 ceph-mon[33862]: pgmap v943: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.22k op/s 2024-03-21T14:22:25.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:25.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:25.402 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:25.433 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:25.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:25.578 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:25.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 72 16 91968 91952 79.8057 72.375 0.0146104 0.0125256 2024-03-21T14:22:25.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:25.712 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:25.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:25.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:25.945 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:25.977 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:25.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:26.079 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:26.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:26.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:26.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:26.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.269 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:26.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:26.370 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:26.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:26.530 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:26.553 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:26.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 73 16 93072 93056 79.6576 69 0.00632679 0.0125486 2024-03-21T14:22:26.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:26.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:26.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:22:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:22:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:22:26.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:26.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:26.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:26.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:26.970 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:26.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:26.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:27.072 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:27.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:26 smithi003 ceph-mon[29576]: pgmap v944: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:27.084 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:26 smithi003 ceph-mon[40510]: pgmap v944: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:27.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:26 smithi118 ceph-mon[33862]: pgmap v944: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:27.207 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.235 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:27.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:27.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:27.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:27.503 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:27.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:27.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 74 16 94097 94081 79.4468 64.0625 0.0353592 0.0125809 2024-03-21T14:22:27.630 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:27.657 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:27.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:27.677 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:27.677 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:27.750 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:22:27.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:27.772 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:22:27.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:27.885 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:22:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:22:27.887 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:27.914 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:27.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:28.016 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:28.044 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:28.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:28.146 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:28.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:28.277 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.319 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:28.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:28.420 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:28.464 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:28.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:28.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:28.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 75 16 95143 95127 79.2591 65.375 0.0124687 0.0126114 2024-03-21T14:22:28.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:28.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:28.712 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:28.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:28.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:28.844 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:22:28.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:28.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:28.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:28.995 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:29.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.158 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:29.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:28 smithi118 ceph-mon[33862]: pgmap v945: 158 pgs: 158 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-21T14:22:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:28 smithi118 ceph-mon[33862]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:22:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3669717404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:22:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:28 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:22:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[29576]: pgmap v945: 158 pgs: 158 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-21T14:22:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[29576]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:22:29.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3669717404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:22:29.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:22:29.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:29.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[40510]: pgmap v945: 158 pgs: 158 active+clean; 5.6 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 511 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-21T14:22:29.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[40510]: from='client.25648 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:22:29.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3669717404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:22:29.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:28 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:22:29.259 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:29.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:29.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:29.431 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:29.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:29.572 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:29.601 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 76 16 96302 96286 79.1692 72.4375 0.0138369 0.0126259 2024-03-21T14:22:29.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:29.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:29.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:29.855 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:29.956 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:29.985 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:29.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:30.086 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:30.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:30.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:30.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:22:30.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:22:30.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: osdmap e439: 8 total, 8 up, 8 in 2024-03-21T14:22:30.129 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:22:30.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:22:30.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:22:30.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:22:30.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:22:30.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:22:30.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: osdmap e439: 8 total, 8 up, 8 in 2024-03-21T14:22:30.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:22:30.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:22:30.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:22:30.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:29 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: osdmap e439: 8 total, 8 up, 8 in 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:22:30.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:29 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:22:30.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:30.258 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:30.360 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:30.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:30.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:30.488 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:30.516 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:30.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 77 16 97459 97443 79.0801 72.3125 0.00822602 0.0126407 2024-03-21T14:22:30.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:30.635 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":945,"stamp":"2024-03-21T14:22:27.638134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6032529832,"num_objects":92238,"num_object_clones":0,"num_object_copies":276714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92238,"num_whiteouts":0,"num_read":2253,"num_read_kb":2005,"num_write":92762,"num_write_kb":5890605,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3952,"num_bytes_recovered":246878200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72549,"ondisk_log_size":72549,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":23922068,"kb_used_data":18476956,"kb_used_omap":21,"kb_used_meta":5444842,"kb_avail":726039148,"statfs":{"total":767960285184,"available":743464087552,"internally_reserved":0,"allocated":18920402944,"data_stored":18911855512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22067,"internal_metadata":5575518669},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,6,2,11,8,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":645988352,"num_objects":9857,"num_object_clones":0,"num_object_copies":29571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9857,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9857,"num_write_kb":630848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.007323"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":798,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810446+0000","last_change":"2024-03-21T14:18:38.441703+0000","last_active":"2024-03-21T14:22:19.810446+0000","last_peered":"2024-03-21T14:22:19.810446+0000","last_clean":"2024-03-21T14:22:19.810446+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:22:19.810446+0000","last_undegraded":"2024-03-21T14:22:19.810446+0000","last_fullsized":"2024-03-21T14:22:19.810446+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:38.441623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:38.441623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.814837+0000","last_change":"2024-03-21T14:19:09.223055+0000","last_active":"2024-03-21T14:22:19.814837+0000","last_peered":"2024-03-21T14:22:19.814837+0000","last_clean":"2024-03-21T14:22:19.814837+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:22:19.814837+0000","last_undegraded":"2024-03-21T14:22:19.814837+0000","last_fullsized":"2024-03-21T14:22:19.814837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:09.222911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:19:09.222911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687565+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":375,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.808633+0000","last_change":"2024-03-21T14:20:42.558129+0000","last_active":"2024-03-21T14:22:19.808633+0000","last_peered":"2024-03-21T14:22:19.808633+0000","last_clean":"2024-03-21T14:22:19.808633+0000","last_became_active":"2024-03-21T14:19:34.695760+0000","last_became_peered":"2024-03-21T14:19:34.695760+0000","last_unstale":"2024-03-21T14:22:19.808633+0000","last_undegraded":"2024-03-21T14:22:19.808633+0000","last_fullsized":"2024-03-21T14:22:19.808633+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.558048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.558048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"236'15","reported_seq":865,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810136+0000","last_change":"2024-03-21T14:20:41.426448+0000","last_active":"2024-03-21T14:22:19.810136+0000","last_peered":"2024-03-21T14:22:19.810136+0000","last_clean":"2024-03-21T14:22:19.810136+0000","last_became_active":"2024-03-21T14:20:21.782187+0000","last_became_peered":"2024-03-21T14:20:21.782187+0000","last_unstale":"2024-03-21T14:22:19.810136+0000","last_undegraded":"2024-03-21T14:22:19.810136+0000","last_fullsized":"2024-03-21T14:22:19.810136+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:20:41.426344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:41.426344+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.545233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":103,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.829731+0000","last_change":"2024-03-21T14:20:21.782736+0000","last_active":"2024-03-21T14:22:19.829731+0000","last_peered":"2024-03-21T14:22:19.829731+0000","last_clean":"2024-03-21T14:22:19.829731+0000","last_became_active":"2024-03-21T14:20:21.782401+0000","last_became_peered":"2024-03-21T14:20:21.782401+0000","last_unstale":"2024-03-21T14:22:19.829731+0000","last_undegraded":"2024-03-21T14:22:19.829731+0000","last_fullsized":"2024-03-21T14:22:19.829731+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:15.807973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.807973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274204+0000","scrub_duration":195,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":818,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810005+0000","last_change":"2024-03-21T14:20:52.477097+0000","last_active":"2024-03-21T14:22:19.810005+0000","last_peered":"2024-03-21T14:22:19.810005+0000","last_clean":"2024-03-21T14:22:19.810005+0000","last_became_active":"2024-03-21T14:19:14.689459+0000","last_became_peered":"2024-03-21T14:19:14.689459+0000","last_unstale":"2024-03-21T14:22:19.810005+0000","last_undegraded":"2024-03-21T14:22:19.810005+0000","last_fullsized":"2024-03-21T14:22:19.810005+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:52.477009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.477009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":828,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832748+0000","last_change":"2024-03-21T14:20:37.939217+0000","last_active":"2024-03-21T14:22:19.832748+0000","last_peered":"2024-03-21T14:22:19.832748+0000","last_clean":"2024-03-21T14:22:19.832748+0000","last_became_active":"2024-03-21T14:19:14.695703+0000","last_became_peered":"2024-03-21T14:19:14.695703+0000","last_unstale":"2024-03-21T14:22:19.832748+0000","last_undegraded":"2024-03-21T14:22:19.832748+0000","last_fullsized":"2024-03-21T14:22:19.832748+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:37.939080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.939080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274435+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"237'34","reported_seq":867,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.831548+0000","last_change":"2024-03-21T14:20:40.919011+0000","last_active":"2024-03-21T14:22:19.831548+0000","last_peered":"2024-03-21T14:22:19.831548+0000","last_clean":"2024-03-21T14:22:19.831548+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:22:19.831548+0000","last_undegraded":"2024-03-21T14:22:19.831548+0000","last_fullsized":"2024-03-21T14:22:19.831548+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:20:40.918895+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:20:40.918895+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":680,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813508+0000","last_change":"2024-03-21T14:19:34.691274+0000","last_active":"2024-03-21T14:22:19.813508+0000","last_peered":"2024-03-21T14:22:19.813508+0000","last_clean":"2024-03-21T14:22:19.813508+0000","last_became_active":"2024-03-21T14:19:34.690300+0000","last_became_peered":"2024-03-21T14:19:34.690300+0000","last_unstale":"2024-03-21T14:22:19.813508+0000","last_undegraded":"2024-03-21T14:22:19.813508+0000","last_fullsized":"2024-03-21T14:22:19.813508+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:23.464551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:23.464551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687766+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":826,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.795917+0000","last_change":"2024-03-21T14:21:30.614515+0000","last_active":"2024-03-21T14:22:19.795917+0000","last_peered":"2024-03-21T14:22:19.795917+0000","last_clean":"2024-03-21T14:22:19.795917+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:22:19.795917+0000","last_undegraded":"2024-03-21T14:22:19.795917+0000","last_fullsized":"2024-03-21T14:22:19.795917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:30.614440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:21:30.614440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:47.977666+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":619,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.811264+0000","last_change":"2024-03-21T14:17:49.153687+0000","last_active":"2024-03-21T14:22:19.811264+0000","last_peered":"2024-03-21T14:22:19.811264+0000","last_clean":"2024-03-21T14:22:19.811264+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:22:19.811264+0000","last_undegraded":"2024-03-21T14:22:19.811264+0000","last_fullsized":"2024-03-21T14:22:19.811264+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:49.153572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:17:49.153572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"236'15","reported_seq":823,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810443+0000","last_change":"2024-03-21T14:17:21.179086+0000","last_active":"2024-03-21T14:22:19.810443+0000","last_peered":"2024-03-21T14:22:19.810443+0000","last_clean":"2024-03-21T14:22:19.810443+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:22:19.810443+0000","last_undegraded":"2024-03-21T14:22:19.810443+0000","last_fullsized":"2024-03-21T14:22:19.810443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:17:21.178967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:21.178967+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":769,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.815185+0000","last_change":"2024-03-21T14:20:21.776714+0000","last_active":"2024-03-21T14:22:19.815185+0000","last_peered":"2024-03-21T14:22:19.815185+0000","last_clean":"2024-03-21T14:22:19.815185+0000","last_became_active":"2024-03-21T14:20:21.776360+0000","last_became_peered":"2024-03-21T14:20:21.776360+0000","last_unstale":"2024-03-21T14:22:19.815185+0000","last_undegraded":"2024-03-21T14:22:19.815185+0000","last_fullsized":"2024-03-21T14:22:19.815185+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:18.436540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:18.436540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687738+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":827,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825068+0000","last_change":"2024-03-21T14:21:47.782509+0000","last_active":"2024-03-21T14:22:19.825068+0000","last_peered":"2024-03-21T14:22:19.825068+0000","last_clean":"2024-03-21T14:22:19.825068+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:22:19.825068+0000","last_undegraded":"2024-03-21T14:22:19.825068+0000","last_fullsized":"2024-03-21T14:22:19.825068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:47.782401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.782401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:13.997696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":293,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810607+0000","last_change":"2024-03-21T14:21:53.335510+0000","last_active":"2024-03-21T14:22:19.810607+0000","last_peered":"2024-03-21T14:22:19.810607+0000","last_clean":"2024-03-21T14:22:19.810607+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:22:19.810607+0000","last_undegraded":"2024-03-21T14:22:19.810607+0000","last_fullsized":"2024-03-21T14:22:19.810607+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_clean_scrub_stamp":"2024-03-21T14:21:53.335395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.820889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"236'7","reported_seq":819,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810647+0000","last_change":"2024-03-21T14:18:04.289351+0000","last_active":"2024-03-21T14:22:19.810647+0000","last_peered":"2024-03-21T14:22:19.810647+0000","last_clean":"2024-03-21T14:22:19.810647+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:22:19.810647+0000","last_undegraded":"2024-03-21T14:22:19.810647+0000","last_fullsized":"2024-03-21T14:22:19.810647+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'7","last_scrub_stamp":"2024-03-21T14:18:04.289228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:04.289228+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":798,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810791+0000","last_change":"2024-03-21T14:17:06.425215+0000","last_active":"2024-03-21T14:22:19.810791+0000","last_peered":"2024-03-21T14:22:19.810791+0000","last_clean":"2024-03-21T14:22:19.810791+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:22:19.810791+0000","last_undegraded":"2024-03-21T14:22:19.810791+0000","last_fullsized":"2024-03-21T14:22:19.810791+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:06.425104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:17:06.425104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":811,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822047+0000","last_change":"2024-03-21T14:18:34.187073+0000","last_active":"2024-03-21T14:22:19.822047+0000","last_peered":"2024-03-21T14:22:19.822047+0000","last_clean":"2024-03-21T14:22:19.822047+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:22:19.822047+0000","last_undegraded":"2024-03-21T14:22:19.822047+0000","last_fullsized":"2024-03-21T14:22:19.822047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:34.186897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:34.186897+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":670,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810597+0000","last_change":"2024-03-21T14:19:34.686397+0000","last_active":"2024-03-21T14:22:19.810597+0000","last_peered":"2024-03-21T14:22:19.810597+0000","last_clean":"2024-03-21T14:22:19.810597+0000","last_became_active":"2024-03-21T14:19:34.686106+0000","last_became_peered":"2024-03-21T14:19:34.686106+0000","last_unstale":"2024-03-21T14:22:19.810597+0000","last_undegraded":"2024-03-21T14:22:19.810597+0000","last_fullsized":"2024-03-21T14:22:19.810597+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:33.422232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:18:33.422232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.466551+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"432'34","reported_seq":733,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812805+0000","last_change":"2024-03-21T14:21:50.323084+0000","last_active":"2024-03-21T14:22:19.812805+0000","last_peered":"2024-03-21T14:22:19.812805+0000","last_clean":"2024-03-21T14:22:19.812805+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:22:19.812805+0000","last_undegraded":"2024-03-21T14:22:19.812805+0000","last_fullsized":"2024-03-21T14:22:19.812805+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'34","last_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_deep_scrub":"432'34","last_deep_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.322580+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:12.777689+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":823,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832664+0000","last_change":"2024-03-21T14:21:33.288900+0000","last_active":"2024-03-21T14:22:19.832664+0000","last_peered":"2024-03-21T14:22:19.832664+0000","last_clean":"2024-03-21T14:22:19.832664+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:22:19.832664+0000","last_undegraded":"2024-03-21T14:22:19.832664+0000","last_fullsized":"2024-03-21T14:22:19.832664+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:33.288792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:33.288792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:47.021065+0000","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":803,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810799+0000","last_change":"2024-03-21T14:21:15.304334+0000","last_active":"2024-03-21T14:22:19.810799+0000","last_peered":"2024-03-21T14:22:19.810799+0000","last_clean":"2024-03-21T14:22:19.810799+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:22:19.810799+0000","last_undegraded":"2024-03-21T14:22:19.810799+0000","last_fullsized":"2024-03-21T14:22:19.810799+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:15.304232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:15.304232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:29.257097+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":811,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810536+0000","last_change":"2024-03-21T14:20:31.407178+0000","last_active":"2024-03-21T14:22:19.810536+0000","last_peered":"2024-03-21T14:22:19.810536+0000","last_clean":"2024-03-21T14:22:19.810536+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:22:19.810536+0000","last_undegraded":"2024-03-21T14:22:19.810536+0000","last_fullsized":"2024-03-21T14:22:19.810536+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.407065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.407065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"237'18","reported_seq":847,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.834280+0000","last_change":"2024-03-21T14:18:03.889554+0000","last_active":"2024-03-21T14:22:19.834280+0000","last_peered":"2024-03-21T14:22:19.834280+0000","last_clean":"2024-03-21T14:22:19.834280+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:22:19.834280+0000","last_undegraded":"2024-03-21T14:22:19.834280+0000","last_fullsized":"2024-03-21T14:22:19.834280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:18:03.889428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:03.889428+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.273777+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":327,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812291+0000","last_change":"2024-03-21T14:21:19.290180+0000","last_active":"2024-03-21T14:22:19.812291+0000","last_peered":"2024-03-21T14:22:19.812291+0000","last_clean":"2024-03-21T14:22:19.812291+0000","last_became_active":"2024-03-21T14:19:14.686523+0000","last_became_peered":"2024-03-21T14:19:14.686523+0000","last_unstale":"2024-03-21T14:22:19.812291+0000","last_undegraded":"2024-03-21T14:22:19.812291+0000","last_fullsized":"2024-03-21T14:22:19.812291+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:21:19.289804+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.289804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:46.633069+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"425'15","reported_seq":63,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.801338+0000","last_change":"2024-03-21T14:21:41.066114+0000","last_active":"2024-03-21T14:22:19.801338+0000","last_peered":"2024-03-21T14:22:19.801338+0000","last_clean":"2024-03-21T14:22:19.801338+0000","last_became_active":"2024-03-21T14:21:26.767116+0000","last_became_peered":"2024-03-21T14:21:26.767116+0000","last_unstale":"2024-03-21T14:22:19.801338+0000","last_undegraded":"2024-03-21T14:22:19.801338+0000","last_fullsized":"2024-03-21T14:22:19.801338+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'15","last_scrub_stamp":"2024-03-21T14:21:41.066020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.066020+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:43.952905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":837,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.814739+0000","last_change":"2024-03-21T14:20:37.151817+0000","last_active":"2024-03-21T14:22:19.814739+0000","last_peered":"2024-03-21T14:22:19.814739+0000","last_clean":"2024-03-21T14:22:19.814739+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:22:19.814739+0000","last_undegraded":"2024-03-21T14:22:19.814739+0000","last_fullsized":"2024-03-21T14:22:19.814739+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:20:37.151721+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.151721+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"237'20","reported_seq":542,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.826461+0000","last_change":"2024-03-21T14:21:19.102404+0000","last_active":"2024-03-21T14:22:19.826461+0000","last_peered":"2024-03-21T14:22:19.826461+0000","last_clean":"2024-03-21T14:22:19.826461+0000","last_became_active":"2024-03-21T14:20:21.790051+0000","last_became_peered":"2024-03-21T14:20:21.790051+0000","last_unstale":"2024-03-21T14:22:19.826461+0000","last_undegraded":"2024-03-21T14:22:19.826461+0000","last_fullsized":"2024-03-21T14:22:19.826461+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:21:19.102261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.102261+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.151553+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":815,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.821846+0000","last_change":"2024-03-21T14:21:19.457230+0000","last_active":"2024-03-21T14:22:19.821846+0000","last_peered":"2024-03-21T14:22:19.821846+0000","last_clean":"2024-03-21T14:22:19.821846+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:22:19.821846+0000","last_undegraded":"2024-03-21T14:22:19.821846+0000","last_fullsized":"2024-03-21T14:22:19.821846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:19.457101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.457101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:43.030243+0000","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":476,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.826337+0000","last_change":"2024-03-21T14:20:49.417668+0000","last_active":"2024-03-21T14:22:19.826337+0000","last_peered":"2024-03-21T14:22:19.826337+0000","last_clean":"2024-03-21T14:22:19.826337+0000","last_became_active":"2024-03-21T14:19:14.675463+0000","last_became_peered":"2024-03-21T14:19:14.675463+0000","last_unstale":"2024-03-21T14:22:19.826337+0000","last_undegraded":"2024-03-21T14:22:19.826337+0000","last_fullsized":"2024-03-21T14:22:19.826337+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:49.417593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.417593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.846514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":841,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.831559+0000","last_change":"2024-03-21T14:20:45.936416+0000","last_active":"2024-03-21T14:22:19.831559+0000","last_peered":"2024-03-21T14:22:19.831559+0000","last_clean":"2024-03-21T14:22:19.831559+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:22:19.831559+0000","last_undegraded":"2024-03-21T14:22:19.831559+0000","last_fullsized":"2024-03-21T14:22:19.831559+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.936318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.936318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":551,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.828226+0000","last_change":"2024-03-21T14:20:46.036522+0000","last_active":"2024-03-21T14:22:19.828226+0000","last_peered":"2024-03-21T14:22:19.828226+0000","last_clean":"2024-03-21T14:22:19.828226+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:22:19.828226+0000","last_undegraded":"2024-03-21T14:22:19.828226+0000","last_fullsized":"2024-03-21T14:22:19.828226+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.036377+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.688108+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":174,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813116+0000","last_change":"2024-03-21T14:19:14.680267+0000","last_active":"2024-03-21T14:22:19.813116+0000","last_peered":"2024-03-21T14:22:19.813116+0000","last_clean":"2024-03-21T14:22:19.813116+0000","last_became_active":"2024-03-21T14:19:14.679790+0000","last_became_peered":"2024-03-21T14:19:14.679790+0000","last_unstale":"2024-03-21T14:22:19.813116+0000","last_undegraded":"2024-03-21T14:22:19.813116+0000","last_fullsized":"2024-03-21T14:22:19.813116+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:49.859204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:14:49.859204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.416883+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":493,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810003+0000","last_change":"2024-03-21T14:18:42.513377+0000","last_active":"2024-03-21T14:22:19.810003+0000","last_peered":"2024-03-21T14:22:19.810003+0000","last_clean":"2024-03-21T14:22:19.810003+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:22:19.810003+0000","last_undegraded":"2024-03-21T14:22:19.810003+0000","last_fullsized":"2024-03-21T14:22:19.810003+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.513167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":295,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832825+0000","last_change":"2024-03-21T14:21:21.656390+0000","last_active":"2024-03-21T14:22:19.832825+0000","last_peered":"2024-03-21T14:22:19.832825+0000","last_clean":"2024-03-21T14:22:19.832825+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:22:19.832825+0000","last_undegraded":"2024-03-21T14:22:19.832825+0000","last_fullsized":"2024-03-21T14:22:19.832825+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:21.656278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:21.656278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:47.879644+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"275'10","reported_seq":850,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.831748+0000","last_change":"2024-03-21T14:18:45.023742+0000","last_active":"2024-03-21T14:22:19.831748+0000","last_peered":"2024-03-21T14:22:19.831748+0000","last_clean":"2024-03-21T14:22:19.831748+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:22:19.831748+0000","last_undegraded":"2024-03-21T14:22:19.831748+0000","last_fullsized":"2024-03-21T14:22:19.831748+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"275'10","last_scrub_stamp":"2024-03-21T14:18:45.023596+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:18:45.023596+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274004+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":840,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.830432+0000","last_change":"2024-03-21T14:21:34.144597+0000","last_active":"2024-03-21T14:22:19.830432+0000","last_peered":"2024-03-21T14:22:19.830432+0000","last_clean":"2024-03-21T14:22:19.830432+0000","last_became_active":"2024-03-21T14:19:34.693279+0000","last_became_peered":"2024-03-21T14:19:34.693279+0000","last_unstale":"2024-03-21T14:22:19.830432+0000","last_undegraded":"2024-03-21T14:22:19.830432+0000","last_fullsized":"2024-03-21T14:22:19.830432+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.144484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.144484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:45.749565+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":850,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825041+0000","last_change":"2024-03-21T14:21:51.742321+0000","last_active":"2024-03-21T14:22:19.825041+0000","last_peered":"2024-03-21T14:22:19.825041+0000","last_clean":"2024-03-21T14:22:19.825041+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:22:19.825041+0000","last_undegraded":"2024-03-21T14:22:19.825041+0000","last_fullsized":"2024-03-21T14:22:19.825041+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.742159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.742159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:07.946180+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":290,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822248+0000","last_change":"2024-03-21T14:19:42.258049+0000","last_active":"2024-03-21T14:22:19.822248+0000","last_peered":"2024-03-21T14:22:19.822248+0000","last_clean":"2024-03-21T14:22:19.822248+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:22:19.822248+0000","last_undegraded":"2024-03-21T14:22:19.822248+0000","last_fullsized":"2024-03-21T14:22:19.822248+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:19:42.257908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:19:42.257908+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687855+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"261'24","reported_seq":878,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.831447+0000","last_change":"2024-03-21T14:18:42.061939+0000","last_active":"2024-03-21T14:22:19.831447+0000","last_peered":"2024-03-21T14:22:19.831447+0000","last_clean":"2024-03-21T14:22:19.831447+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:22:19.831447+0000","last_undegraded":"2024-03-21T14:22:19.831447+0000","last_fullsized":"2024-03-21T14:22:19.831447+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'24","last_scrub_stamp":"2024-03-21T14:18:42.061791+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.061791+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274058+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":296,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813329+0000","last_change":"2024-03-21T14:19:50.405106+0000","last_active":"2024-03-21T14:22:19.813329+0000","last_peered":"2024-03-21T14:22:19.813329+0000","last_clean":"2024-03-21T14:22:19.813329+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:22:19.813329+0000","last_undegraded":"2024-03-21T14:22:19.813329+0000","last_fullsized":"2024-03-21T14:22:19.813329+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:50.404999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:50.404999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687824+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":820,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.830323+0000","last_change":"2024-03-21T14:21:41.085311+0000","last_active":"2024-03-21T14:22:19.830323+0000","last_peered":"2024-03-21T14:22:19.830323+0000","last_clean":"2024-03-21T14:22:19.830323+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:22:19.830323+0000","last_undegraded":"2024-03-21T14:22:19.830323+0000","last_fullsized":"2024-03-21T14:22:19.830323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:41.085192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.085192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:00.757692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":836,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825548+0000","last_change":"2024-03-21T14:20:45.456427+0000","last_active":"2024-03-21T14:22:19.825548+0000","last_peered":"2024-03-21T14:22:19.825548+0000","last_clean":"2024-03-21T14:22:19.825548+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:22:19.825548+0000","last_undegraded":"2024-03-21T14:22:19.825548+0000","last_fullsized":"2024-03-21T14:22:19.825548+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.456309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.456309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.846463+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"236'15","reported_seq":866,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822545+0000","last_change":"2024-03-21T14:19:25.467111+0000","last_active":"2024-03-21T14:22:19.822545+0000","last_peered":"2024-03-21T14:22:19.822545+0000","last_clean":"2024-03-21T14:22:19.822545+0000","last_became_active":"2024-03-21T14:19:14.682980+0000","last_became_peered":"2024-03-21T14:19:14.682980+0000","last_unstale":"2024-03-21T14:22:19.822545+0000","last_undegraded":"2024-03-21T14:22:19.822545+0000","last_fullsized":"2024-03-21T14:22:19.822545+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:19:25.465595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:25.465595+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687797+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":111,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.831268+0000","last_change":"2024-03-21T14:20:42.950594+0000","last_active":"2024-03-21T14:22:19.831268+0000","last_peered":"2024-03-21T14:22:19.831268+0000","last_clean":"2024-03-21T14:22:19.831268+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:22:19.831268+0000","last_undegraded":"2024-03-21T14:22:19.831268+0000","last_fullsized":"2024-03-21T14:22:19.831268+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.950500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.950500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274467+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":840,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822357+0000","last_change":"2024-03-21T14:21:36.498785+0000","last_active":"2024-03-21T14:22:19.822357+0000","last_peered":"2024-03-21T14:22:19.822357+0000","last_clean":"2024-03-21T14:22:19.822357+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:22:19.822357+0000","last_undegraded":"2024-03-21T14:22:19.822357+0000","last_fullsized":"2024-03-21T14:22:19.822357+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:36.498684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:36.498684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:36.695221+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":834,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810690+0000","last_change":"2024-03-21T14:21:20.339156+0000","last_active":"2024-03-21T14:22:19.810690+0000","last_peered":"2024-03-21T14:22:19.810690+0000","last_clean":"2024-03-21T14:22:19.810690+0000","last_became_active":"2024-03-21T14:19:14.690097+0000","last_became_peered":"2024-03-21T14:19:14.690097+0000","last_unstale":"2024-03-21T14:22:19.810690+0000","last_undegraded":"2024-03-21T14:22:19.810690+0000","last_fullsized":"2024-03-21T14:22:19.810690+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:20.339046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.339046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:32.698436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"236'24","reported_seq":913,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810691+0000","last_change":"2024-03-21T14:21:50.314232+0000","last_active":"2024-03-21T14:22:19.810691+0000","last_peered":"2024-03-21T14:22:19.810691+0000","last_clean":"2024-03-21T14:22:19.810691+0000","last_became_active":"2024-03-21T14:21:26.767846+0000","last_became_peered":"2024-03-21T14:21:26.767846+0000","last_unstale":"2024-03-21T14:22:19.810691+0000","last_undegraded":"2024-03-21T14:22:19.810691+0000","last_fullsized":"2024-03-21T14:22:19.810691+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:21:50.314134+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.314134+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:04.354530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":701,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.826064+0000","last_change":"2024-03-21T14:21:46.894626+0000","last_active":"2024-03-21T14:22:19.826064+0000","last_peered":"2024-03-21T14:22:19.826064+0000","last_clean":"2024-03-21T14:22:19.826064+0000","last_became_active":"2024-03-21T14:20:21.784416+0000","last_became_peered":"2024-03-21T14:20:21.784416+0000","last_unstale":"2024-03-21T14:22:19.826064+0000","last_undegraded":"2024-03-21T14:22:19.826064+0000","last_fullsized":"2024-03-21T14:22:19.826064+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.894547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.894547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:00.934567+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":1039,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.821981+0000","last_change":"2024-03-21T14:21:50.631849+0000","last_active":"2024-03-21T14:22:19.821981+0000","last_peered":"2024-03-21T14:22:19.821981+0000","last_clean":"2024-03-21T14:22:19.821981+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:22:19.821981+0000","last_undegraded":"2024-03-21T14:22:19.821981+0000","last_fullsized":"2024-03-21T14:22:19.821981+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'10","last_scrub_stamp":"2024-03-21T14:21:50.631459+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.631459+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:10.844377+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":838,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825361+0000","last_change":"2024-03-21T14:21:34.062701+0000","last_active":"2024-03-21T14:22:19.825361+0000","last_peered":"2024-03-21T14:22:19.825361+0000","last_clean":"2024-03-21T14:22:19.825361+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:22:19.825361+0000","last_undegraded":"2024-03-21T14:22:19.825361+0000","last_fullsized":"2024-03-21T14:22:19.825361+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.062588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.062588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:52.687684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"236'25","reported_seq":851,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832233+0000","last_change":"2024-03-21T14:21:18.712180+0000","last_active":"2024-03-21T14:22:19.832233+0000","last_peered":"2024-03-21T14:22:19.832233+0000","last_clean":"2024-03-21T14:22:19.832233+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:22:19.832233+0000","last_undegraded":"2024-03-21T14:22:19.832233+0000","last_fullsized":"2024-03-21T14:22:19.832233+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:18.712011+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:21:18.712011+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:42.923742+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":800,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.811066+0000","last_change":"2024-03-21T14:20:38.602527+0000","last_active":"2024-03-21T14:22:19.811066+0000","last_peered":"2024-03-21T14:22:19.811066+0000","last_clean":"2024-03-21T14:22:19.811066+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:22:19.811066+0000","last_undegraded":"2024-03-21T14:22:19.811066+0000","last_fullsized":"2024-03-21T14:22:19.811066+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:38.602405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:20:38.602405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":140,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832434+0000","last_change":"2024-03-21T14:20:44.917571+0000","last_active":"2024-03-21T14:22:19.832434+0000","last_peered":"2024-03-21T14:22:19.832434+0000","last_clean":"2024-03-21T14:22:19.832434+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:22:19.832434+0000","last_undegraded":"2024-03-21T14:22:19.832434+0000","last_fullsized":"2024-03-21T14:22:19.832434+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:20:44.917480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.917480+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.318821+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":103,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.829970+0000","last_change":"2024-03-21T14:20:21.784789+0000","last_active":"2024-03-21T14:22:19.829970+0000","last_peered":"2024-03-21T14:22:19.829970+0000","last_clean":"2024-03-21T14:22:19.829970+0000","last_became_active":"2024-03-21T14:20:21.784349+0000","last_became_peered":"2024-03-21T14:20:21.784349+0000","last_unstale":"2024-03-21T14:22:19.829970+0000","last_undegraded":"2024-03-21T14:22:19.829970+0000","last_fullsized":"2024-03-21T14:22:19.829970+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:40.140340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:18:40.140340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.273908+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"338'19","reported_seq":712,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825834+0000","last_change":"2024-03-21T14:21:37.034754+0000","last_active":"2024-03-21T14:22:19.825834+0000","last_peered":"2024-03-21T14:22:19.825834+0000","last_clean":"2024-03-21T14:22:19.825834+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:22:19.825834+0000","last_undegraded":"2024-03-21T14:22:19.825834+0000","last_fullsized":"2024-03-21T14:22:19.825834+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'19","last_scrub_stamp":"2024-03-21T14:21:37.034609+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:21:37.034609+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:01.752809+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":657,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813087+0000","last_change":"2024-03-21T14:21:49.278092+0000","last_active":"2024-03-21T14:22:19.813087+0000","last_peered":"2024-03-21T14:22:19.813087+0000","last_clean":"2024-03-21T14:22:19.813087+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:22:19.813087+0000","last_undegraded":"2024-03-21T14:22:19.813087+0000","last_fullsized":"2024-03-21T14:22:19.813087+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:49.277585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.277585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:14.479063+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":103,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.833015+0000","last_change":"2024-03-21T14:20:21.787792+0000","last_active":"2024-03-21T14:22:19.833015+0000","last_peered":"2024-03-21T14:22:19.833015+0000","last_clean":"2024-03-21T14:22:19.833015+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:22:19.833015+0000","last_undegraded":"2024-03-21T14:22:19.833015+0000","last_fullsized":"2024-03-21T14:22:19.833015+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.399758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":804,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810504+0000","last_change":"2024-03-21T14:21:26.768598+0000","last_active":"2024-03-21T14:22:19.810504+0000","last_peered":"2024-03-21T14:22:19.810504+0000","last_clean":"2024-03-21T14:22:19.810504+0000","last_became_active":"2024-03-21T14:21:26.767611+0000","last_became_peered":"2024-03-21T14:21:26.767611+0000","last_unstale":"2024-03-21T14:22:19.810504+0000","last_undegraded":"2024-03-21T14:22:19.810504+0000","last_fullsized":"2024-03-21T14:22:19.810504+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:05.373895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:05.373895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"432'19","reported_seq":521,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832961+0000","last_change":"2024-03-21T14:21:49.128441+0000","last_active":"2024-03-21T14:22:19.832961+0000","last_peered":"2024-03-21T14:22:19.832961+0000","last_clean":"2024-03-21T14:22:19.832961+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:22:19.832961+0000","last_undegraded":"2024-03-21T14:22:19.832961+0000","last_fullsized":"2024-03-21T14:22:19.832961+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'19","last_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_deep_scrub":"432'19","last_deep_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.128273+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:08.920118+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":830,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825466+0000","last_change":"2024-03-21T14:20:34.570950+0000","last_active":"2024-03-21T14:22:19.825466+0000","last_peered":"2024-03-21T14:22:19.825466+0000","last_clean":"2024-03-21T14:22:19.825466+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:22:19.825466+0000","last_undegraded":"2024-03-21T14:22:19.825466+0000","last_fullsized":"2024-03-21T14:22:19.825466+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:34.570712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:34.570712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.846432+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":294,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832848+0000","last_change":"2024-03-21T14:21:26.752265+0000","last_active":"2024-03-21T14:22:19.832848+0000","last_peered":"2024-03-21T14:22:19.832848+0000","last_clean":"2024-03-21T14:22:19.832848+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:22:19.832848+0000","last_undegraded":"2024-03-21T14:22:19.832848+0000","last_fullsized":"2024-03-21T14:22:19.832848+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:26.752126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:21:26.752126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:48.668885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":47,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.801135+0000","last_change":"2024-03-21T14:21:54.122932+0000","last_active":"2024-03-21T14:22:19.801135+0000","last_peered":"2024-03-21T14:22:19.801135+0000","last_clean":"2024-03-21T14:22:19.801135+0000","last_became_active":"2024-03-21T14:21:26.765914+0000","last_became_peered":"2024-03-21T14:21:26.765914+0000","last_unstale":"2024-03-21T14:22:19.801135+0000","last_undegraded":"2024-03-21T14:22:19.801135+0000","last_fullsized":"2024-03-21T14:22:19.801135+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.122791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.122791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:54.160613+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"236'12","reported_seq":111,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832456+0000","last_change":"2024-03-21T14:20:21.796765+0000","last_active":"2024-03-21T14:22:19.832456+0000","last_peered":"2024-03-21T14:22:19.832456+0000","last_clean":"2024-03-21T14:22:19.832456+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:22:19.832456+0000","last_undegraded":"2024-03-21T14:22:19.832456+0000","last_fullsized":"2024-03-21T14:22:19.832456+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:19:32.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:32.302574+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"438'5623","reported_seq":5686,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:22.626931+0000","last_change":"2024-03-21T14:20:59.387014+0000","last_active":"2024-03-21T14:22:22.626931+0000","last_peered":"2024-03-21T14:22:22.626931+0000","last_clean":"2024-03-21T14:22:22.626931+0000","last_became_active":"2024-03-21T14:20:59.386663+0000","last_became_peered":"2024-03-21T14:20:59.386663+0000","last_unstale":"2024-03-21T14:22:22.626931+0000","last_undegraded":"2024-03-21T14:22:22.626931+0000","last_fullsized":"2024-03-21T14:22:22.626931+0000","mapping_epoch":411,"log_start":"426'1600","ondisk_log_start":"426'1600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.688140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368508928,"num_objects":5623,"num_object_clones":0,"num_object_copies":16869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5623,"num_write_kb":359872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":693,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.821544+0000","last_change":"2024-03-21T14:19:11.368761+0000","last_active":"2024-03-21T14:22:19.821544+0000","last_peered":"2024-03-21T14:22:19.821544+0000","last_clean":"2024-03-21T14:22:19.821544+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:22:19.821544+0000","last_undegraded":"2024-03-21T14:22:19.821544+0000","last_fullsized":"2024-03-21T14:22:19.821544+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:11.368572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:25.324018+0000","last_clean_scrub_stamp":"2024-03-21T14:19:11.368572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687681+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":874,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.826365+0000","last_change":"2024-03-21T14:21:41.979701+0000","last_active":"2024-03-21T14:22:19.826365+0000","last_peered":"2024-03-21T14:22:19.826365+0000","last_clean":"2024-03-21T14:22:19.826365+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:22:19.826365+0000","last_undegraded":"2024-03-21T14:22:19.826365+0000","last_fullsized":"2024-03-21T14:22:19.826365+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:21:41.979582+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.979582+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:45.246205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"236'27","reported_seq":748,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.824930+0000","last_change":"2024-03-21T14:20:39.417922+0000","last_active":"2024-03-21T14:22:19.824930+0000","last_peered":"2024-03-21T14:22:19.824930+0000","last_clean":"2024-03-21T14:22:19.824930+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:22:19.824930+0000","last_undegraded":"2024-03-21T14:22:19.824930+0000","last_fullsized":"2024-03-21T14:22:19.824930+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:20:39.417781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:39.417781+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.846414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"372'10","reported_seq":1087,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825339+0000","last_change":"2024-03-21T14:21:35.014048+0000","last_active":"2024-03-21T14:22:19.825339+0000","last_peered":"2024-03-21T14:22:19.825339+0000","last_clean":"2024-03-21T14:22:19.825339+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:22:19.825339+0000","last_undegraded":"2024-03-21T14:22:19.825339+0000","last_fullsized":"2024-03-21T14:22:19.825339+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:21:35.013952+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_clean_scrub_stamp":"2024-03-21T14:21:35.013952+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:01.805652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"438'5659","reported_seq":5723,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:23.031868+0000","last_change":"2024-03-21T14:20:59.242772+0000","last_active":"2024-03-21T14:22:23.031868+0000","last_peered":"2024-03-21T14:22:23.031868+0000","last_clean":"2024-03-21T14:22:23.031868+0000","last_became_active":"2024-03-21T14:20:59.241791+0000","last_became_peered":"2024-03-21T14:20:59.241791+0000","last_unstale":"2024-03-21T14:22:23.031868+0000","last_undegraded":"2024-03-21T14:22:23.031868+0000","last_fullsized":"2024-03-21T14:22:23.031868+0000","mapping_epoch":411,"log_start":"418'600","ondisk_log_start":"418'600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5059,"log_dups_size":0,"ondisk_log_size":5059,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.447534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370868224,"num_objects":5659,"num_object_clones":0,"num_object_copies":16977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5659,"num_write_kb":362176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":554,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.829913+0000","last_change":"2024-03-21T14:18:26.997703+0000","last_active":"2024-03-21T14:22:19.829913+0000","last_peered":"2024-03-21T14:22:19.829913+0000","last_clean":"2024-03-21T14:22:19.829913+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:22:19.829913+0000","last_undegraded":"2024-03-21T14:22:19.829913+0000","last_fullsized":"2024-03-21T14:22:19.829913+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:26.997615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:26.997615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.273879+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":852,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.826216+0000","last_change":"2024-03-21T14:21:50.747507+0000","last_active":"2024-03-21T14:22:19.826216+0000","last_peered":"2024-03-21T14:22:19.826216+0000","last_clean":"2024-03-21T14:22:19.826216+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:22:19.826216+0000","last_undegraded":"2024-03-21T14:22:19.826216+0000","last_fullsized":"2024-03-21T14:22:19.826216+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.747341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:53.677359+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"236'18","reported_seq":894,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810357+0000","last_change":"2024-03-21T14:20:21.797847+0000","last_active":"2024-03-21T14:22:19.810357+0000","last_peered":"2024-03-21T14:22:19.810357+0000","last_clean":"2024-03-21T14:22:19.810357+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:22:19.810357+0000","last_undegraded":"2024-03-21T14:22:19.810357+0000","last_fullsized":"2024-03-21T14:22:19.810357+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:18:29.432719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.432719+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":669,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.807184+0000","last_change":"2024-03-21T14:21:54.295031+0000","last_active":"2024-03-21T14:22:19.807184+0000","last_peered":"2024-03-21T14:22:19.807184+0000","last_clean":"2024-03-21T14:22:19.807184+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:22:19.807184+0000","last_undegraded":"2024-03-21T14:22:19.807184+0000","last_fullsized":"2024-03-21T14:22:19.807184+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:57.916723+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"438'5806","reported_seq":5870,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.497500+0000","last_change":"2024-03-21T14:20:59.387129+0000","last_active":"2024-03-21T14:22:27.497500+0000","last_peered":"2024-03-21T14:22:27.497500+0000","last_clean":"2024-03-21T14:22:27.497500+0000","last_became_active":"2024-03-21T14:20:59.386704+0000","last_became_peered":"2024-03-21T14:20:59.386704+0000","last_unstale":"2024-03-21T14:22:27.497500+0000","last_undegraded":"2024-03-21T14:22:27.497500+0000","last_fullsized":"2024-03-21T14:22:27.497500+0000","mapping_epoch":411,"log_start":"426'1600","ondisk_log_start":"426'1600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4206,"log_dups_size":0,"ondisk_log_size":4206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.972995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380502016,"num_objects":5806,"num_object_clones":0,"num_object_copies":17418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5806,"num_write_kb":371584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":182,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.807283+0000","last_change":"2024-03-21T14:20:44.543695+0000","last_active":"2024-03-21T14:22:19.807283+0000","last_peered":"2024-03-21T14:22:19.807283+0000","last_clean":"2024-03-21T14:22:19.807283+0000","last_became_active":"2024-03-21T14:19:14.700785+0000","last_became_peered":"2024-03-21T14:19:14.700785+0000","last_unstale":"2024-03-21T14:22:19.807283+0000","last_undegraded":"2024-03-21T14:22:19.807283+0000","last_fullsized":"2024-03-21T14:22:19.807283+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:44.543611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.543611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":622,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.807503+0000","last_change":"2024-03-21T14:20:51.545141+0000","last_active":"2024-03-21T14:22:19.807503+0000","last_peered":"2024-03-21T14:22:19.807503+0000","last_clean":"2024-03-21T14:22:19.807503+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:22:19.807503+0000","last_undegraded":"2024-03-21T14:22:19.807503+0000","last_fullsized":"2024-03-21T14:22:19.807503+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:51.545057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.545057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":47,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.802412+0000","last_change":"2024-03-21T14:21:51.064832+0000","last_active":"2024-03-21T14:22:19.802412+0000","last_peered":"2024-03-21T14:22:19.802412+0000","last_clean":"2024-03-21T14:22:19.802412+0000","last_became_active":"2024-03-21T14:21:26.764088+0000","last_became_peered":"2024-03-21T14:21:26.764088+0000","last_unstale":"2024-03-21T14:22:19.802412+0000","last_undegraded":"2024-03-21T14:22:19.802412+0000","last_fullsized":"2024-03-21T14:22:19.802412+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.064713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.064713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:06.062019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"237'44","reported_seq":777,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822165+0000","last_change":"2024-03-21T14:20:49.145974+0000","last_active":"2024-03-21T14:22:19.822165+0000","last_peered":"2024-03-21T14:22:19.822165+0000","last_clean":"2024-03-21T14:22:19.822165+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:22:19.822165+0000","last_undegraded":"2024-03-21T14:22:19.822165+0000","last_fullsized":"2024-03-21T14:22:19.822165+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.145792+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687955+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"438'5820","reported_seq":8701,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.057125+0000","last_change":"2024-03-21T14:21:34.718068+0000","last_active":"2024-03-21T14:22:27.057125+0000","last_peered":"2024-03-21T14:22:27.057125+0000","last_clean":"2024-03-21T14:22:27.057125+0000","last_became_active":"2024-03-21T14:21:34.716304+0000","last_became_peered":"2024-03-21T14:21:34.716304+0000","last_unstale":"2024-03-21T14:22:27.057125+0000","last_undegraded":"2024-03-21T14:22:27.057125+0000","last_fullsized":"2024-03-21T14:22:27.057125+0000","mapping_epoch":427,"log_start":"421'1200","ondisk_log_start":"421'1200","created":411,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:31.711886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":381419520,"num_objects":5820,"num_object_clones":0,"num_object_copies":17460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5820,"num_write_kb":372480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1405,"num_bytes_recovered":92078080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":181,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812646+0000","last_change":"2024-03-21T14:19:34.691261+0000","last_active":"2024-03-21T14:22:19.812646+0000","last_peered":"2024-03-21T14:22:19.812646+0000","last_clean":"2024-03-21T14:22:19.812646+0000","last_became_active":"2024-03-21T14:19:34.690841+0000","last_became_peered":"2024-03-21T14:19:34.690841+0000","last_unstale":"2024-03-21T14:22:19.812646+0000","last_undegraded":"2024-03-21T14:22:19.812646+0000","last_fullsized":"2024-03-21T14:22:19.812646+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:14:44.865488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:14:44.865488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.416854+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":669,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813415+0000","last_change":"2024-03-21T14:21:46.343726+0000","last_active":"2024-03-21T14:22:19.813415+0000","last_peered":"2024-03-21T14:22:19.813415+0000","last_clean":"2024-03-21T14:22:19.813415+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:22:19.813415+0000","last_undegraded":"2024-03-21T14:22:19.813415+0000","last_fullsized":"2024-03-21T14:22:19.813415+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.343600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.343600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:11.677559+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":620,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832147+0000","last_change":"2024-03-21T14:18:40.092328+0000","last_active":"2024-03-21T14:22:19.832147+0000","last_peered":"2024-03-21T14:22:19.832147+0000","last_clean":"2024-03-21T14:22:19.832147+0000","last_became_active":"2024-03-21T14:16:21.865349+0000","last_became_peered":"2024-03-21T14:16:21.865349+0000","last_unstale":"2024-03-21T14:22:19.832147+0000","last_undegraded":"2024-03-21T14:22:19.832147+0000","last_fullsized":"2024-03-21T14:22:19.832147+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:40.092182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:18:40.092182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.273958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"236'15","reported_seq":324,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.807764+0000","last_change":"2024-03-21T14:21:20.291857+0000","last_active":"2024-03-21T14:22:19.807764+0000","last_peered":"2024-03-21T14:22:19.807764+0000","last_clean":"2024-03-21T14:22:19.807764+0000","last_became_active":"2024-03-21T14:19:14.692456+0000","last_became_peered":"2024-03-21T14:19:14.692456+0000","last_unstale":"2024-03-21T14:22:19.807764+0000","last_undegraded":"2024-03-21T14:22:19.807764+0000","last_fullsized":"2024-03-21T14:22:19.807764+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:21:20.291754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.291754+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:45.708814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"438'5676","reported_seq":5740,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.529287+0000","last_change":"2024-03-21T14:20:59.241783+0000","last_active":"2024-03-21T14:22:27.529287+0000","last_peered":"2024-03-21T14:22:27.529287+0000","last_clean":"2024-03-21T14:22:27.529287+0000","last_became_active":"2024-03-21T14:20:59.241477+0000","last_became_peered":"2024-03-21T14:20:59.241477+0000","last_unstale":"2024-03-21T14:22:27.529287+0000","last_undegraded":"2024-03-21T14:22:27.529287+0000","last_fullsized":"2024-03-21T14:22:27.529287+0000","mapping_epoch":411,"log_start":"426'1500","ondisk_log_start":"426'1500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:32.356092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371982336,"num_objects":5676,"num_object_clones":0,"num_object_copies":17028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5676,"num_write_kb":363264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":797,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812855+0000","last_change":"2024-03-21T14:20:35.508783+0000","last_active":"2024-03-21T14:22:19.812855+0000","last_peered":"2024-03-21T14:22:19.812855+0000","last_clean":"2024-03-21T14:22:19.812855+0000","last_became_active":"2024-03-21T14:19:14.689666+0000","last_became_peered":"2024-03-21T14:19:14.689666+0000","last_unstale":"2024-03-21T14:22:19.812855+0000","last_undegraded":"2024-03-21T14:22:19.812855+0000","last_fullsized":"2024-03-21T14:22:19.812855+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:35.508660+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:35.508660+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417034+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":809,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.833068+0000","last_change":"2024-03-21T14:20:50.816435+0000","last_active":"2024-03-21T14:22:19.833068+0000","last_peered":"2024-03-21T14:22:19.833068+0000","last_clean":"2024-03-21T14:22:19.833068+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:22:19.833068+0000","last_undegraded":"2024-03-21T14:22:19.833068+0000","last_fullsized":"2024-03-21T14:22:19.833068+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:50.816334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:50.816334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274571+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":320,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822001+0000","last_change":"2024-03-21T14:19:14.319436+0000","last_active":"2024-03-21T14:22:19.822001+0000","last_peered":"2024-03-21T14:22:19.822001+0000","last_clean":"2024-03-21T14:22:19.822001+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:22:19.822001+0000","last_undegraded":"2024-03-21T14:22:19.822001+0000","last_fullsized":"2024-03-21T14:22:19.822001+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:19:14.319346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:19:14.319346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":810,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810398+0000","last_change":"2024-03-21T14:16:57.382377+0000","last_active":"2024-03-21T14:22:19.810398+0000","last_peered":"2024-03-21T14:22:19.810398+0000","last_clean":"2024-03-21T14:22:19.810398+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:22:19.810398+0000","last_undegraded":"2024-03-21T14:22:19.810398+0000","last_fullsized":"2024-03-21T14:22:19.810398+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:16:57.382291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:16:57.382291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"360'54","reported_seq":752,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812342+0000","last_change":"2024-03-21T14:20:31.540383+0000","last_active":"2024-03-21T14:22:19.812342+0000","last_peered":"2024-03-21T14:22:19.812342+0000","last_clean":"2024-03-21T14:22:19.812342+0000","last_became_active":"2024-03-21T14:19:14.693616+0000","last_became_peered":"2024-03-21T14:19:14.693616+0000","last_unstale":"2024-03-21T14:22:19.812342+0000","last_undegraded":"2024-03-21T14:22:19.812342+0000","last_fullsized":"2024-03-21T14:22:19.812342+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'54","last_scrub_stamp":"2024-03-21T14:20:31.540283+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.540283+0000","objects_scrubbed":10,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.416952+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":695,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822082+0000","last_change":"2024-03-21T14:21:32.597365+0000","last_active":"2024-03-21T14:22:19.822082+0000","last_peered":"2024-03-21T14:22:19.822082+0000","last_clean":"2024-03-21T14:22:19.822082+0000","last_became_active":"2024-03-21T14:19:34.690636+0000","last_became_peered":"2024-03-21T14:19:34.690636+0000","last_unstale":"2024-03-21T14:22:19.822082+0000","last_undegraded":"2024-03-21T14:22:19.822082+0000","last_fullsized":"2024-03-21T14:22:19.822082+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:32.597257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:32.597257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:02.167935+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"438'5894","reported_seq":5958,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.486942+0000","last_change":"2024-03-21T14:20:59.387006+0000","last_active":"2024-03-21T14:22:27.486942+0000","last_peered":"2024-03-21T14:22:27.486942+0000","last_clean":"2024-03-21T14:22:27.486942+0000","last_became_active":"2024-03-21T14:20:59.386643+0000","last_became_peered":"2024-03-21T14:20:59.386643+0000","last_unstale":"2024-03-21T14:22:27.486942+0000","last_undegraded":"2024-03-21T14:22:27.486942+0000","last_fullsized":"2024-03-21T14:22:27.486942+0000","mapping_epoch":411,"log_start":"426'1700","ondisk_log_start":"426'1700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4194,"log_dups_size":0,"ondisk_log_size":4194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.582520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386269184,"num_objects":5894,"num_object_clones":0,"num_object_copies":17682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5894,"num_write_kb":377216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":1042,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810081+0000","last_change":"2024-03-21T14:21:52.357400+0000","last_active":"2024-03-21T14:22:19.810081+0000","last_peered":"2024-03-21T14:22:19.810081+0000","last_clean":"2024-03-21T14:22:19.810081+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:22:19.810081+0000","last_undegraded":"2024-03-21T14:22:19.810081+0000","last_fullsized":"2024-03-21T14:22:19.810081+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:52.357256+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:52.357256+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:09.449933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825858+0000","last_change":"2024-03-21T14:20:51.457667+0000","last_active":"2024-03-21T14:22:19.825858+0000","last_peered":"2024-03-21T14:22:19.825858+0000","last_clean":"2024-03-21T14:22:19.825858+0000","last_became_active":"2024-03-21T14:19:14.685026+0000","last_became_peered":"2024-03-21T14:19:14.685026+0000","last_unstale":"2024-03-21T14:22:19.825858+0000","last_undegraded":"2024-03-21T14:22:19.825858+0000","last_fullsized":"2024-03-21T14:22:19.825858+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:51.457549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.457549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.846529+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":828,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813852+0000","last_change":"2024-03-21T14:19:17.321024+0000","last_active":"2024-03-21T14:22:19.813852+0000","last_peered":"2024-03-21T14:22:19.813852+0000","last_clean":"2024-03-21T14:22:19.813852+0000","last_became_active":"2024-03-21T14:19:14.697007+0000","last_became_peered":"2024-03-21T14:19:14.697007+0000","last_unstale":"2024-03-21T14:22:19.813852+0000","last_undegraded":"2024-03-21T14:22:19.813852+0000","last_fullsized":"2024-03-21T14:22:19.813852+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:17.320902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:19:17.320902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687594+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":664,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812554+0000","last_change":"2024-03-21T14:20:48.551092+0000","last_active":"2024-03-21T14:22:19.812554+0000","last_peered":"2024-03-21T14:22:19.812554+0000","last_clean":"2024-03-21T14:22:19.812554+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:22:19.812554+0000","last_undegraded":"2024-03-21T14:22:19.812554+0000","last_fullsized":"2024-03-21T14:22:19.812554+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.550975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.550975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"236'22","reported_seq":915,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.827975+0000","last_change":"2024-03-21T14:21:47.628148+0000","last_active":"2024-03-21T14:22:19.827975+0000","last_peered":"2024-03-21T14:22:19.827975+0000","last_clean":"2024-03-21T14:22:19.827975+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:22:19.827975+0000","last_undegraded":"2024-03-21T14:22:19.827975+0000","last_fullsized":"2024-03-21T14:22:19.827975+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_deep_scrub":"236'22","last_deep_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.628002+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:56.266104+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":647,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.807183+0000","last_change":"2024-03-21T14:21:42.301835+0000","last_active":"2024-03-21T14:22:19.807183+0000","last_peered":"2024-03-21T14:22:19.807183+0000","last_clean":"2024-03-21T14:22:19.807183+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:22:19.807183+0000","last_undegraded":"2024-03-21T14:22:19.807183+0000","last_fullsized":"2024-03-21T14:22:19.807183+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:42.301114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:21:42.301114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:04.764280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"438'5884","reported_seq":5948,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.481043+0000","last_change":"2024-03-21T14:20:59.387064+0000","last_active":"2024-03-21T14:22:27.481043+0000","last_peered":"2024-03-21T14:22:27.481043+0000","last_clean":"2024-03-21T14:22:27.481043+0000","last_became_active":"2024-03-21T14:20:59.386748+0000","last_became_peered":"2024-03-21T14:20:59.386748+0000","last_unstale":"2024-03-21T14:22:27.481043+0000","last_undegraded":"2024-03-21T14:22:27.481043+0000","last_fullsized":"2024-03-21T14:22:27.481043+0000","mapping_epoch":411,"log_start":"426'1700","ondisk_log_start":"426'1700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4184,"log_dups_size":0,"ondisk_log_size":4184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:32.356248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385613824,"num_objects":5884,"num_object_clones":0,"num_object_copies":17652,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5884,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5884,"num_write_kb":376576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":293,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.833104+0000","last_change":"2024-03-21T14:21:23.819765+0000","last_active":"2024-03-21T14:22:19.833104+0000","last_peered":"2024-03-21T14:22:19.833104+0000","last_clean":"2024-03-21T14:22:19.833104+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:22:19.833104+0000","last_undegraded":"2024-03-21T14:22:19.833104+0000","last_fullsized":"2024-03-21T14:22:19.833104+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:23.819644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:23.819644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:39.020173+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1143,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.824809+0000","last_change":"2024-03-21T14:21:44.886680+0000","last_active":"2024-03-21T14:22:19.824809+0000","last_peered":"2024-03-21T14:22:19.824809+0000","last_clean":"2024-03-21T14:22:19.824809+0000","last_became_active":"2024-03-21T14:19:37.727165+0000","last_became_peered":"2024-03-21T14:19:37.727165+0000","last_unstale":"2024-03-21T14:22:19.824809+0000","last_undegraded":"2024-03-21T14:22:19.824809+0000","last_fullsized":"2024-03-21T14:22:19.824809+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:44.886598+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:44.886598+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:47.507181+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":835,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.830526+0000","last_change":"2024-03-21T14:20:49.815275+0000","last_active":"2024-03-21T14:22:19.830526+0000","last_peered":"2024-03-21T14:22:19.830526+0000","last_clean":"2024-03-21T14:22:19.830526+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:22:19.830526+0000","last_undegraded":"2024-03-21T14:22:19.830526+0000","last_fullsized":"2024-03-21T14:22:19.830526+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:49.815144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.815144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.315841+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'29","reported_seq":690,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:23.726736+0000","last_change":"2024-03-21T14:17:59.176788+0000","last_active":"2024-03-21T14:22:23.726736+0000","last_peered":"2024-03-21T14:22:23.726736+0000","last_clean":"2024-03-21T14:22:23.726736+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:22:23.726736+0000","last_undegraded":"2024-03-21T14:22:23.726736+0000","last_fullsized":"2024-03-21T14:22:23.726736+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'29","last_scrub_stamp":"2024-03-21T14:17:59.176541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:59.176541+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.466444+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":181,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813190+0000","last_change":"2024-03-21T14:20:33.512686+0000","last_active":"2024-03-21T14:22:19.813190+0000","last_peered":"2024-03-21T14:22:19.813190+0000","last_clean":"2024-03-21T14:22:19.813190+0000","last_became_active":"2024-03-21T14:19:14.690145+0000","last_became_peered":"2024-03-21T14:19:14.690145+0000","last_unstale":"2024-03-21T14:22:19.813190+0000","last_undegraded":"2024-03-21T14:22:19.813190+0000","last_fullsized":"2024-03-21T14:22:19.813190+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:33.512571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:33.512571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417141+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"438'5950","reported_seq":6013,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.499579+0000","last_change":"2024-03-21T14:20:59.239970+0000","last_active":"2024-03-21T14:22:27.499579+0000","last_peered":"2024-03-21T14:22:27.499579+0000","last_clean":"2024-03-21T14:22:27.499579+0000","last_became_active":"2024-03-21T14:20:59.239659+0000","last_became_peered":"2024-03-21T14:20:59.239659+0000","last_unstale":"2024-03-21T14:22:27.499579+0000","last_undegraded":"2024-03-21T14:22:27.499579+0000","last_fullsized":"2024-03-21T14:22:27.499579+0000","mapping_epoch":411,"log_start":"426'1800","ondisk_log_start":"426'1800","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:32.356279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":389939200,"num_objects":5950,"num_object_clones":0,"num_object_copies":17850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5950,"num_write_kb":380800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":890,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822749+0000","last_change":"2024-03-21T14:18:43.149509+0000","last_active":"2024-03-21T14:22:19.822749+0000","last_peered":"2024-03-21T14:22:19.822749+0000","last_clean":"2024-03-21T14:22:19.822749+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:22:19.822749+0000","last_undegraded":"2024-03-21T14:22:19.822749+0000","last_fullsized":"2024-03-21T14:22:19.822749+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:18:43.148036+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:18:43.148036+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687507+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":835,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.821831+0000","last_change":"2024-03-21T14:20:48.131092+0000","last_active":"2024-03-21T14:22:19.821831+0000","last_peered":"2024-03-21T14:22:19.821831+0000","last_clean":"2024-03-21T14:22:19.821831+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:22:19.821831+0000","last_undegraded":"2024-03-21T14:22:19.821831+0000","last_fullsized":"2024-03-21T14:22:19.821831+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.130989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.130989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.688045+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":619,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810245+0000","last_change":"2024-03-21T14:20:21.778652+0000","last_active":"2024-03-21T14:22:19.810245+0000","last_peered":"2024-03-21T14:22:19.810245+0000","last_clean":"2024-03-21T14:22:19.810245+0000","last_became_active":"2024-03-21T14:20:21.778326+0000","last_became_peered":"2024-03-21T14:20:21.778326+0000","last_unstale":"2024-03-21T14:22:19.810245+0000","last_undegraded":"2024-03-21T14:22:19.810245+0000","last_fullsized":"2024-03-21T14:22:19.810245+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:50.207374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:50.207374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1928,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:21.550709+0000","last_change":"2024-03-21T14:21:41.487654+0000","last_active":"2024-03-21T14:22:21.550709+0000","last_peered":"2024-03-21T14:22:21.550709+0000","last_clean":"2024-03-21T14:22:21.550709+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:22:21.550709+0000","last_undegraded":"2024-03-21T14:22:21.550709+0000","last_fullsized":"2024-03-21T14:22:21.550709+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:21:41.487571+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.487571+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:10.898181+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1123,"num_read_kb":1118,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"236'24","reported_seq":763,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822603+0000","last_change":"2024-03-21T14:19:34.715158+0000","last_active":"2024-03-21T14:22:19.822603+0000","last_peered":"2024-03-21T14:22:19.822603+0000","last_clean":"2024-03-21T14:22:19.822603+0000","last_became_active":"2024-03-21T14:19:34.692301+0000","last_became_peered":"2024-03-21T14:19:34.692301+0000","last_unstale":"2024-03-21T14:22:19.822603+0000","last_undegraded":"2024-03-21T14:22:19.822603+0000","last_fullsized":"2024-03-21T14:22:19.822603+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:19:24.502253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:24.502253+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687654+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":295,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.828382+0000","last_change":"2024-03-21T14:21:40.525563+0000","last_active":"2024-03-21T14:22:19.828382+0000","last_peered":"2024-03-21T14:22:19.828382+0000","last_clean":"2024-03-21T14:22:19.828382+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:22:19.828382+0000","last_undegraded":"2024-03-21T14:22:19.828382+0000","last_fullsized":"2024-03-21T14:22:19.828382+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:40.525392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:21:40.525392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:48.788089+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"438'5541","reported_seq":5605,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:22.627203+0000","last_change":"2024-03-21T14:20:59.110120+0000","last_active":"2024-03-21T14:22:22.627203+0000","last_peered":"2024-03-21T14:22:22.627203+0000","last_clean":"2024-03-21T14:22:22.627203+0000","last_became_active":"2024-03-21T14:20:59.109825+0000","last_became_peered":"2024-03-21T14:20:59.109825+0000","last_unstale":"2024-03-21T14:22:22.627203+0000","last_undegraded":"2024-03-21T14:22:22.627203+0000","last_fullsized":"2024-03-21T14:22:22.627203+0000","mapping_epoch":411,"log_start":"425'1500","ondisk_log_start":"425'1500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4041,"log_dups_size":0,"ondisk_log_size":4041,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363134976,"num_objects":5541,"num_object_clones":0,"num_object_copies":16623,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5541,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5541,"num_write_kb":354624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":626,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832544+0000","last_change":"2024-03-21T14:20:30.945601+0000","last_active":"2024-03-21T14:22:19.832544+0000","last_peered":"2024-03-21T14:22:19.832544+0000","last_clean":"2024-03-21T14:22:19.832544+0000","last_became_active":"2024-03-21T14:20:24.825637+0000","last_became_peered":"2024-03-21T14:20:24.825637+0000","last_unstale":"2024-03-21T14:22:19.832544+0000","last_undegraded":"2024-03-21T14:22:19.832544+0000","last_fullsized":"2024-03-21T14:22:19.832544+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:30.945471+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.945471+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274405+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":832,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.815017+0000","last_change":"2024-03-21T14:21:15.242152+0000","last_active":"2024-03-21T14:22:19.815017+0000","last_peered":"2024-03-21T14:22:19.815017+0000","last_clean":"2024-03-21T14:22:19.815017+0000","last_became_active":"2024-03-21T14:19:14.697998+0000","last_became_peered":"2024-03-21T14:19:14.697998+0000","last_unstale":"2024-03-21T14:22:19.815017+0000","last_undegraded":"2024-03-21T14:22:19.815017+0000","last_fullsized":"2024-03-21T14:22:19.815017+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:15.242057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:21:15.242057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:39.980548+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":830,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822940+0000","last_change":"2024-03-21T14:19:26.469534+0000","last_active":"2024-03-21T14:22:19.822940+0000","last_peered":"2024-03-21T14:22:19.822940+0000","last_clean":"2024-03-21T14:22:19.822940+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:22:19.822940+0000","last_undegraded":"2024-03-21T14:22:19.822940+0000","last_fullsized":"2024-03-21T14:22:19.822940+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:26.469255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:19:26.469255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687622+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"236'29","reported_seq":551,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.796018+0000","last_change":"2024-03-21T14:20:31.530370+0000","last_active":"2024-03-21T14:22:19.796018+0000","last_peered":"2024-03-21T14:22:19.796018+0000","last_clean":"2024-03-21T14:22:19.796018+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:22:19.796018+0000","last_undegraded":"2024-03-21T14:22:19.796018+0000","last_fullsized":"2024-03-21T14:22:19.796018+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:20:31.530270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.530270+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.505706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":185,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812511+0000","last_change":"2024-03-21T14:20:34.551705+0000","last_active":"2024-03-21T14:22:19.812511+0000","last_peered":"2024-03-21T14:22:19.812511+0000","last_clean":"2024-03-21T14:22:19.812511+0000","last_became_active":"2024-03-21T14:19:14.675538+0000","last_became_peered":"2024-03-21T14:19:14.675538+0000","last_unstale":"2024-03-21T14:22:19.812511+0000","last_undegraded":"2024-03-21T14:22:19.812511+0000","last_fullsized":"2024-03-21T14:22:19.812511+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:34.551577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:20:34.551577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.416978+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"438'5626","reported_seq":8613,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:23.034035+0000","last_change":"2024-03-21T14:21:41.926494+0000","last_active":"2024-03-21T14:22:23.034035+0000","last_peered":"2024-03-21T14:22:23.034035+0000","last_clean":"2024-03-21T14:22:23.034035+0000","last_became_active":"2024-03-21T14:21:41.923763+0000","last_became_peered":"2024-03-21T14:21:41.923763+0000","last_unstale":"2024-03-21T14:22:23.034035+0000","last_undegraded":"2024-03-21T14:22:23.034035+0000","last_fullsized":"2024-03-21T14:22:23.034035+0000","mapping_epoch":431,"log_start":"418'600","ondisk_log_start":"418'600","created":411,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5026,"log_dups_size":0,"ondisk_log_size":5026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.417376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368705536,"num_objects":5626,"num_object_clones":0,"num_object_copies":16878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5626,"num_write_kb":360064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":95420416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":800,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832556+0000","last_change":"2024-03-21T14:20:24.821031+0000","last_active":"2024-03-21T14:22:19.832556+0000","last_peered":"2024-03-21T14:22:19.832556+0000","last_clean":"2024-03-21T14:22:19.832556+0000","last_became_active":"2024-03-21T14:20:24.819844+0000","last_became_peered":"2024-03-21T14:20:24.819844+0000","last_unstale":"2024-03-21T14:22:19.832556+0000","last_undegraded":"2024-03-21T14:22:19.832556+0000","last_fullsized":"2024-03-21T14:22:19.832556+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:15.665015+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.665015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":324,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.830890+0000","last_change":"2024-03-21T14:19:15.688754+0000","last_active":"2024-03-21T14:22:19.830890+0000","last_peered":"2024-03-21T14:22:19.830890+0000","last_clean":"2024-03-21T14:22:19.830890+0000","last_became_active":"2024-03-21T14:19:15.688238+0000","last_became_peered":"2024-03-21T14:19:15.688238+0000","last_unstale":"2024-03-21T14:22:19.830890+0000","last_undegraded":"2024-03-21T14:22:19.830890+0000","last_fullsized":"2024-03-21T14:22:19.830890+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:11.920076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:11.920076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274137+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":760,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.834185+0000","last_change":"2024-03-21T14:20:32.884624+0000","last_active":"2024-03-21T14:22:19.834185+0000","last_peered":"2024-03-21T14:22:19.834185+0000","last_clean":"2024-03-21T14:22:19.834185+0000","last_became_active":"2024-03-21T14:19:14.695943+0000","last_became_peered":"2024-03-21T14:19:14.695943+0000","last_unstale":"2024-03-21T14:22:19.834185+0000","last_undegraded":"2024-03-21T14:22:19.834185+0000","last_fullsized":"2024-03-21T14:22:19.834185+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:32.884514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:32.884514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.274352+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"236'25","reported_seq":868,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822441+0000","last_change":"2024-03-21T14:21:43.496288+0000","last_active":"2024-03-21T14:22:19.822441+0000","last_peered":"2024-03-21T14:22:19.822441+0000","last_clean":"2024-03-21T14:22:19.822441+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:22:19.822441+0000","last_undegraded":"2024-03-21T14:22:19.822441+0000","last_fullsized":"2024-03-21T14:22:19.822441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:43.496137+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:21:43.496137+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:49.418037+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":816,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.829865+0000","last_change":"2024-03-21T14:18:44.046070+0000","last_active":"2024-03-21T14:22:19.829865+0000","last_peered":"2024-03-21T14:22:19.829865+0000","last_clean":"2024-03-21T14:22:19.829865+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:22:19.829865+0000","last_undegraded":"2024-03-21T14:22:19.829865+0000","last_fullsized":"2024-03-21T14:22:19.829865+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'11","last_scrub_stamp":"2024-03-21T14:18:44.045985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:44.045985+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.318495+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"438'5878","reported_seq":5942,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:26.844935+0000","last_change":"2024-03-21T14:20:59.110795+0000","last_active":"2024-03-21T14:22:26.844935+0000","last_peered":"2024-03-21T14:22:26.844935+0000","last_clean":"2024-03-21T14:22:26.844935+0000","last_became_active":"2024-03-21T14:20:59.110468+0000","last_became_peered":"2024-03-21T14:20:59.110468+0000","last_unstale":"2024-03-21T14:22:26.844935+0000","last_undegraded":"2024-03-21T14:22:26.844935+0000","last_fullsized":"2024-03-21T14:22:26.844935+0000","mapping_epoch":411,"log_start":"425'1500","ondisk_log_start":"425'1500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.466867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":385220608,"num_objects":5878,"num_object_clones":0,"num_object_copies":17634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5878,"num_write_kb":376192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":882,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.829736+0000","last_change":"2024-03-21T14:20:52.764101+0000","last_active":"2024-03-21T14:22:19.829736+0000","last_peered":"2024-03-21T14:22:19.829736+0000","last_clean":"2024-03-21T14:22:19.829736+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:22:19.829736+0000","last_undegraded":"2024-03-21T14:22:19.829736+0000","last_fullsized":"2024-03-21T14:22:19.829736+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:52.763988+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.763988+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.315875+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.825541+0000","last_change":"2024-03-21T14:20:29.685381+0000","last_active":"2024-03-21T14:22:19.825541+0000","last_peered":"2024-03-21T14:22:19.825541+0000","last_clean":"2024-03-21T14:22:19.825541+0000","last_became_active":"2024-03-21T14:19:14.684876+0000","last_became_peered":"2024-03-21T14:19:14.684876+0000","last_unstale":"2024-03-21T14:22:19.825541+0000","last_undegraded":"2024-03-21T14:22:19.825541+0000","last_fullsized":"2024-03-21T14:22:19.825541+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:29.685267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:27.609702+0000","last_clean_scrub_stamp":"2024-03-21T14:20:29.685267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:23.846447+0000","scrub_duration":155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":696,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.822431+0000","last_change":"2024-03-21T14:20:51.116248+0000","last_active":"2024-03-21T14:22:19.822431+0000","last_peered":"2024-03-21T14:22:19.822431+0000","last_clean":"2024-03-21T14:22:19.822431+0000","last_became_active":"2024-03-21T14:19:34.692258+0000","last_became_peered":"2024-03-21T14:19:34.692258+0000","last_unstale":"2024-03-21T14:22:19.822431+0000","last_undegraded":"2024-03-21T14:22:19.822431+0000","last_fullsized":"2024-03-21T14:22:19.822431+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:20:51.116139+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.116139+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.688076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"237'28","reported_seq":894,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810216+0000","last_change":"2024-03-21T14:21:26.778465+0000","last_active":"2024-03-21T14:22:19.810216+0000","last_peered":"2024-03-21T14:22:19.810216+0000","last_clean":"2024-03-21T14:22:19.810216+0000","last_became_active":"2024-03-21T14:21:26.768622+0000","last_became_peered":"2024-03-21T14:21:26.768622+0000","last_unstale":"2024-03-21T14:22:19.810216+0000","last_undegraded":"2024-03-21T14:22:19.810216+0000","last_fullsized":"2024-03-21T14:22:19.810216+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'28","last_scrub_stamp":"2024-03-21T14:17:56.127254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:17:56.127254+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":1060,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.831886+0000","last_change":"2024-03-21T14:18:38.115200+0000","last_active":"2024-03-21T14:22:19.831886+0000","last_peered":"2024-03-21T14:22:19.831886+0000","last_clean":"2024-03-21T14:22:19.831886+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:22:19.831886+0000","last_undegraded":"2024-03-21T14:22:19.831886+0000","last_fullsized":"2024-03-21T14:22:19.831886+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:18:38.115111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:18:38.115111+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.316164+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"438'5862","reported_seq":6712,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:26.849849+0000","last_change":"2024-03-21T14:21:28.769176+0000","last_active":"2024-03-21T14:22:26.849849+0000","last_peered":"2024-03-21T14:22:26.849849+0000","last_clean":"2024-03-21T14:22:26.849849+0000","last_became_active":"2024-03-21T14:21:28.766274+0000","last_became_peered":"2024-03-21T14:21:28.766274+0000","last_unstale":"2024-03-21T14:22:26.849849+0000","last_undegraded":"2024-03-21T14:22:26.849849+0000","last_fullsized":"2024-03-21T14:22:26.849849+0000","mapping_epoch":425,"log_start":"426'1500","ondisk_log_start":"426'1500","created":411,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4362,"log_dups_size":0,"ondisk_log_size":4362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.466809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":384172032,"num_objects":5862,"num_object_clones":0,"num_object_copies":17586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5862,"num_write_kb":375168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":59375616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":884,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.811073+0000","last_change":"2024-03-21T14:21:19.302873+0000","last_active":"2024-03-21T14:22:19.811073+0000","last_peered":"2024-03-21T14:22:19.811073+0000","last_clean":"2024-03-21T14:22:19.811073+0000","last_became_active":"2024-03-21T14:19:14.682251+0000","last_became_peered":"2024-03-21T14:19:14.682251+0000","last_unstale":"2024-03-21T14:22:19.811073+0000","last_undegraded":"2024-03-21T14:22:19.811073+0000","last_fullsized":"2024-03-21T14:22:19.811073+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:19.302784+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.302784+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:22.201633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810344+0000","last_change":"2024-03-21T14:20:21.782339+0000","last_active":"2024-03-21T14:22:19.810344+0000","last_peered":"2024-03-21T14:22:19.810344+0000","last_clean":"2024-03-21T14:22:19.810344+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:22:19.810344+0000","last_undegraded":"2024-03-21T14:22:19.810344+0000","last_fullsized":"2024-03-21T14:22:19.810344+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:11.405823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:17:11.405823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":829,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832915+0000","last_change":"2024-03-21T14:21:39.060809+0000","last_active":"2024-03-21T14:22:19.832915+0000","last_peered":"2024-03-21T14:22:19.832915+0000","last_clean":"2024-03-21T14:22:19.832915+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:22:19.832915+0000","last_undegraded":"2024-03-21T14:22:19.832915+0000","last_fullsized":"2024-03-21T14:22:19.832915+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:39.060697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:21:39.060697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:03.254471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"262'38","reported_seq":122,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.821474+0000","last_change":"2024-03-21T14:20:21.803142+0000","last_active":"2024-03-21T14:22:19.821474+0000","last_peered":"2024-03-21T14:22:19.821474+0000","last_clean":"2024-03-21T14:22:19.821474+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:22:19.821474+0000","last_undegraded":"2024-03-21T14:22:19.821474+0000","last_fullsized":"2024-03-21T14:22:19.821474+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'38","last_scrub_stamp":"2024-03-21T14:19:12.604917+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:19:12.604917+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.318553+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":796,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810061+0000","last_change":"2024-03-21T14:17:14.254451+0000","last_active":"2024-03-21T14:22:19.810061+0000","last_peered":"2024-03-21T14:22:19.810061+0000","last_clean":"2024-03-21T14:22:19.810061+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:22:19.810061+0000","last_undegraded":"2024-03-21T14:22:19.810061+0000","last_fullsized":"2024-03-21T14:22:19.810061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:14.254334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:14.254334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"438'5518","reported_seq":5582,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:22.631299+0000","last_change":"2024-03-21T14:20:59.109540+0000","last_active":"2024-03-21T14:22:22.631299+0000","last_peered":"2024-03-21T14:22:22.631299+0000","last_clean":"2024-03-21T14:22:22.631299+0000","last_became_active":"2024-03-21T14:20:59.109191+0000","last_became_peered":"2024-03-21T14:20:59.109191+0000","last_unstale":"2024-03-21T14:22:22.631299+0000","last_undegraded":"2024-03-21T14:22:22.631299+0000","last_fullsized":"2024-03-21T14:22:22.631299+0000","mapping_epoch":411,"log_start":"426'1500","ondisk_log_start":"426'1500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4018,"log_dups_size":0,"ondisk_log_size":4018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:22.998158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361627648,"num_objects":5518,"num_object_clones":0,"num_object_copies":16554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5518,"num_write_kb":353152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":797,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.813323+0000","last_change":"2024-03-21T14:20:36.503605+0000","last_active":"2024-03-21T14:22:19.813323+0000","last_peered":"2024-03-21T14:22:19.813323+0000","last_clean":"2024-03-21T14:22:19.813323+0000","last_became_active":"2024-03-21T14:19:14.692919+0000","last_became_peered":"2024-03-21T14:19:14.692919+0000","last_unstale":"2024-03-21T14:22:19.813323+0000","last_undegraded":"2024-03-21T14:22:19.813323+0000","last_fullsized":"2024-03-21T14:22:19.813323+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:36.503498+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:36.503498+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":837,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.821792+0000","last_change":"2024-03-21T14:20:43.142589+0000","last_active":"2024-03-21T14:22:19.821792+0000","last_peered":"2024-03-21T14:22:19.821792+0000","last_clean":"2024-03-21T14:22:19.821792+0000","last_became_active":"2024-03-21T14:19:34.691121+0000","last_became_peered":"2024-03-21T14:19:34.691121+0000","last_unstale":"2024-03-21T14:22:19.821792+0000","last_undegraded":"2024-03-21T14:22:19.821792+0000","last_fullsized":"2024-03-21T14:22:19.821792+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:43.142495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:20:43.142495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.688017+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1050,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810967+0000","last_change":"2024-03-21T14:17:03.353514+0000","last_active":"2024-03-21T14:22:19.810967+0000","last_peered":"2024-03-21T14:22:19.810967+0000","last_clean":"2024-03-21T14:22:19.810967+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:22:19.810967+0000","last_undegraded":"2024-03-21T14:22:19.810967+0000","last_fullsized":"2024-03-21T14:22:19.810967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:17:03.353394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:17:03.353394+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:27.466256+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":111,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.828412+0000","last_change":"2024-03-21T14:21:54.066071+0000","last_active":"2024-03-21T14:22:19.828412+0000","last_peered":"2024-03-21T14:22:19.828412+0000","last_clean":"2024-03-21T14:22:19.828412+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:22:19.828412+0000","last_undegraded":"2024-03-21T14:22:19.828412+0000","last_fullsized":"2024-03-21T14:22:19.828412+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.065922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.065922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:13.357461+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":889,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.828077+0000","last_change":"2024-03-21T14:21:19.723743+0000","last_active":"2024-03-21T14:22:19.828077+0000","last_peered":"2024-03-21T14:22:19.828077+0000","last_clean":"2024-03-21T14:22:19.828077+0000","last_became_active":"2024-03-21T14:21:19.713483+0000","last_became_peered":"2024-03-21T14:21:19.713483+0000","last_unstale":"2024-03-21T14:22:19.828077+0000","last_undegraded":"2024-03-21T14:22:19.828077+0000","last_fullsized":"2024-03-21T14:22:19.828077+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:20:42.035180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.035180+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.687918+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"438'5911","reported_seq":5975,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.483920+0000","last_change":"2024-03-21T14:20:59.240921+0000","last_active":"2024-03-21T14:22:27.483920+0000","last_peered":"2024-03-21T14:22:27.483920+0000","last_clean":"2024-03-21T14:22:27.483920+0000","last_became_active":"2024-03-21T14:20:59.240641+0000","last_became_peered":"2024-03-21T14:20:59.240641+0000","last_unstale":"2024-03-21T14:22:27.483920+0000","last_undegraded":"2024-03-21T14:22:27.483920+0000","last_fullsized":"2024-03-21T14:22:27.483920+0000","mapping_epoch":411,"log_start":"426'1800","ondisk_log_start":"426'1800","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4111,"log_dups_size":0,"ondisk_log_size":4111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:32.356218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":387383296,"num_objects":5911,"num_object_clones":0,"num_object_copies":17733,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5911,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5911,"num_write_kb":378304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":814,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.832038+0000","last_change":"2024-03-21T14:18:28.880752+0000","last_active":"2024-03-21T14:22:19.832038+0000","last_peered":"2024-03-21T14:22:19.832038+0000","last_clean":"2024-03-21T14:22:19.832038+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:22:19.832038+0000","last_undegraded":"2024-03-21T14:22:19.832038+0000","last_fullsized":"2024-03-21T14:22:19.832038+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:28.880620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:18:28.880620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.273847+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":812,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.811270+0000","last_change":"2024-03-21T14:21:51.322736+0000","last_active":"2024-03-21T14:22:19.811270+0000","last_peered":"2024-03-21T14:22:19.811270+0000","last_clean":"2024-03-21T14:22:19.811270+0000","last_became_active":"2024-03-21T14:21:26.767687+0000","last_became_peered":"2024-03-21T14:21:26.767687+0000","last_unstale":"2024-03-21T14:22:19.811270+0000","last_undegraded":"2024-03-21T14:22:19.811270+0000","last_fullsized":"2024-03-21T14:22:19.811270+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.322658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.322658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:12.324516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":821,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810113+0000","last_change":"2024-03-21T14:20:48.538833+0000","last_active":"2024-03-21T14:22:19.810113+0000","last_peered":"2024-03-21T14:22:19.810113+0000","last_clean":"2024-03-21T14:22:19.810113+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:22:19.810113+0000","last_undegraded":"2024-03-21T14:22:19.810113+0000","last_fullsized":"2024-03-21T14:22:19.810113+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.538744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.538744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:24.545211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"237'19","reported_seq":856,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810607+0000","last_change":"2024-03-21T14:18:32.414098+0000","last_active":"2024-03-21T14:22:19.810607+0000","last_peered":"2024-03-21T14:22:19.810607+0000","last_clean":"2024-03-21T14:22:19.810607+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:22:19.810607+0000","last_undegraded":"2024-03-21T14:22:19.810607+0000","last_fullsized":"2024-03-21T14:22:19.810607+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:18:32.413975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:32.413975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"438'5539","reported_seq":5603,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:23.007078+0000","last_change":"2024-03-21T14:20:59.239976+0000","last_active":"2024-03-21T14:22:23.007078+0000","last_peered":"2024-03-21T14:22:23.007078+0000","last_clean":"2024-03-21T14:22:23.007078+0000","last_became_active":"2024-03-21T14:20:59.239396+0000","last_became_peered":"2024-03-21T14:20:59.239396+0000","last_unstale":"2024-03-21T14:22:23.007078+0000","last_undegraded":"2024-03-21T14:22:23.007078+0000","last_fullsized":"2024-03-21T14:22:23.007078+0000","mapping_epoch":411,"log_start":"418'500","ondisk_log_start":"418'500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5039,"log_dups_size":0,"ondisk_log_size":5039,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:26.447375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363003904,"num_objects":5539,"num_object_clones":0,"num_object_copies":16617,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5539,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5539,"num_write_kb":354496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":797,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.806971+0000","last_change":"2024-03-21T14:20:37.552524+0000","last_active":"2024-03-21T14:22:19.806971+0000","last_peered":"2024-03-21T14:22:19.806971+0000","last_clean":"2024-03-21T14:22:19.806971+0000","last_became_active":"2024-03-21T14:19:14.700353+0000","last_became_peered":"2024-03-21T14:19:14.700353+0000","last_unstale":"2024-03-21T14:22:19.806971+0000","last_undegraded":"2024-03-21T14:22:19.806971+0000","last_fullsized":"2024-03-21T14:22:19.806971+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:37.552424+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.552424+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":616,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.808736+0000","last_change":"2024-03-21T14:20:45.611947+0000","last_active":"2024-03-21T14:22:19.808736+0000","last_peered":"2024-03-21T14:22:19.808736+0000","last_clean":"2024-03-21T14:22:19.808736+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:22:19.808736+0000","last_undegraded":"2024-03-21T14:22:19.808736+0000","last_fullsized":"2024-03-21T14:22:19.808736+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.611844+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417231+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":828,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.823106+0000","last_change":"2024-03-21T14:21:54.619575+0000","last_active":"2024-03-21T14:22:19.823106+0000","last_peered":"2024-03-21T14:22:19.823106+0000","last_clean":"2024-03-21T14:22:19.823106+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:22:19.823106+0000","last_undegraded":"2024-03-21T14:22:19.823106+0000","last_fullsized":"2024-03-21T14:22:19.823106+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.617817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:08.722294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":107,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.812398+0000","last_change":"2024-03-21T14:20:19.703923+0000","last_active":"2024-03-21T14:22:19.812398+0000","last_peered":"2024-03-21T14:22:19.812398+0000","last_clean":"2024-03-21T14:22:19.812398+0000","last_became_active":"2024-03-21T14:20:19.703587+0000","last_became_peered":"2024-03-21T14:20:19.703587+0000","last_unstale":"2024-03-21T14:22:19.812398+0000","last_undegraded":"2024-03-21T14:22:19.812398+0000","last_fullsized":"2024-03-21T14:22:19.812398+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:17:16.247145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:17:16.247145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.416910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":857,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.811299+0000","last_change":"2024-03-21T14:20:46.480561+0000","last_active":"2024-03-21T14:22:19.811299+0000","last_peered":"2024-03-21T14:22:19.811299+0000","last_clean":"2024-03-21T14:22:19.811299+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:22:19.811299+0000","last_undegraded":"2024-03-21T14:22:19.811299+0000","last_fullsized":"2024-03-21T14:22:19.811299+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:20:46.480457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.480457+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"438'5853","reported_seq":5917,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:27.083609+0000","last_change":"2024-03-21T14:20:59.387035+0000","last_active":"2024-03-21T14:22:27.083609+0000","last_peered":"2024-03-21T14:22:27.083609+0000","last_clean":"2024-03-21T14:22:27.083609+0000","last_became_active":"2024-03-21T14:20:59.386715+0000","last_became_peered":"2024-03-21T14:20:59.386715+0000","last_unstale":"2024-03-21T14:22:27.083609+0000","last_undegraded":"2024-03-21T14:22:27.083609+0000","last_fullsized":"2024-03-21T14:22:27.083609+0000","mapping_epoch":411,"log_start":"421'1200","ondisk_log_start":"421'1200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4653,"log_dups_size":0,"ondisk_log_size":4653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:31.711924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":383582208,"num_objects":5853,"num_object_clones":0,"num_object_copies":17559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5853,"num_write_kb":374592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1141,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.824700+0000","last_change":"2024-03-21T14:21:18.094654+0000","last_active":"2024-03-21T14:22:19.824700+0000","last_peered":"2024-03-21T14:22:19.824700+0000","last_clean":"2024-03-21T14:22:19.824700+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:22:19.824700+0000","last_undegraded":"2024-03-21T14:22:19.824700+0000","last_fullsized":"2024-03-21T14:22:19.824700+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:18.094572+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:18.094572+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810961+0000","last_change":"2024-03-21T14:19:15.689377+0000","last_active":"2024-03-21T14:22:19.810961+0000","last_peered":"2024-03-21T14:22:19.810961+0000","last_clean":"2024-03-21T14:22:19.810961+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:22:19.810961+0000","last_undegraded":"2024-03-21T14:22:19.810961+0000","last_fullsized":"2024-03-21T14:22:19.810961+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:13.493205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:13.493205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":824,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.807768+0000","last_change":"2024-03-21T14:20:49.535756+0000","last_active":"2024-03-21T14:22:19.807768+0000","last_peered":"2024-03-21T14:22:19.807768+0000","last_clean":"2024-03-21T14:22:19.807768+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:22:19.807768+0000","last_undegraded":"2024-03-21T14:22:19.807768+0000","last_fullsized":"2024-03-21T14:22:19.807768+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.535652+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:22:21.417287+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":840,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.830071+0000","last_change":"2024-03-21T14:21:46.110127+0000","last_active":"2024-03-21T14:22:19.830071+0000","last_peered":"2024-03-21T14:22:19.830071+0000","last_clean":"2024-03-21T14:22:19.830071+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:22:19.830071+0000","last_undegraded":"2024-03-21T14:22:19.830071+0000","last_fullsized":"2024-03-21T14:22:19.830071+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.110008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.110008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:13.331279+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"236'25","reported_seq":860,"reported_epoch":438,"state":"active+clean","last_fresh":"2024-03-21T14:22:19.810963+0000","last_change":"2024-03-21T14:18:34.406758+0000","last_active":"2024-03-21T14:22:19.810963+0000","last_peered":"2024-03-21T14:22:19.810963+0000","last_clean":"2024-03-21T14:22:19.810963+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:22:19.810963+0000","last_undegraded":"2024-03-21T14:22:19.810963+0000","last_fullsized":"2024-03-21T14:22:19.810963+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:18:34.406635+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:18:34.406635+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6031933440,"num_objects":92040,"num_object_clones":0,"num_object_copies":276120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":92040,"num_write_kb":5890560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3767,"num_bytes_recovered":246874112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18099863552,"data_stored":18099863552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70240,"ondisk_log_size":70240,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1123,"num_read_kb":1118,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1113,"num_read_kb":870,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":3812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103411,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3105912,"kb_used_data":2236208,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":90639240,"statfs":{"total":95995035648,"available":92814581760,"internally_reserved":0,"allocated":2289876992,"data_stored":2288812700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":890567114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":351,"seq":1507533520938,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":806204,"kb_used_data":728692,"kb_used_omap":4,"kb_used_meta":77499,"kb_avail":92938948,"statfs":{"total":95995035648,"available":95169482752,"internally_reserved":0,"allocated":746180608,"data_stored":745174850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":79359652},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575122,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4018376,"kb_used_data":3166536,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":89726776,"statfs":{"total":95995035648,"available":91880218624,"internally_reserved":0,"allocated":3242532864,"data_stored":3241444587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738648,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3065288,"kb_used_data":2240000,"kb_used_omap":1,"kb_used_meta":825278,"kb_avail":90679864,"statfs":{"total":95995035648,"available":92856180736,"internally_reserved":0,"allocated":2293760000,"data_stored":2292678548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":845085125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598292007,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3679984,"kb_used_data":3375336,"kb_used_omap":7,"kb_used_meta":304632,"kb_avail":90065168,"statfs":{"total":95995035648,"available":92226732032,"internally_reserved":0,"allocated":3456344064,"data_stored":3455287003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":311943300},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182663,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3765268,"kb_used_data":2994124,"kb_used_omap":1,"kb_used_meta":771134,"kb_avail":89979884,"statfs":{"total":95995035648,"available":92139401216,"internally_reserved":0,"allocated":3065982976,"data_stored":3064890043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":789641671},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,0,2,2,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019139,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2749896,"kb_used_data":1870464,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":90995256,"statfs":{"total":95995035648,"available":93179142144,"internally_reserved":0,"allocated":1915355136,"data_stored":1914273661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692094,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2731140,"kb_used_data":1865596,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":91014012,"statfs":{"total":95995035648,"available":93198348288,"internally_reserved":0,"allocated":1910370304,"data_stored":1909294120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":886307269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2306211840,"data_stored":2306211840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3276734464,"data_stored":3276734464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2562260992,"data_stored":2562260992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3068526592,"data_stored":3068526592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1922695168,"data_stored":1922695168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1918369792,"data_stored":1918369792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":746651648,"data_stored":746651648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2298413056,"data_stored":2298413056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:22:30.643 INFO:tasks.thrashosds.thrasher:Setting 3.c to [0, 6, 7] 2024-03-21T14:22:30.643 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.c', '0', '6', '7'] 2024-03-21T14:22:30.643 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 3.c 0 6 7 2024-03-21T14:22:30.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:30.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:30.771 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:30.796 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:30.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:30.898 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:30.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:31.048 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:31.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:31.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:30 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:22:31.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:30 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:22:31.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:30 smithi003 ceph-mon[29576]: pgmap v947: 158 pgs: 158 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:30 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:22:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:30 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:22:31.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:30 smithi003 ceph-mon[40510]: pgmap v947: 158 pgs: 158 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:30 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:22:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:30 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:22:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:30 smithi118 ceph-mon[33862]: pgmap v947: 158 pgs: 158 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-21T14:22:31.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:31.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:31.380 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:31.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:31.509 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:31.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 78 16 98434 98418 78.8475 60.9375 0.015677 0.0126775 2024-03-21T14:22:31.610 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:31.638 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:31.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:31.740 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:31.768 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:31.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:31.897 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:31.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:31.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:32.027 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:32.129 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:31 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:22:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:31 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:22:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:31 smithi118 ceph-mon[33862]: 11.0 scrub starts 2024-03-21T14:22:32.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:32.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:31 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:22:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:31 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:22:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:31 smithi003 ceph-mon[29576]: 11.0 scrub starts 2024-03-21T14:22:32.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:31 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:22:32.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:31 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:22:32.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:31 smithi003 ceph-mon[40510]: 11.0 scrub starts 2024-03-21T14:22:32.295 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:32.342 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:32.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:32.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:32.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:32.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:32.597 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:32.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:32.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 79 16 99423 99407 78.6318 61.8125 0.0149894 0.0127127 2024-03-21T14:22:32.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:32.724 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:32.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:32.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:32.868 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:32.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:32.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:32.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:33.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:33.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:33.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:33.134 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:33.155 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:22:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:32 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:22:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:32 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:22:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:32 smithi118 ceph-mon[33862]: pgmap v948: 158 pgs: 158 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[29576]: pgmap v948: 158 pgs: 158 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[40510]: pgmap v948: 158 pgs: 158 active+clean; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:33.197 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:33.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:33.266 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:33.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:33.394 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:33.496 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:33.527 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:33.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:2024-03-21T14:22:33.602936+0000 min lat: 0.00125552 max lat: 2.43115 avg lat: 0.0127551 2024-03-21T14:22:33.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-21T14:22:33.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 80 16 100346 100330 78.3699 57.6875 0.00844088 0.0127551 2024-03-21T14:22:33.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:33.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.672 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:33.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:33.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:33.798 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:33.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:33.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:33.939 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:33.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:34.041 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:34.082 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:34.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:33 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:22:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:33 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:22:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:33 smithi118 ceph-mon[33862]: 1.3 scrub starts 2024-03-21T14:22:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:33 smithi118 ceph-mon[33862]: 1.3 scrub ok 2024-03-21T14:22:34.229 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:34.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:22:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:22:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[29576]: 1.3 scrub starts 2024-03-21T14:22:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[29576]: 1.3 scrub ok 2024-03-21T14:22:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:22:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:22:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[40510]: 1.3 scrub starts 2024-03-21T14:22:34.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:33 smithi003 ceph-mon[40510]: 1.3 scrub ok 2024-03-21T14:22:34.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.361 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:34.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:34.462 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:34.491 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:34.593 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:34.602 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 81 16 101470 101454 78.2694 70.25 0.0197164 0.0127705 2024-03-21T14:22:34.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:34.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:34.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:34.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:34.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:34.894 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:34.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:34.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:34.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:35.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:35.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:35.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[29576]: 11.0 scrub ok 2024-03-21T14:22:35.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:22:35.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:22:35.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[29576]: pgmap v949: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:35.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[40510]: 11.0 scrub ok 2024-03-21T14:22:35.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:22:35.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:22:35.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:34 smithi003 ceph-mon[40510]: pgmap v949: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:35.145 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:22:35.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:34 smithi118 ceph-mon[33862]: 11.0 scrub ok 2024-03-21T14:22:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:34 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:22:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:34 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:22:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:34 smithi118 ceph-mon[33862]: pgmap v949: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:35.193 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:22:35.238 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:22:35.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:35.280 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:22:35.322 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:35.323 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:22:35.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:35.363 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:22:35.424 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:35.437 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:22:35.450 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:35.476 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:22:35.483 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:22:35.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:35.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 82 16 102491 102475 78.0925 63.8125 0.0310062 0.0127997 2024-03-21T14:22:35.604 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:35.705 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:35.734 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:35.836 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:35.868 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:35.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:35.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:35.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:36.011 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:36.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:36.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3103685376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:36.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3103685376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:36.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.155 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:36.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3103685376' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:36.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:36.284 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:36.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:36.386 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:36.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:36.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:36.540 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:22:36.540 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:22:36.563 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:36.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:36.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 83 16 103593 103577 77.9814 68.875 0.0089937 0.0128185 2024-03-21T14:22:36.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:36.694 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:22:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:22:36.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:36.821 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.822 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:36.923 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:36.952 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:36.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:37.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:37.094 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:37.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: 6.2 scrub starts 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: 6.2 scrub ok 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: pgmap v950: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:22:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:36 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:22:37.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:37.236 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:37.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:22:37.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: 6.2 scrub starts 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: 6.2 scrub ok 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: pgmap v950: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: 6.2 scrub starts 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: 6.2 scrub ok 2024-03-21T14:22:37.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: pgmap v950: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-21T14:22:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:22:37.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:36 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:22:37.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:37.367 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:37.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:37.469 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:37.498 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:37.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:37.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 84 16 104648 104632 77.8379 65.9375 0.0178964 0.0128422 2024-03-21T14:22:37.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:37.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:37.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:37.772 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:37.874 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:37.904 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:22:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:22:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:37.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:38.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:38.043 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:38.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:38.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:38.173 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:38.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:37 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:37 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:37 smithi118 ceph-mon[33862]: osdmap e440: 8 total, 8 up, 8 in 2024-03-21T14:22:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:37 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:37 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:38.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:37 smithi003 ceph-mon[29576]: osdmap e440: 8 total, 8 up, 8 in 2024-03-21T14:22:38.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:37 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:38.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:37 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:38.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:37 smithi003 ceph-mon[40510]: osdmap e440: 8 total, 8 up, 8 in 2024-03-21T14:22:38.275 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:38.301 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:38.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:38.434 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:38.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:38.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:22:38.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:38.581 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:38.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 85 16 105584 105568 77.6104 58.5 0.00533602 0.0128798 2024-03-21T14:22:38.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:38.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.727 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:38.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:38.828 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:38.874 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:38.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:38.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:38.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:39.004 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:39.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:39.106 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:39.150 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:39.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:39.251 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[29576]: pgmap v952: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 581 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[40510]: pgmap v952: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 581 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:22:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:38 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:22:39.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.294 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:39.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:39.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:39.426 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:39.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:38 smithi118 ceph-mon[33862]: pgmap v952: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.3 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 581 B/s rd, 61 MiB/s wr, 969 op/s 2024-03-21T14:22:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:38 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:22:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:38 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:22:39.528 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:39.558 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:39.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 86 16 106646 106630 77.4797 66.375 0.00466206 0.0129017 2024-03-21T14:22:39.661 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:39.699 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:39.801 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:39.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.826 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:39.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:39.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:39.972 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:39.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:39.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:40.074 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:40.105 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:40.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:40.248 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:22:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:22:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[29576]: osdmap e441: 8 total, 8 up, 8 in 2024-03-21T14:22:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:22:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:22:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:39 smithi003 ceph-mon[40510]: osdmap e441: 8 total, 8 up, 8 in 2024-03-21T14:22:40.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:40.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:39 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:22:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:39 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:22:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:39 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:39 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:39 smithi118 ceph-mon[33862]: osdmap e441: 8 total, 8 up, 8 in 2024-03-21T14:22:40.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:40.511 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:40.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 87 16 107675 107659 77.3283 64.3125 0.00717475 0.0129273 2024-03-21T14:22:40.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:40.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:40.757 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:40.782 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:40.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:40.884 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:40.914 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:40.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:40.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:41.015 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:41.143 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:41.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.186 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:41.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:22:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:22:41.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: pgmap v954: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: 6.12 scrub starts 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[29576]: 6.12 scrub ok 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: pgmap v954: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: 6.12 scrub starts 2024-03-21T14:22:41.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:40 smithi003 ceph-mon[40510]: 6.12 scrub ok 2024-03-21T14:22:41.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:41.314 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:41.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: pgmap v954: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: 6.12 scrub starts 2024-03-21T14:22:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:40 smithi118 ceph-mon[33862]: 6.12 scrub ok 2024-03-21T14:22:41.444 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:41.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:41.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:41.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 88 16 108761 108745 77.2208 67.875 0.00974188 0.0129436 2024-03-21T14:22:41.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:41.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:41.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:41.847 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:41.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:41.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:41.974 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:41.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:41.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:42.076 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:42.104 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:42.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:42.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:42.205 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:42.245 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:22:42.246 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:41 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:22:42.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:42.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:42.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:42.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-21T14:22:42.348 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:42.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:42.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:42.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:41 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:22:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:41 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:22:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:41 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:22:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:41 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:22:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:41 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:22:42.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:41 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:22:42.517 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:42.564 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:42.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:42.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout: 89 16 109861 109845 77.1256 68.75 0.0246754 0.0129607 2024-03-21T14:22:42.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:42.693 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:42.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:42.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:42.763 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:42.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:42.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:42.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:42.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:43.012 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:43.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:43.113 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:43.113 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:43.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:43.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:43.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:42 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:22:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:42 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:22:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:42 smithi003 ceph-mon[29576]: pgmap v955: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 650 op/s 2024-03-21T14:22:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:42 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:22:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:42 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:22:43.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:42 smithi003 ceph-mon[40510]: pgmap v955: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 650 op/s 2024-03-21T14:22:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.295 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:43.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:43.396 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:42 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:42 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:22:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:42 smithi118 ceph-mon[33862]: pgmap v955: 158 pgs: 158 active+clean; 6.4 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 41 MiB/s wr, 650 op/s 2024-03-21T14:22:43.442 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:43.543 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:43.571 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:43.603 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total time run: 90.007 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Total writes made: 110879 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Write size: 65536 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Object size: 65536 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Bandwidth (MB/sec): 76.9934 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Bandwidth: 29.1104 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max bandwidth (MB/sec): 183.438 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min bandwidth (MB/sec): 0 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average IOPS: 1231 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev IOPS: 465.784 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max IOPS: 2935 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min IOPS: 0 2024-03-21T14:22:43.604 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Average Latency(s): 0.0129836 2024-03-21T14:22:43.605 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Stddev Latency(s): 0.0370977 2024-03-21T14:22:43.605 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Max latency(s): 2.43115 2024-03-21T14:22:43.605 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Min latency(s): 0.00125552 2024-03-21T14:22:43.608 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Cleaning up (deleting benchmark objects) 2024-03-21T14:22:43.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.710 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:43.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:43.793 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:43.812 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:43.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:43.967 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:43.997 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:43.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:44.100 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:44.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:44.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:44.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:43 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:22:44.214 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:43 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:22:44.215 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:43 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:22:44.215 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:43 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:22:44.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:44.274 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:44.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:44.376 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:44.415 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:43 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:22:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:43 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:22:44.517 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:44.544 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:44.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:44.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:44.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:44.812 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:44.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:44.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:44.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:44.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:45.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:45.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:45.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:45.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[29576]: pgmap v956: 158 pgs: 158 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 40 KiB/s rd, 82 MiB/s wr, 1.37k op/s 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:22:45.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:44 smithi003 ceph-mon[40510]: pgmap v956: 158 pgs: 158 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 40 KiB/s rd, 82 MiB/s wr, 1.37k op/s 2024-03-21T14:22:45.225 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:45.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:45.327 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:45.354 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:45.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:44 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:22:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:44 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:22:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:44 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:22:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:44 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:22:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:44 smithi118 ceph-mon[33862]: pgmap v956: 158 pgs: 158 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 40 KiB/s rd, 82 MiB/s wr, 1.37k op/s 2024-03-21T14:22:45.456 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:45.487 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:45.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:45.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:45.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:45.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:45.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:45.847 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:45.883 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:45.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:45.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:45.985 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:46.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:46.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:46.164 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:45 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:22:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:22:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[29576]: 4.4 scrub starts 2024-03-21T14:22:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[29576]: 4.4 scrub ok 2024-03-21T14:22:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:22:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:22:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[40510]: 4.4 scrub starts 2024-03-21T14:22:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-mon[40510]: 4.4 scrub ok 2024-03-21T14:22:46.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:46.296 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:46.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:46.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:45 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:22:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:45 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:22:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:45 smithi118 ceph-mon[33862]: 4.4 scrub starts 2024-03-21T14:22:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:45 smithi118 ceph-mon[33862]: 4.4 scrub ok 2024-03-21T14:22:46.434 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:46.536 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:46.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:46.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:46.667 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:46.705 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:46.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:22:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:22:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:22:46.808 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:46.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.836 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:46.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:46.937 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:46.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:46.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:47.071 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:47.101 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:47.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:47.232 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:47.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[29576]: pgmap v957: 158 pgs: 158 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 37 KiB/s rd, 70 MiB/s wr, 1.19k op/s 2024-03-21T14:22:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[29576]: 3.0 scrub starts 2024-03-21T14:22:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[29576]: 3.0 scrub ok 2024-03-21T14:22:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[40510]: pgmap v957: 158 pgs: 158 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 37 KiB/s rd, 70 MiB/s wr, 1.19k op/s 2024-03-21T14:22:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[40510]: 3.0 scrub starts 2024-03-21T14:22:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[40510]: 3.0 scrub ok 2024-03-21T14:22:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:47 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:47.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:47.363 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:47 smithi118 ceph-mon[33862]: pgmap v957: 158 pgs: 158 active+clean; 6.7 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 37 KiB/s rd, 70 MiB/s wr, 1.19k op/s 2024-03-21T14:22:47.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:47 smithi118 ceph-mon[33862]: 3.0 scrub starts 2024-03-21T14:22:47.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:47 smithi118 ceph-mon[33862]: 3.0 scrub ok 2024-03-21T14:22:47.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:47 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:47.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:47 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:47.465 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:47.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:47.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:47.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:47.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:47.783 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:47.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:47.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:22:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:22:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:47.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:48.033 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:48.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:48.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[29576]: osdmap e442: 8 total, 8 up, 8 in 2024-03-21T14:22:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:48.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:48.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[40510]: osdmap e442: 8 total, 8 up, 8 in 2024-03-21T14:22:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:48 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:48.178 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:48.217 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:48.319 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:48.357 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:48 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:48 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:48 smithi118 ceph-mon[33862]: osdmap e442: 8 total, 8 up, 8 in 2024-03-21T14:22:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:48 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:22:48.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:48.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:48.584 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:22:48.603 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:48.631 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:48.631 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:22:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:48.679 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:22:48.723 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:22:48.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:48.762 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:22:48.763 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:48.806 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:22:48.845 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:22:48.865 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:48.891 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:22:48.903 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:22:48.904 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:22:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:48.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:49.006 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:49.036 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:49.038 INFO:teuthology.orchestra.run.smithi003.stderr:set 3.c pg_upmap mapping to [0,6,7] 2024-03-21T14:22:49.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:49.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:49.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:49.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:49.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:22:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:22:49.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:49.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: pgmap v959: 158 pgs: 158 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 48 KiB/s rd, 43 MiB/s wr, 1.10k op/s 2024-03-21T14:22:49.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2746976549' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [0, 6, 7]}]: dispatch 2024-03-21T14:22:49.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:22:49.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:22:49.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:49 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/857858000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:49.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: pgmap v959: 158 pgs: 158 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 48 KiB/s rd, 43 MiB/s wr, 1.10k op/s 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2746976549' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [0, 6, 7]}]: dispatch 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:22:49.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/857858000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:49.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:22:49.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:22:49.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:49.451 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: pgmap v959: 158 pgs: 158 active+clean; 6.6 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 48 KiB/s rd, 43 MiB/s wr, 1.10k op/s 2024-03-21T14:22:49.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2746976549' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [0, 6, 7]}]: dispatch 2024-03-21T14:22:49.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:22:49.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:22:49.452 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:49 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/857858000' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:22:49.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:49.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:49.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:49.619 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:49.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:49.721 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:49.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:49.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:49.907 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:49.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:50.008 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:50.034 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:50.136 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:50.174 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:50.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:50.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:22:50.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2746976549' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [0, 6, 7]}]': finished 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: osdmap e443: 8 total, 8 up, 8 in 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2746976549' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [0, 6, 7]}]': finished 2024-03-21T14:22:50.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: osdmap e443: 8 total, 8 up, 8 in 2024-03-21T14:22:50.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:22:50.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:50 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:22:50.276 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:50.302 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:50.404 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:22:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:22:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:22:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:22:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2746976549' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.c", "id": [0, 6, 7]}]': finished 2024-03-21T14:22:50.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: osdmap e443: 8 total, 8 up, 8 in 2024-03-21T14:22:50.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:22:50.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:50 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:22:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:50.547 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:50.589 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:50.691 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:50.729 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:50.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:50.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.859 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:50.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:50.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:50.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:50.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:51.084 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:22:51.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:51.118 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:51.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:51.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: pgmap v961: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 KiB/s rd, 50 MiB/s wr, 1.48k op/s 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: osdmap e444: 8 total, 8 up, 8 in 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3702476943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:51.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: pgmap v961: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 KiB/s rd, 50 MiB/s wr, 1.48k op/s 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: osdmap e444: 8 total, 8 up, 8 in 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3702476943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:51.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:51 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:51.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:51.362 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:51.393 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:22:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:22:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:22:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:22:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:22:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:22:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: pgmap v961: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 55 KiB/s rd, 50 MiB/s wr, 1.48k op/s 2024-03-21T14:22:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: osdmap e444: 8 total, 8 up, 8 in 2024-03-21T14:22:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:22:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:22:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3702476943' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:51 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:22:51.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:51.539 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:51.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:51.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:51.670 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:51.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.815 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:51.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:51.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:51.944 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:51.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:52.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:52.071 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:52.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:52.173 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:52.202 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:52.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:52 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:22:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:52 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:22:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:52 smithi003 ceph-mon[29576]: osdmap e445: 8 total, 8 up, 8 in 2024-03-21T14:22:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:52 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:22:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:52 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:22:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:52 smithi003 ceph-mon[40510]: osdmap e445: 8 total, 8 up, 8 in 2024-03-21T14:22:52.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:52.340 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:52.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:52 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:22:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:52 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:22:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:52 smithi118 ceph-mon[33862]: osdmap e445: 8 total, 8 up, 8 in 2024-03-21T14:22:52.442 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:52.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:52.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:52.574 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:52.602 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:52.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:52.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.746 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:52.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:52.847 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:52.877 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:52.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:52.979 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:53.010 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:53.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.153 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:53.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:53.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:53.298 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:53.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:53.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:22:53.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:22:53.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:22:53.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:22:53.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: pgmap v964: 158 pgs: 1 peering, 157 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: 4.6 deep-scrub starts 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: 4.6 deep-scrub ok 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[29576]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:22:53.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: pgmap v964: 158 pgs: 1 peering, 157 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-03-21T14:22:53.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: 4.6 deep-scrub starts 2024-03-21T14:22:53.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: 4.6 deep-scrub ok 2024-03-21T14:22:53.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:53 smithi003 ceph-mon[40510]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:22:53.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: pgmap v964: 158 pgs: 1 peering, 157 active+clean; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 16 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: 4.6 deep-scrub starts 2024-03-21T14:22:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: 4.6 deep-scrub ok 2024-03-21T14:22:53.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:53 smithi118 ceph-mon[33862]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-21T14:22:53.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:53.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:53.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:53.567 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:53.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:53.669 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:53.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.722 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:53.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:53.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:53.878 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:53.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:53.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:53.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:54.022 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:54.126 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:54.153 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[29576]: 6.a deep-scrub starts 2024-03-21T14:22:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[29576]: 6.a deep-scrub ok 2024-03-21T14:22:54.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:22:54.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:22:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[40510]: 6.a deep-scrub starts 2024-03-21T14:22:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[40510]: 6.a deep-scrub ok 2024-03-21T14:22:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:22:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:54 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:22:54.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:54.296 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:54.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:54.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:54 smithi118 ceph-mon[33862]: 6.a deep-scrub starts 2024-03-21T14:22:54.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:54 smithi118 ceph-mon[33862]: 6.a deep-scrub ok 2024-03-21T14:22:54.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:54 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:22:54.333 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:54 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:22:54.398 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:54.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:54.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.569 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:54.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:54.670 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:54.704 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:54.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:54.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:54.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:54.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:54.963 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:54.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:54.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:55.065 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:124186 11.b e445/411 11:d6961de4:::benchmark_data_smithi003_188959_object13304:head v 445'7484, mlcod=445'7480)", 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:22:55.094457+0000", 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0079759789999999994, 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0080125779999999994, 2024-03-21T14:22:55.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094457+0000", 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094457+0000", 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.105 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094458+0000", 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2200000000000002e-07 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094467+0000", 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4400000000000005e-06 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094482+0000", 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5356999999999999e-05 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.106 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094491+0000", 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5629999999999992e-06 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094559+0000", 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8269999999999995e-05 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.094582+0000", 2024-03-21T14:22:55.107 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3430000000000001e-05 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:124191 11.3 e445/411 11:cb077e01:::benchmark_data_smithi003_188959_object13309:head v 445'7809, mlcod=445'7807)", 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:22:55.097889+0000", 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0045444259999999998, 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0046487710000000003, 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:22:55.108 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.097889+0000", 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.097889+0000", 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.097890+0000", 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5000000000000001e-07 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.109 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.097899+0000", 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6240000000000008e-06 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.097912+0000", 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3334e-05 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.097920+0000", 2024-03-21T14:22:55.110 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6680000000000001e-06 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098767+0000", 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084780299999999995 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098788+0000", 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0469999999999999e-05 2024-03-21T14:22:55.111 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:124195 11.a e445/411 11:5e61daec:::benchmark_data_smithi003_188959_object13313:head v 445'7788, mlcod=445'7786)", 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:22:55.098257+0000", 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0041767109999999996, 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043454720000000004, 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:22:55.112 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098257+0000", 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098257+0000", 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098258+0000", 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4099999999999997e-07 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:22:55.113 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098265+0000", 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9009999999999994e-06 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098277+0000", 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1623999999999999e-05 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098285+0000", 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5440000000000001e-06 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.114 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098364+0000", 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9814000000000005e-05 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098386+0000", 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1936000000000001e-05 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:124199 11.c 11:36139622:::benchmark_data_smithi003_188959_object13317:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:22:55.098529+0000", 2024-03-21T14:22:55.115 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0039040479999999998, 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041363650000000004, 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 124199 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098529+0000", 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098529+0000", 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098531+0000", 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.711e-06 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.117 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098536+0000", 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1410000000000004e-06 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098548+0000", 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1545999999999999e-05 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098556+0000", 2024-03-21T14:22:55.118 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3550000000000005e-06 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099305+0000", 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00074895199999999999 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099442+0000", 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000137069 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:22:55.119 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099565+0000", 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012286400000000001 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:124198 11.3 e445/411 11:cb53022c:::benchmark_data_smithi003_188959_object13316:head v 445'7810, mlcod=445'7807)", 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:22:55.098776+0000", 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0036578100000000001, 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039740319999999997, 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:22:55.120 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098776+0000", 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098776+0000", 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.121 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098776+0000", 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0399999999999996e-07 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098782+0000", 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3340000000000001e-06 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098789+0000", 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4399999999999999e-06 2024-03-21T14:22:55.122 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.098795+0000", 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9800000000000003e-06 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.101562+0000", 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0027665599999999999 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.101582+0000", 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0190000000000002e-05 2024-03-21T14:22:55.123 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:124201 11.7 11:e92cee19:::benchmark_data_smithi003_188959_object13319:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e445)", 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:22:55.099334+0000", 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.003099746, 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003485981, 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:22:55.124 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 124201 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099334+0000", 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:22:55.125 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099334+0000", 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099335+0000", 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5000000000000001e-07 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099339+0000", 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8119999999999998e-06 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.126 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099349+0000", 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9350000000000006e-06 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099362+0000", 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3046e-05 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099419+0000", 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6499999999999998e-05 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.127 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099528+0000", 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000109413 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:22:55.099637+0000", 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108167 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 6 2024-03-21T14:22:55.128 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:22:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.130 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:55.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:55.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[29576]: pgmap v965: 158 pgs: 1 peering, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.8 MiB/s wr, 1.33k op/s 2024-03-21T14:22:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:22:55.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[40510]: pgmap v965: 158 pgs: 1 peering, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.8 MiB/s wr, 1.33k op/s 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:22:55.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:55 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:22:55.268 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:55.370 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.402 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:55.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:55 smithi118 ceph-mon[33862]: pgmap v965: 158 pgs: 1 peering, 157 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.8 MiB/s wr, 1.33k op/s 2024-03-21T14:22:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:55 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:22:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:55 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:22:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:55 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:22:55.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:55 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:22:55.503 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:55.546 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:55.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:55.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:55.696 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:55.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:55.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.835 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:55.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:55.850 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:22:55.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-21T14:22:55.936 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:55.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:55.967 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:55.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:56.068 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:56.096 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:56.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:56.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:56.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:56.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:56.350 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:56.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:56.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:22:56.421 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:22:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:22:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:22:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:22:56.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:22:56.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 1.1 scrub starts 2024-03-21T14:22:56.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 1.1 scrub ok 2024-03-21T14:22:56.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:56 smithi118 ceph-mon[33862]: 11.4 scrub starts 2024-03-21T14:22:56.433 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:56.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:56.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:22:56.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:22:56.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:22:56.480 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:22:56.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 1.1 scrub starts 2024-03-21T14:22:56.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 1.1 scrub ok 2024-03-21T14:22:56.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[29576]: 11.4 scrub starts 2024-03-21T14:22:56.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:22:56.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:22:56.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:22:56.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:22:56.482 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 1.1 scrub starts 2024-03-21T14:22:56.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 1.1 scrub ok 2024-03-21T14:22:56.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-mon[40510]: 11.4 scrub starts 2024-03-21T14:22:56.538 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:56.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:56.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:56.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:56.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:56.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:56.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:22:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:22:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:22:56.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:56.853 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:56.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:56.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:56.955 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:57.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:57.103 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:57.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.132 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:57.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:57.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:57.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:57.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:57.411 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:57.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:57 smithi118 ceph-mon[33862]: 5.3 scrub starts 2024-03-21T14:22:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:57 smithi118 ceph-mon[33862]: 5.3 scrub ok 2024-03-21T14:22:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:57 smithi118 ceph-mon[33862]: pgmap v966: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 867 op/s 2024-03-21T14:22:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:57 smithi118 ceph-mon[33862]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:22:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:57 smithi118 ceph-mon[33862]: 11.4 scrub ok 2024-03-21T14:22:57.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[29576]: 5.3 scrub starts 2024-03-21T14:22:57.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[29576]: 5.3 scrub ok 2024-03-21T14:22:57.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[29576]: pgmap v966: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 867 op/s 2024-03-21T14:22:57.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[29576]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:22:57.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[29576]: 11.4 scrub ok 2024-03-21T14:22:57.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[40510]: 5.3 scrub starts 2024-03-21T14:22:57.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[40510]: 5.3 scrub ok 2024-03-21T14:22:57.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[40510]: pgmap v966: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 774 B/s rd, 0 B/s wr, 867 op/s 2024-03-21T14:22:57.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[40510]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-21T14:22:57.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:57 smithi003 ceph-mon[40510]: 11.4 scrub ok 2024-03-21T14:22:57.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:57.543 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:57.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:57.685 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:57.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:57.787 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:57.826 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:57.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:57.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:57.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:22:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:22:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:57.969 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:57.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:58.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:22:58.102 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:58.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:22:58.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:58.210 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:22:58.210 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:22:58.210 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-21T14:22:58.210 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-21T14:22:58.210 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-21T14:22:58.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-21T14:22:58.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:22:58.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:22:58.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:22:58.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:22:58.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 1.8 scrub starts 2024-03-21T14:22:58.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 1.8 scrub ok 2024-03-21T14:22:58.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[29576]: 11.3 scrub starts 2024-03-21T14:22:58.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:22:58.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:22:58.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:22:58.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:22:58.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 1.8 scrub starts 2024-03-21T14:22:58.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 1.8 scrub ok 2024-03-21T14:22:58.233 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:58 smithi003 ceph-mon[40510]: 11.3 scrub starts 2024-03-21T14:22:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:58.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:58.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:58.393 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:22:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:22:58.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:22:58.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:22:58.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 1.8 scrub starts 2024-03-21T14:22:58.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 1.8 scrub ok 2024-03-21T14:22:58.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:58 smithi118 ceph-mon[33862]: 11.3 scrub starts 2024-03-21T14:22:58.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:58.636 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:58.660 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:22:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:22:58.781 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:58.815 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:58.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:58.917 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:22:58.953 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:58.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:22:59.054 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:59.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:59.200 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:22:59.230 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:22:59.331 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: 4.1b deep-scrub starts 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: 4.1b deep-scrub ok 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: pgmap v967: 158 pgs: 158 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: 4.1b deep-scrub starts 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: 4.1b deep-scrub ok 2024-03-21T14:22:59.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:59.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: pgmap v967: 158 pgs: 158 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:22:59.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:22:59.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:22:59.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:22:59.350 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:22:59 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:22:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.423 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:59.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: 4.1b deep-scrub starts 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: 4.1b deep-scrub ok 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: pgmap v967: 158 pgs: 158 active+clean; 5.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:22:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:22:59 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:22:59.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:22:59.562 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:22:59.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:22:59.691 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:22:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:22:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:22:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:22:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:22:59.713 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:22:59.714 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:22:59.715 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:22:59.715 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:22:59.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:22:59.841 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:59.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:22:59.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:22:59.981 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:22:59.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:22:59.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:00.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:00.112 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:00.214 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:00.343 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:00.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:00.383 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:00 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:23:00.383 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:00 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:23:00.383 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:00 smithi118 ceph-mon[33862]: 11.3 scrub ok 2024-03-21T14:23:00.383 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/126174765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:00.431 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:23:00.431 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:23:00.431 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[29576]: 11.3 scrub ok 2024-03-21T14:23:00.431 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/126174765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:00.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:23:00.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:23:00.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[40510]: 11.3 scrub ok 2024-03-21T14:23:00.432 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/126174765' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:00.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:00.510 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:00.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:00.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:00.651 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:00.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:00.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:00.793 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:00.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:00.895 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:00.935 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:00.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:00.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:00.990 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:01.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:01.066 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:23:01.067 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:23:01.097 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:01.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:01 smithi003 ceph-mon[29576]: pgmap v968: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:01 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:23:01.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:01 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:23:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:01 smithi003 ceph-mon[40510]: pgmap v968: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:01 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:23:01.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:01 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:23:01.329 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:01.359 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:01.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:01 smithi118 ceph-mon[33862]: pgmap v968: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:01 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:23:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:01 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:23:01.460 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:01.591 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:01.629 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:01.731 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:01.758 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:01.860 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:01.904 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:01.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:02.005 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:02.036 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:02.138 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:02.178 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:02.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:02.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.321 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:02.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:02.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-21T14:23:02.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:02.449 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:02.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:02.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:23:02.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:02.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:02.677 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:02.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.708 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:02.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:02.809 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:02.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:02.995 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:03.046 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:03.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:03.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:03.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:03.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:03.284 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:03.286 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:23:03.286 INFO:teuthology.orchestra.run.smithi118.stdout: "success": "" 2024-03-21T14:23:03.286 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:23:03.286 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:03 smithi118 ceph-mon[33862]: pgmap v969: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-21T14:23:03.286 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:03 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:03.286 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:03.319 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:03.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:03.421 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:03.468 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:03.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:03 smithi003 ceph-mon[29576]: pgmap v969: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-21T14:23:03.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:03 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:03.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:03 smithi003 ceph-mon[40510]: pgmap v969: 158 pgs: 158 active+clean; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 967 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-21T14:23:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:03 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:03.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:03.599 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:03.700 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:03.732 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:03.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:03.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:03.881 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:03.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:03.983 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:04.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:04.120 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:04.147 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:04.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:04.249 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:04.278 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:04.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:04.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:04.410 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:04.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:04 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:23:04.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:04 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:23:04.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:04 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:23:04.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:04 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:23:04.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:04 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:23:04.420 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:04 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:23:04.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:23:04.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:23:04.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:23:04.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:23:04.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:23:04.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:23:04.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:23:04.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:23:04.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:23:04.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:23:04.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:23:04.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:04 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:23:04.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:04.535 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:04.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:04.637 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:04.681 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:04.783 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:04.812 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:04.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:04.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:04.944 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:04.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:04.979 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:05.027 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:05.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:05.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:05.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:05.234 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: pgmap v970: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-21T14:23:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:23:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:23:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:23:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:23:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 1.9 scrub starts 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 1.9 scrub ok 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: pgmap v970: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 1.9 scrub starts 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 1.9 scrub ok 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:23:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:05 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:23:05.336 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:05.367 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: pgmap v970: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 1.9 scrub starts 2024-03-21T14:23:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 1.9 scrub ok 2024-03-21T14:23:05.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:23:05.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:05 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:23:05.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:05.512 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:05.614 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:05.640 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:05.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:05.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:05.774 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:05.876 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:05.905 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:05.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:06.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:06.038 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:06.140 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:06.185 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:06.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:06.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:06.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:06.405 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:23:06.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:23:06.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-mon[29576]: pgmap v971: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 941 op/s 2024-03-21T14:23:06.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:23:06.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:23:06.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-mon[40510]: pgmap v971: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 941 op/s 2024-03-21T14:23:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:06 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:23:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:06 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:23:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:06 smithi118 ceph-mon[33862]: pgmap v971: 158 pgs: 158 active+clean; 5.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 941 op/s 2024-03-21T14:23:06.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:06.481 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:06.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:06.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.615 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:06.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:06.654 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:23:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:23:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:23:06.715 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:06.742 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:06.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:06.886 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:06.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:06.988 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:07.015 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:07.117 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:07.143 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:07.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:07 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:23:07.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:07 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:23:07.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:07 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:23:07.239 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:07 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:23:07.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:07.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:07.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:07.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:07.413 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:07 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:23:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:07 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:23:07.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:07.542 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:07.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:07.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.671 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:07.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:07.772 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:07.811 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:07.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:07.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:23:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:23:07.955 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:07.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:07.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:08.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:08.093 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:08.182 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:23:08.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:08.233 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:08.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:08.379 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1230629245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:23:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:23:08.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:08.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:23:08.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:08 smithi118 ceph-mon[33862]: pgmap v972: 158 pgs: 158 active+clean; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T14:23:08.481 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:23:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:23:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:23:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1230629245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:23:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:23:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[29576]: pgmap v972: 158 pgs: 158 active+clean; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1230629245' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:23:08.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:08 smithi003 ceph-mon[40510]: pgmap v972: 158 pgs: 158 active+clean; 5.2 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-21T14:23:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:08.610 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:08.635 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:08.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:08.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:08.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:08.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:08.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:08.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:08.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:08.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:09.019 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:09.121 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:09.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:09.261 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:09.293 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:09.395 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:09 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:09 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:23:09.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:09 smithi118 ceph-mon[33862]: osdmap e446: 8 total, 8 up, 8 in 2024-03-21T14:23:09.438 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:09.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:09.457 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:23:09.457 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:23:09.458 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-21T14:23:09.458 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-21T14:23:09.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:23:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:09 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:09 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:23:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:09 smithi003 ceph-mon[29576]: osdmap e446: 8 total, 8 up, 8 in 2024-03-21T14:23:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:09 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:09 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:23:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:09 smithi003 ceph-mon[40510]: osdmap e446: 8 total, 8 up, 8 in 2024-03-21T14:23:09.540 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:09.568 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:09.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:09.733 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:09.836 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:09.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:09.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:09.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:09.965 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:10.002 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:10.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:10.104 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:10.131 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:10.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:10.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:10.379 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:10.516 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:10.560 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:10.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:10.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:10.690 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:10.792 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.832 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:10.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:10.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:10.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:10.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:11.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:11.075 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:10 smithi003 ceph-mon[29576]: pgmap v974: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:11.075 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:10 smithi003 ceph-mon[40510]: pgmap v974: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:11.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:11.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:10 smithi118 ceph-mon[33862]: pgmap v974: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:11.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:11.306 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:11.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:11.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:11.445 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:11.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:11.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:11.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:11.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:11.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:11.702 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:11.740 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:11.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:11.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:11.842 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:11.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:11.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:11.969 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:12.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.001 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:12.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:12.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:12.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.179 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:12.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:12.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:12.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:12.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:12.488 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:12.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:12.590 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:12.614 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:23:12.614 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:23:12.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.648 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:12.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:12.749 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:12.774 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:12.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:12.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:12.915 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:12.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:12.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:12.916 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:12 smithi003 ceph-mon[29576]: pgmap v975: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:12.916 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:12 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:23:12.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:12 smithi003 ceph-mon[40510]: pgmap v975: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:12.917 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:12 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:23:12.918 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:23:12.918 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:23:12.966 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:23:13.003 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:13.013 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:23:13.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:13.042 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:13.053 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:23:13.092 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:23:13.145 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:13.146 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:23:13.173 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:12 smithi118 ceph-mon[33862]: pgmap v975: 158 pgs: 158 active+clean; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:13.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:12 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:23:13.184 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:23:13.192 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:23:13.275 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:13.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:13.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:13.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:13.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:13.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:13.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:23:13.601 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:13.703 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:13.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:13.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:13.832 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:13.875 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:13.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:13.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:13.999 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2106970487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:23:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:23:14.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:14.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2106970487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:14.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:23:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:23:14.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:13 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:14.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:14.128 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2106970487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:13 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:23:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:13 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:23:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:13 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:14.230 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:14.260 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:14.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:14.389 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:14.491 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:14.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:14.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:14.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:14.660 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:14.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:14.791 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:14.892 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:14.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:14.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:14.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:15.024 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:15.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:15.125 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:14 smithi003 ceph-mon[29576]: pgmap v976: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:15.125 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:14 smithi003 ceph-mon[40510]: pgmap v976: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:15.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:15.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-21T14:23:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:14 smithi118 ceph-mon[33862]: pgmap v976: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:15.280 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:15.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:15.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:15.431 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:15.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:15.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:15.588 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:15.689 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:15.717 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:15.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:15.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:15.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:15.991 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:15.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:16.092 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:16.117 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:16.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:16.219 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:16.250 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:16.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:16.378 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:16.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:16.529 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:16.606 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:23:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:23:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:23:16.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:16.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:16.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:16.761 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:16.788 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:16.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:16 smithi003 ceph-mon[29576]: pgmap v977: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:16.871 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:16 smithi003 ceph-mon[40510]: pgmap v977: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:16.890 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:16.917 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:16.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:17.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:17.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:17.090 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:17.095 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:17.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:16 smithi118 ceph-mon[33862]: pgmap v977: 158 pgs: 158 active+clean; 4.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:17.201 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:17.305 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:17.362 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:17.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:17.502 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:17.604 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:17.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.643 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:17.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:17.744 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:17.772 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:17.874 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:17.905 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:17.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:17.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:23:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:23:18.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:18.033 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:18.135 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:18.172 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:18.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:18.274 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:18.314 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:18.315 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:23:18.416 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:18.442 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:18.544 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:18.568 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:18.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:18.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[40510]: pgmap v978: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 931 op/s 2024-03-21T14:23:18.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:18.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:18.660 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:18.660 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[29576]: pgmap v978: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 931 op/s 2024-03-21T14:23:18.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:18.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:18.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:18.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:18.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:18.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:18.799 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:18.836 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:18.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:18 smithi118 ceph-mon[33862]: pgmap v978: 158 pgs: 158 active+clean; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 931 op/s 2024-03-21T14:23:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:18 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:23:18.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:18.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:18.966 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:18.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:19.067 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:19.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:19.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:19.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.260 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:19.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:19.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:19.393 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:19.495 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.523 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:19.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:19.623 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:19.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:19.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:19.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:19.894 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:19.923 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:19.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:20.023 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:20.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:20.152 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:20.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:20.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:20.284 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:20.415 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:20.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:20.560 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.626 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:20.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:20.727 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:20.748 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:23:20.753 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:20.758 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:23:20.855 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:20.858 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[29576]: pgmap v979: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[29576]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[40510]: pgmap v979: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:20.859 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:20 smithi003 ceph-mon[40510]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:23:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:20.880 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:20.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:20.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:20 smithi118 ceph-mon[33862]: pgmap v979: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 916 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-21T14:23:20.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:20 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:20.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:20 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:20.934 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:20 smithi118 ceph-mon[33862]: from='client.25898 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:23:20.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:21.026 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:21.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:21.156 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:21.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:21.287 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:21.389 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:21.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:21.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:21.569 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:21.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:21.671 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:21.697 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:21.799 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.827 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:21.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:21.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:21.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:21 smithi118 ceph-mon[33862]: osdmap e447: 8 total, 8 up, 8 in 2024-03-21T14:23:21.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:21.959 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:21.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:22.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:21 smithi003 ceph-mon[29576]: osdmap e447: 8 total, 8 up, 8 in 2024-03-21T14:23:22.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:21 smithi003 ceph-mon[40510]: osdmap e447: 8 total, 8 up, 8 in 2024-03-21T14:23:22.060 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:22.083 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:22.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:22.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:22.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:22.349 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:22.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:22.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:22.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.480 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:22.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:22.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:22.642 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:22.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:22.694 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:22 smithi003 ceph-mon[29576]: pgmap v981: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:22.694 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:22 smithi003 ceph-mon[40510]: pgmap v981: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:22.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:22.779 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:22.881 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:22.923 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:22.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:22.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:22 smithi118 ceph-mon[33862]: pgmap v981: 158 pgs: 158 active+clean; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:23.025 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:23.056 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:23.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:23.157 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.204 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:23.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:23.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:23.346 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:23.448 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:23.478 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:23.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:23.597 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:23:23.597 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:23:23.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:156325 11.2 e447/411 11:4ed811c8:::benchmark_data_smithi003_188959_object45443:head v 447'9624, mlcod=447'9620)", 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:23.583261+0000", 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010828747, 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010860104000000001, 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583261+0000", 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583261+0000", 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583262+0000", 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.7700000000000004e-07 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583269+0000", 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1069999999999999e-06 2024-03-21T14:23:23.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583279+0000", 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0294e-05 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583286+0000", 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9600000000000003e-06 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.584130+0000", 2024-03-21T14:23:23.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00084407999999999998 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.584151+0000", 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0330999999999999e-05 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:156326 11.9 11:9af44877:::benchmark_data_smithi003_188959_object45444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:23.583314+0000", 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010775420000000001, 2024-03-21T14:23:23.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010880643000000001, 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 156326 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583314+0000", 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583314+0000", 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583315+0000", 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0920000000000001e-06 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583320+0000", 2024-03-21T14:23:23.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.036e-06 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583328+0000", 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3899999999999993e-06 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583334+0000", 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4479999999999999e-06 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583393+0000", 2024-03-21T14:23:23.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8841000000000001e-05 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583527+0000", 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013336899999999999 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583651+0000", 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124466 2024-03-21T14:23:23.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.591785+0000", 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0081340360000000007 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.591859+0000", 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4041000000000005e-05 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.607 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:156328 11.1 11:8b51605f:::benchmark_data_smithi003_188959_object45446:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-03-21T14:23:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:23.583378+0000", 2024-03-21T14:23:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.010711428, 2024-03-21T14:23:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010906297000000001, 2024-03-21T14:23:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:23:23.609 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 156328 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583378+0000", 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583378+0000", 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583379+0000", 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.9900000000000001e-07 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583381+0000", 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2460000000000002e-06 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583386+0000", 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2850000000000002e-06 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583391+0000", 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.797e-06 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583672+0000", 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028047 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583779+0000", 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107487 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.583983+0000", 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000203914 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.591867+0000", 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0078843460000000004 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:23.613 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:156331 11.8 11:1e443513:::benchmark_data_smithi003_188959_object45449:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:23.586156+0000", 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.0079336019999999997, 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082098929999999994, 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 156331 2024-03-21T14:23:23.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586156+0000", 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586156+0000", 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586157+0000", 2024-03-21T14:23:23.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.114e-06 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586162+0000", 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0849999999999996e-06 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586173+0000", 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0828e-05 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:23.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586182+0000", 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3429999999999996e-06 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586475+0000", 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029343199999999999 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586643+0000", 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016792099999999999 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:23:23.617 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.586828+0000", 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018520799999999999 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.592065+0000", 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052370630000000001 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.25811.0:156339 11.0 e447/411 11:06d945b7:::benchmark_data_smithi003_188959_object45457:head v 447'9682, mlcod=447'9680)", 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:23.593898+0000", 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.000191464, 2024-03-21T14:23:23.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00054606000000000001, 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.593898+0000", 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.593898+0000", 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.593899+0000", 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5899999999999995e-07 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.593907+0000", 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4109999999999999e-06 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.593920+0000", 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3699e-05 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.593933+0000", 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2325e-05 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.594031+0000", 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8505000000000006e-05 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:23.594051+0000", 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0239999999999999e-05 2024-03-21T14:23:23.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:23.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:23.622 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:23:23.622 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 5 2024-03-21T14:23:23.622 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:23:23.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:23.647 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:23.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:23.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:23.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:23.894 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:23.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:24.052 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:24.079 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:24.181 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:24.215 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:24.317 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:24.344 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:24.447 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:24.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:24.573 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:24.602 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:24.614 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:23:24.704 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:24.730 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:24.832 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:24.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.863 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:24 smithi118 ceph-mon[33862]: pgmap v982: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:24 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2345562839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:24.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:24 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:24.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:24.995 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:24.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:24 smithi003 ceph-mon[29576]: pgmap v982: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:24 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2345562839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:25.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:24 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:24 smithi003 ceph-mon[40510]: pgmap v982: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:24 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2345562839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:25.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:24 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:25.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:25.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:25.242 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:25.270 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:25.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:25.372 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:25.398 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:25.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:25.544 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:25.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-21T14:23:25.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:25.673 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:25.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:25.775 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:25.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:25.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:25 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:25 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:23:25.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:25 smithi118 ceph-mon[33862]: osdmap e448: 8 total, 8 up, 8 in 2024-03-21T14:23:25.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:25.975 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:25.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:25 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:25 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:23:26.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:25 smithi003 ceph-mon[29576]: osdmap e448: 8 total, 8 up, 8 in 2024-03-21T14:23:26.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:25 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:26.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:25 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:23:26.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:25 smithi003 ceph-mon[40510]: osdmap e448: 8 total, 8 up, 8 in 2024-03-21T14:23:26.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:26.108 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:26.210 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:26.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:26.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:26.345 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.373 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:26.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:26.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:26.488 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":979,"stamp":"2024-03-21T14:23:19.349616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4731181504,"num_objects":72382,"num_object_clones":0,"num_object_copies":217146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72382,"num_whiteouts":0,"num_read":2738,"num_read_kb":2490,"num_write":150584,"num_write_kb":7096302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3952,"num_bytes_recovered":246878200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73330,"ondisk_log_size":73330,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20000232,"kb_used_data":14363368,"kb_used_omap":21,"kb_used_meta":5636394,"kb_avail":729960984,"statfs":{"total":767960285184,"available":747480047616,"internally_reserved":0,"allocated":14708088832,"data_stored":14704001568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22067,"internal_metadata":5771667917},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,5,2,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-799932416,"num_objects":-12206,"num_object_clones":0,"num_object_copies":-36618,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12206,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12206,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.177253"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":823,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170327+0000","last_change":"2024-03-21T14:23:04.359116+0000","last_active":"2024-03-21T14:23:08.170327+0000","last_peered":"2024-03-21T14:23:08.170327+0000","last_clean":"2024-03-21T14:23:08.170327+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:23:08.170327+0000","last_undegraded":"2024-03-21T14:23:08.170327+0000","last_fullsized":"2024-03-21T14:23:08.170327+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:04.358989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.358989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:15.061861+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":844,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189638+0000","last_change":"2024-03-21T14:22:40.829274+0000","last_active":"2024-03-21T14:23:08.189638+0000","last_peered":"2024-03-21T14:23:08.189638+0000","last_clean":"2024-03-21T14:23:08.189638+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:23:08.189638+0000","last_undegraded":"2024-03-21T14:23:08.189638+0000","last_fullsized":"2024-03-21T14:23:08.189638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:40.829154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.829154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:43.086824+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":399,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178316+0000","last_change":"2024-03-21T14:23:06.316701+0000","last_active":"2024-03-21T14:23:08.178316+0000","last_peered":"2024-03-21T14:23:08.178316+0000","last_clean":"2024-03-21T14:23:08.178316+0000","last_became_active":"2024-03-21T14:19:34.695760+0000","last_became_peered":"2024-03-21T14:19:34.695760+0000","last_unstale":"2024-03-21T14:23:08.178316+0000","last_undegraded":"2024-03-21T14:23:08.178316+0000","last_fullsized":"2024-03-21T14:23:08.178316+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:06.316132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:06.316132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:29.219900+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":890,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.169995+0000","last_change":"2024-03-21T14:20:41.426448+0000","last_active":"2024-03-21T14:23:08.169995+0000","last_peered":"2024-03-21T14:23:08.169995+0000","last_clean":"2024-03-21T14:23:08.169995+0000","last_became_active":"2024-03-21T14:20:21.782187+0000","last_became_peered":"2024-03-21T14:20:21.782187+0000","last_unstale":"2024-03-21T14:23:08.169995+0000","last_undegraded":"2024-03-21T14:23:08.169995+0000","last_fullsized":"2024-03-21T14:23:08.169995+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:20:41.426344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:41.426344+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":128,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.195027+0000","last_change":"2024-03-21T14:23:07.529805+0000","last_active":"2024-03-21T14:23:08.195027+0000","last_peered":"2024-03-21T14:23:08.195027+0000","last_clean":"2024-03-21T14:23:08.195027+0000","last_became_active":"2024-03-21T14:20:21.782401+0000","last_became_peered":"2024-03-21T14:20:21.782401+0000","last_unstale":"2024-03-21T14:23:08.195027+0000","last_undegraded":"2024-03-21T14:23:08.195027+0000","last_fullsized":"2024-03-21T14:23:08.195027+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:07.529663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:07.529663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:22.958204+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":834,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.169878+0000","last_change":"2024-03-21T14:20:52.477097+0000","last_active":"2024-03-21T14:23:08.169878+0000","last_peered":"2024-03-21T14:23:08.169878+0000","last_clean":"2024-03-21T14:23:08.169878+0000","last_became_active":"2024-03-21T14:19:14.689459+0000","last_became_peered":"2024-03-21T14:19:14.689459+0000","last_unstale":"2024-03-21T14:23:08.169878+0000","last_undegraded":"2024-03-21T14:23:08.169878+0000","last_fullsized":"2024-03-21T14:23:08.169878+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:52.477009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.477009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":845,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187483+0000","last_change":"2024-03-21T14:20:37.939217+0000","last_active":"2024-03-21T14:23:08.187483+0000","last_peered":"2024-03-21T14:23:08.187483+0000","last_clean":"2024-03-21T14:23:08.187483+0000","last_became_active":"2024-03-21T14:19:14.695703+0000","last_became_peered":"2024-03-21T14:19:14.695703+0000","last_unstale":"2024-03-21T14:23:08.187483+0000","last_undegraded":"2024-03-21T14:23:08.187483+0000","last_fullsized":"2024-03-21T14:23:08.187483+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:37.939080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.939080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'46","reported_seq":901,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.185581+0000","last_change":"2024-03-21T14:20:40.919011+0000","last_active":"2024-03-21T14:23:08.185581+0000","last_peered":"2024-03-21T14:23:08.185581+0000","last_clean":"2024-03-21T14:23:08.185581+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:23:08.185581+0000","last_undegraded":"2024-03-21T14:23:08.185581+0000","last_fullsized":"2024-03-21T14:23:08.185581+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:20:40.918895+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:20:40.918895+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":704,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187753+0000","last_change":"2024-03-21T14:22:57.984639+0000","last_active":"2024-03-21T14:23:08.187753+0000","last_peered":"2024-03-21T14:23:08.187753+0000","last_clean":"2024-03-21T14:23:08.187753+0000","last_became_active":"2024-03-21T14:19:34.690300+0000","last_became_peered":"2024-03-21T14:19:34.690300+0000","last_unstale":"2024-03-21T14:23:08.187753+0000","last_undegraded":"2024-03-21T14:23:08.187753+0000","last_fullsized":"2024-03-21T14:23:08.187753+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:57.984394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.984394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.446523+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":851,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178980+0000","last_change":"2024-03-21T14:22:48.510243+0000","last_active":"2024-03-21T14:23:08.178980+0000","last_peered":"2024-03-21T14:23:08.178980+0000","last_clean":"2024-03-21T14:23:08.178980+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:23:08.178980+0000","last_undegraded":"2024-03-21T14:23:08.178980+0000","last_fullsized":"2024-03-21T14:23:08.178980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:48.510139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.510139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:13.480887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":643,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171157+0000","last_change":"2024-03-21T14:22:54.329383+0000","last_active":"2024-03-21T14:23:08.171157+0000","last_peered":"2024-03-21T14:23:08.171157+0000","last_clean":"2024-03-21T14:23:08.171157+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:23:08.171157+0000","last_undegraded":"2024-03-21T14:23:08.171157+0000","last_fullsized":"2024-03-21T14:23:08.171157+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.329270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.329270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:10.420864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"441'21","reported_seq":857,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170422+0000","last_change":"2024-03-21T14:22:48.380073+0000","last_active":"2024-03-21T14:23:08.170422+0000","last_peered":"2024-03-21T14:23:08.170422+0000","last_clean":"2024-03-21T14:23:08.170422+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:23:08.170422+0000","last_undegraded":"2024-03-21T14:23:08.170422+0000","last_fullsized":"2024-03-21T14:23:08.170422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:22:48.379990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.379990+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:05.754736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":793,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189825+0000","last_change":"2024-03-21T14:22:57.054038+0000","last_active":"2024-03-21T14:23:08.189825+0000","last_peered":"2024-03-21T14:23:08.189825+0000","last_clean":"2024-03-21T14:23:08.189825+0000","last_became_active":"2024-03-21T14:20:21.776360+0000","last_became_peered":"2024-03-21T14:20:21.776360+0000","last_unstale":"2024-03-21T14:23:08.189825+0000","last_undegraded":"2024-03-21T14:23:08.189825+0000","last_fullsized":"2024-03-21T14:23:08.189825+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:57.053921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.053921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.452840+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":844,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.174794+0000","last_change":"2024-03-21T14:21:47.782509+0000","last_active":"2024-03-21T14:23:08.174794+0000","last_peered":"2024-03-21T14:23:08.174794+0000","last_clean":"2024-03-21T14:23:08.174794+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:23:08.174794+0000","last_undegraded":"2024-03-21T14:23:08.174794+0000","last_fullsized":"2024-03-21T14:23:08.174794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:47.782401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.782401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:13.997696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":309,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170649+0000","last_change":"2024-03-21T14:21:53.335510+0000","last_active":"2024-03-21T14:23:08.170649+0000","last_peered":"2024-03-21T14:23:08.170649+0000","last_clean":"2024-03-21T14:23:08.170649+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:23:08.170649+0000","last_undegraded":"2024-03-21T14:23:08.170649+0000","last_fullsized":"2024-03-21T14:23:08.170649+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_clean_scrub_stamp":"2024-03-21T14:21:53.335395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.820889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"445'10","reported_seq":848,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170764+0000","last_change":"2024-03-21T14:22:57.290034+0000","last_active":"2024-03-21T14:23:08.170764+0000","last_peered":"2024-03-21T14:23:08.170764+0000","last_clean":"2024-03-21T14:23:08.170764+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:23:08.170764+0000","last_undegraded":"2024-03-21T14:23:08.170764+0000","last_fullsized":"2024-03-21T14:23:08.170764+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'10","last_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_deep_scrub":"445'10","last_deep_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.289894+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.557256+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":823,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171379+0000","last_change":"2024-03-21T14:22:32.629084+0000","last_active":"2024-03-21T14:23:08.171379+0000","last_peered":"2024-03-21T14:23:08.171379+0000","last_clean":"2024-03-21T14:23:08.171379+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:23:08.171379+0000","last_undegraded":"2024-03-21T14:23:08.171379+0000","last_fullsized":"2024-03-21T14:23:08.171379+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:32.628950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:32.628950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:35.170470+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":836,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191882+0000","last_change":"2024-03-21T14:22:35.833709+0000","last_active":"2024-03-21T14:23:08.191882+0000","last_peered":"2024-03-21T14:23:08.191882+0000","last_clean":"2024-03-21T14:23:08.191882+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:23:08.191882+0000","last_undegraded":"2024-03-21T14:23:08.191882+0000","last_fullsized":"2024-03-21T14:23:08.191882+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:35.833457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.833457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:03.246243+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":686,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170756+0000","last_change":"2024-03-21T14:19:34.686397+0000","last_active":"2024-03-21T14:23:08.170756+0000","last_peered":"2024-03-21T14:23:08.170756+0000","last_clean":"2024-03-21T14:23:08.170756+0000","last_became_active":"2024-03-21T14:19:34.686106+0000","last_became_peered":"2024-03-21T14:19:34.686106+0000","last_unstale":"2024-03-21T14:23:08.170756+0000","last_undegraded":"2024-03-21T14:23:08.170756+0000","last_fullsized":"2024-03-21T14:23:08.170756+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:33.422232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:18:33.422232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":767,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.179617+0000","last_change":"2024-03-21T14:21:50.323084+0000","last_active":"2024-03-21T14:23:08.179617+0000","last_peered":"2024-03-21T14:23:08.179617+0000","last_clean":"2024-03-21T14:23:08.179617+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:23:08.179617+0000","last_undegraded":"2024-03-21T14:23:08.179617+0000","last_fullsized":"2024-03-21T14:23:08.179617+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'34","last_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_deep_scrub":"432'34","last_deep_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.322580+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:12.777689+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":839,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.186568+0000","last_change":"2024-03-21T14:21:33.288900+0000","last_active":"2024-03-21T14:23:08.186568+0000","last_peered":"2024-03-21T14:23:08.186568+0000","last_clean":"2024-03-21T14:23:08.186568+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:23:08.186568+0000","last_undegraded":"2024-03-21T14:23:08.186568+0000","last_fullsized":"2024-03-21T14:23:08.186568+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:33.288792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:33.288792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":828,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171148+0000","last_change":"2024-03-21T14:22:29.551806+0000","last_active":"2024-03-21T14:23:08.171148+0000","last_peered":"2024-03-21T14:23:08.171148+0000","last_clean":"2024-03-21T14:23:08.171148+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:23:08.171148+0000","last_undegraded":"2024-03-21T14:23:08.171148+0000","last_fullsized":"2024-03-21T14:23:08.171148+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:29.551666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.551666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:59.532498+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":828,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170443+0000","last_change":"2024-03-21T14:20:31.407178+0000","last_active":"2024-03-21T14:23:08.170443+0000","last_peered":"2024-03-21T14:23:08.170443+0000","last_clean":"2024-03-21T14:23:08.170443+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:23:08.170443+0000","last_undegraded":"2024-03-21T14:23:08.170443+0000","last_fullsized":"2024-03-21T14:23:08.170443+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.407065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.407065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":881,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.188263+0000","last_change":"2024-03-21T14:22:35.357291+0000","last_active":"2024-03-21T14:23:08.188263+0000","last_peered":"2024-03-21T14:23:08.188263+0000","last_clean":"2024-03-21T14:23:08.188263+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:23:08.188263+0000","last_undegraded":"2024-03-21T14:23:08.188263+0000","last_fullsized":"2024-03-21T14:23:08.188263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:22:35.357134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.357134+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:38.436579+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":343,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.179260+0000","last_change":"2024-03-21T14:21:19.290180+0000","last_active":"2024-03-21T14:23:08.179260+0000","last_peered":"2024-03-21T14:23:08.179260+0000","last_clean":"2024-03-21T14:23:08.179260+0000","last_became_active":"2024-03-21T14:19:14.686523+0000","last_became_peered":"2024-03-21T14:19:14.686523+0000","last_unstale":"2024-03-21T14:23:08.179260+0000","last_undegraded":"2024-03-21T14:23:08.179260+0000","last_fullsized":"2024-03-21T14:23:08.179260+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:21:19.289804+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.289804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"425'15","reported_seq":97,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.737347+0000","last_change":"2024-03-21T14:22:44.412788+0000","last_active":"2024-03-21T14:23:08.737347+0000","last_peered":"2024-03-21T14:23:08.737347+0000","last_clean":"2024-03-21T14:23:08.737347+0000","last_became_active":"2024-03-21T14:21:26.767116+0000","last_became_peered":"2024-03-21T14:21:26.767116+0000","last_unstale":"2024-03-21T14:23:08.737347+0000","last_undegraded":"2024-03-21T14:23:08.737347+0000","last_fullsized":"2024-03-21T14:23:08.737347+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'15","last_scrub_stamp":"2024-03-21T14:22:44.412659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:44.412659+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:54.694196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":853,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189445+0000","last_change":"2024-03-21T14:20:37.151817+0000","last_active":"2024-03-21T14:23:08.189445+0000","last_peered":"2024-03-21T14:23:08.189445+0000","last_clean":"2024-03-21T14:23:08.189445+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:23:08.189445+0000","last_undegraded":"2024-03-21T14:23:08.189445+0000","last_fullsized":"2024-03-21T14:23:08.189445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:20:37.151721+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.151721+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":573,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175387+0000","last_change":"2024-03-21T14:21:19.102404+0000","last_active":"2024-03-21T14:23:08.175387+0000","last_peered":"2024-03-21T14:23:08.175387+0000","last_clean":"2024-03-21T14:23:08.175387+0000","last_became_active":"2024-03-21T14:20:21.790051+0000","last_became_peered":"2024-03-21T14:20:21.790051+0000","last_unstale":"2024-03-21T14:23:08.175387+0000","last_undegraded":"2024-03-21T14:23:08.175387+0000","last_fullsized":"2024-03-21T14:23:08.175387+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:21:19.102261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.102261+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":832,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191595+0000","last_change":"2024-03-21T14:21:19.457230+0000","last_active":"2024-03-21T14:23:08.191595+0000","last_peered":"2024-03-21T14:23:08.191595+0000","last_clean":"2024-03-21T14:23:08.191595+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:23:08.191595+0000","last_undegraded":"2024-03-21T14:23:08.191595+0000","last_fullsized":"2024-03-21T14:23:08.191595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:19.457101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.457101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":500,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175298+0000","last_change":"2024-03-21T14:22:58.625867+0000","last_active":"2024-03-21T14:23:08.175298+0000","last_peered":"2024-03-21T14:23:08.175298+0000","last_clean":"2024-03-21T14:23:08.175298+0000","last_became_active":"2024-03-21T14:19:14.675463+0000","last_became_peered":"2024-03-21T14:19:14.675463+0000","last_unstale":"2024-03-21T14:23:08.175298+0000","last_undegraded":"2024-03-21T14:23:08.175298+0000","last_fullsized":"2024-03-21T14:23:08.175298+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:58.625708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:22:58.625708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.285980+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":857,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.199055+0000","last_change":"2024-03-21T14:20:45.936416+0000","last_active":"2024-03-21T14:23:08.199055+0000","last_peered":"2024-03-21T14:23:08.199055+0000","last_clean":"2024-03-21T14:23:08.199055+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:23:08.199055+0000","last_undegraded":"2024-03-21T14:23:08.199055+0000","last_fullsized":"2024-03-21T14:23:08.199055+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.936318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.936318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'32","reported_seq":580,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190747+0000","last_change":"2024-03-21T14:20:46.036522+0000","last_active":"2024-03-21T14:23:08.190747+0000","last_peered":"2024-03-21T14:23:08.190747+0000","last_clean":"2024-03-21T14:23:08.190747+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:23:08.190747+0000","last_undegraded":"2024-03-21T14:23:08.190747+0000","last_fullsized":"2024-03-21T14:23:08.190747+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.036377+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":198,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177702+0000","last_change":"2024-03-21T14:22:40.390363+0000","last_active":"2024-03-21T14:23:08.177702+0000","last_peered":"2024-03-21T14:23:08.177702+0000","last_clean":"2024-03-21T14:23:08.177702+0000","last_became_active":"2024-03-21T14:19:14.679790+0000","last_became_peered":"2024-03-21T14:19:14.679790+0000","last_unstale":"2024-03-21T14:23:08.177702+0000","last_undegraded":"2024-03-21T14:23:08.177702+0000","last_fullsized":"2024-03-21T14:23:08.177702+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:40.390274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.390274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:05.109596+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":509,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170147+0000","last_change":"2024-03-21T14:18:42.513377+0000","last_active":"2024-03-21T14:23:08.170147+0000","last_peered":"2024-03-21T14:23:08.170147+0000","last_clean":"2024-03-21T14:23:08.170147+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:23:08.170147+0000","last_undegraded":"2024-03-21T14:23:08.170147+0000","last_fullsized":"2024-03-21T14:23:08.170147+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.513167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":311,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.186790+0000","last_change":"2024-03-21T14:21:21.656390+0000","last_active":"2024-03-21T14:23:08.186790+0000","last_peered":"2024-03-21T14:23:08.186790+0000","last_clean":"2024-03-21T14:23:08.186790+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:23:08.186790+0000","last_undegraded":"2024-03-21T14:23:08.186790+0000","last_fullsized":"2024-03-21T14:23:08.186790+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:21.656278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:21.656278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"441'14","reported_seq":880,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.199368+0000","last_change":"2024-03-21T14:22:49.297359+0000","last_active":"2024-03-21T14:23:08.199368+0000","last_peered":"2024-03-21T14:23:08.199368+0000","last_clean":"2024-03-21T14:23:08.199368+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:23:08.199368+0000","last_undegraded":"2024-03-21T14:23:08.199368+0000","last_fullsized":"2024-03-21T14:23:08.199368+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'14","last_scrub_stamp":"2024-03-21T14:22:49.297239+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.297239+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:58.065833+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":856,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.195578+0000","last_change":"2024-03-21T14:21:34.144597+0000","last_active":"2024-03-21T14:23:08.195578+0000","last_peered":"2024-03-21T14:23:08.195578+0000","last_clean":"2024-03-21T14:23:08.195578+0000","last_became_active":"2024-03-21T14:19:34.693279+0000","last_became_peered":"2024-03-21T14:19:34.693279+0000","last_unstale":"2024-03-21T14:23:08.195578+0000","last_undegraded":"2024-03-21T14:23:08.195578+0000","last_fullsized":"2024-03-21T14:23:08.195578+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.144484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.144484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":866,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.174823+0000","last_change":"2024-03-21T14:21:51.742321+0000","last_active":"2024-03-21T14:23:08.174823+0000","last_peered":"2024-03-21T14:23:08.174823+0000","last_clean":"2024-03-21T14:23:08.174823+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:23:08.174823+0000","last_undegraded":"2024-03-21T14:23:08.174823+0000","last_fullsized":"2024-03-21T14:23:08.174823+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.742159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.742159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":314,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191536+0000","last_change":"2024-03-21T14:23:04.813785+0000","last_active":"2024-03-21T14:23:08.191536+0000","last_peered":"2024-03-21T14:23:08.191536+0000","last_clean":"2024-03-21T14:23:08.191536+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:23:08.191536+0000","last_undegraded":"2024-03-21T14:23:08.191536+0000","last_fullsized":"2024-03-21T14:23:08.191536+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:23:04.813659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.813659+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.008687+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"441'32","reported_seq":914,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.185483+0000","last_change":"2024-03-21T14:22:51.290072+0000","last_active":"2024-03-21T14:23:08.185483+0000","last_peered":"2024-03-21T14:23:08.185483+0000","last_clean":"2024-03-21T14:23:08.185483+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:23:08.185483+0000","last_undegraded":"2024-03-21T14:23:08.185483+0000","last_fullsized":"2024-03-21T14:23:08.185483+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:22:51.289757+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.289757+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:20.191203+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":320,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187593+0000","last_change":"2024-03-21T14:23:03.865434+0000","last_active":"2024-03-21T14:23:08.187593+0000","last_peered":"2024-03-21T14:23:08.187593+0000","last_clean":"2024-03-21T14:23:08.187593+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:23:08.187593+0000","last_undegraded":"2024-03-21T14:23:08.187593+0000","last_fullsized":"2024-03-21T14:23:08.187593+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:03.865222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:03.865222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.684378+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":836,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.195485+0000","last_change":"2024-03-21T14:21:41.085311+0000","last_active":"2024-03-21T14:23:08.195485+0000","last_peered":"2024-03-21T14:23:08.195485+0000","last_clean":"2024-03-21T14:23:08.195485+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:23:08.195485+0000","last_undegraded":"2024-03-21T14:23:08.195485+0000","last_fullsized":"2024-03-21T14:23:08.195485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:41.085192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.085192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":860,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175598+0000","last_change":"2024-03-21T14:22:53.685161+0000","last_active":"2024-03-21T14:23:08.175598+0000","last_peered":"2024-03-21T14:23:08.175598+0000","last_clean":"2024-03-21T14:23:08.175598+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:23:08.175598+0000","last_undegraded":"2024-03-21T14:23:08.175598+0000","last_fullsized":"2024-03-21T14:23:08.175598+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:53.685078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:53.685078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.209520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'21","reported_seq":899,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190119+0000","last_change":"2024-03-21T14:23:00.867969+0000","last_active":"2024-03-21T14:23:08.190119+0000","last_peered":"2024-03-21T14:23:08.190119+0000","last_clean":"2024-03-21T14:23:08.190119+0000","last_became_active":"2024-03-21T14:19:14.682980+0000","last_became_peered":"2024-03-21T14:19:14.682980+0000","last_unstale":"2024-03-21T14:23:08.190119+0000","last_undegraded":"2024-03-21T14:23:08.190119+0000","last_fullsized":"2024-03-21T14:23:08.190119+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:23:00.867800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:00.867800+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.912184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":128,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.185172+0000","last_change":"2024-03-21T14:20:42.950594+0000","last_active":"2024-03-21T14:23:08.185172+0000","last_peered":"2024-03-21T14:23:08.185172+0000","last_clean":"2024-03-21T14:23:08.185172+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:23:08.185172+0000","last_undegraded":"2024-03-21T14:23:08.185172+0000","last_fullsized":"2024-03-21T14:23:08.185172+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.950500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.950500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":856,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190026+0000","last_change":"2024-03-21T14:21:36.498785+0000","last_active":"2024-03-21T14:23:08.190026+0000","last_peered":"2024-03-21T14:23:08.190026+0000","last_clean":"2024-03-21T14:23:08.190026+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:23:08.190026+0000","last_undegraded":"2024-03-21T14:23:08.190026+0000","last_fullsized":"2024-03-21T14:23:08.190026+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:36.498684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:36.498684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":850,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170745+0000","last_change":"2024-03-21T14:21:20.339156+0000","last_active":"2024-03-21T14:23:08.170745+0000","last_peered":"2024-03-21T14:23:08.170745+0000","last_clean":"2024-03-21T14:23:08.170745+0000","last_became_active":"2024-03-21T14:19:14.690097+0000","last_became_peered":"2024-03-21T14:19:14.690097+0000","last_unstale":"2024-03-21T14:23:08.170745+0000","last_undegraded":"2024-03-21T14:23:08.170745+0000","last_fullsized":"2024-03-21T14:23:08.170745+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:20.339046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.339046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":941,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170849+0000","last_change":"2024-03-21T14:21:50.314232+0000","last_active":"2024-03-21T14:23:08.170849+0000","last_peered":"2024-03-21T14:23:08.170849+0000","last_clean":"2024-03-21T14:23:08.170849+0000","last_became_active":"2024-03-21T14:21:26.767846+0000","last_became_peered":"2024-03-21T14:21:26.767846+0000","last_unstale":"2024-03-21T14:23:08.170849+0000","last_undegraded":"2024-03-21T14:23:08.170849+0000","last_fullsized":"2024-03-21T14:23:08.170849+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:21:50.314134+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.314134+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":717,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175189+0000","last_change":"2024-03-21T14:21:46.894626+0000","last_active":"2024-03-21T14:23:08.175189+0000","last_peered":"2024-03-21T14:23:08.175189+0000","last_clean":"2024-03-21T14:23:08.175189+0000","last_became_active":"2024-03-21T14:20:21.784416+0000","last_became_peered":"2024-03-21T14:20:21.784416+0000","last_unstale":"2024-03-21T14:23:08.175189+0000","last_undegraded":"2024-03-21T14:23:08.175189+0000","last_fullsized":"2024-03-21T14:23:08.175189+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.894547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.894547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":1066,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.770280+0000","last_change":"2024-03-21T14:21:50.631849+0000","last_active":"2024-03-21T14:23:08.770280+0000","last_peered":"2024-03-21T14:23:08.770280+0000","last_clean":"2024-03-21T14:23:08.770280+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:23:08.770280+0000","last_undegraded":"2024-03-21T14:23:08.770280+0000","last_fullsized":"2024-03-21T14:23:08.770280+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'10","last_scrub_stamp":"2024-03-21T14:21:50.631459+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.631459+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:17.757891+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":854,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.179222+0000","last_change":"2024-03-21T14:21:34.062701+0000","last_active":"2024-03-21T14:23:08.179222+0000","last_peered":"2024-03-21T14:23:08.179222+0000","last_clean":"2024-03-21T14:23:08.179222+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:23:08.179222+0000","last_undegraded":"2024-03-21T14:23:08.179222+0000","last_fullsized":"2024-03-21T14:23:08.179222+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.062588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.062588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'35","reported_seq":882,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.185255+0000","last_change":"2024-03-21T14:21:18.712180+0000","last_active":"2024-03-21T14:23:08.185255+0000","last_peered":"2024-03-21T14:23:08.185255+0000","last_clean":"2024-03-21T14:23:08.185255+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:23:08.185255+0000","last_undegraded":"2024-03-21T14:23:08.185255+0000","last_fullsized":"2024-03-21T14:23:08.185255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:18.712011+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:21:18.712011+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":816,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171611+0000","last_change":"2024-03-21T14:20:38.602527+0000","last_active":"2024-03-21T14:23:08.171611+0000","last_peered":"2024-03-21T14:23:08.171611+0000","last_clean":"2024-03-21T14:23:08.171611+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:23:08.171611+0000","last_undegraded":"2024-03-21T14:23:08.171611+0000","last_fullsized":"2024-03-21T14:23:08.171611+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:38.602405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:20:38.602405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":167,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.737992+0000","last_change":"2024-03-21T14:20:44.917571+0000","last_active":"2024-03-21T14:23:08.737992+0000","last_peered":"2024-03-21T14:23:08.737992+0000","last_clean":"2024-03-21T14:23:08.737992+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:23:08.737992+0000","last_undegraded":"2024-03-21T14:23:08.737992+0000","last_fullsized":"2024-03-21T14:23:08.737992+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:20:44.917480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.917480+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":128,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.196395+0000","last_change":"2024-03-21T14:22:43.375315+0000","last_active":"2024-03-21T14:23:08.196395+0000","last_peered":"2024-03-21T14:23:08.196395+0000","last_clean":"2024-03-21T14:23:08.196395+0000","last_became_active":"2024-03-21T14:20:21.784349+0000","last_became_peered":"2024-03-21T14:20:21.784349+0000","last_unstale":"2024-03-21T14:23:08.196395+0000","last_undegraded":"2024-03-21T14:23:08.196395+0000","last_fullsized":"2024-03-21T14:23:08.196395+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.375045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.375045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:00.172598+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"441'25","reported_seq":737,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175405+0000","last_change":"2024-03-21T14:21:37.034754+0000","last_active":"2024-03-21T14:23:08.175405+0000","last_peered":"2024-03-21T14:23:08.175405+0000","last_clean":"2024-03-21T14:23:08.175405+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:23:08.175405+0000","last_undegraded":"2024-03-21T14:23:08.175405+0000","last_fullsized":"2024-03-21T14:23:08.175405+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'19","last_scrub_stamp":"2024-03-21T14:21:37.034609+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:21:37.034609+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":673,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.180163+0000","last_change":"2024-03-21T14:21:49.278092+0000","last_active":"2024-03-21T14:23:08.180163+0000","last_peered":"2024-03-21T14:23:08.180163+0000","last_clean":"2024-03-21T14:23:08.180163+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:23:08.180163+0000","last_undegraded":"2024-03-21T14:23:08.180163+0000","last_fullsized":"2024-03-21T14:23:08.180163+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:49.277585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.277585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:14.479063+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":120,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.186972+0000","last_change":"2024-03-21T14:20:21.787792+0000","last_active":"2024-03-21T14:23:08.186972+0000","last_peered":"2024-03-21T14:23:08.186972+0000","last_clean":"2024-03-21T14:23:08.186972+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:23:08.186972+0000","last_undegraded":"2024-03-21T14:23:08.186972+0000","last_fullsized":"2024-03-21T14:23:08.186972+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.399758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":828,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170497+0000","last_change":"2024-03-21T14:22:39.540986+0000","last_active":"2024-03-21T14:23:08.170497+0000","last_peered":"2024-03-21T14:23:08.170497+0000","last_clean":"2024-03-21T14:23:08.170497+0000","last_became_active":"2024-03-21T14:21:26.767611+0000","last_became_peered":"2024-03-21T14:21:26.767611+0000","last_unstale":"2024-03-21T14:23:08.170497+0000","last_undegraded":"2024-03-21T14:23:08.170497+0000","last_fullsized":"2024-03-21T14:23:08.170497+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:39.540852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:39.540852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:44.126115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"441'25","reported_seq":546,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187644+0000","last_change":"2024-03-21T14:21:49.128441+0000","last_active":"2024-03-21T14:23:08.187644+0000","last_peered":"2024-03-21T14:23:08.187644+0000","last_clean":"2024-03-21T14:23:08.187644+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:23:08.187644+0000","last_undegraded":"2024-03-21T14:23:08.187644+0000","last_fullsized":"2024-03-21T14:23:08.187644+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'19","last_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_deep_scrub":"432'19","last_deep_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.128273+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:08.920118+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":854,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.174972+0000","last_change":"2024-03-21T14:22:43.886872+0000","last_active":"2024-03-21T14:23:08.174972+0000","last_peered":"2024-03-21T14:23:08.174972+0000","last_clean":"2024-03-21T14:23:08.174972+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:23:08.174972+0000","last_undegraded":"2024-03-21T14:23:08.174972+0000","last_fullsized":"2024-03-21T14:23:08.174972+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.886754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.886754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.618770+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":310,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187563+0000","last_change":"2024-03-21T14:21:26.752265+0000","last_active":"2024-03-21T14:23:08.187563+0000","last_peered":"2024-03-21T14:23:08.187563+0000","last_clean":"2024-03-21T14:23:08.187563+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:23:08.187563+0000","last_undegraded":"2024-03-21T14:23:08.187563+0000","last_fullsized":"2024-03-21T14:23:08.187563+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:26.752126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:21:26.752126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":71,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177610+0000","last_change":"2024-03-21T14:22:54.477723+0000","last_active":"2024-03-21T14:23:08.177610+0000","last_peered":"2024-03-21T14:23:08.177610+0000","last_clean":"2024-03-21T14:23:08.177610+0000","last_became_active":"2024-03-21T14:21:26.765914+0000","last_became_peered":"2024-03-21T14:21:26.765914+0000","last_unstale":"2024-03-21T14:23:08.177610+0000","last_undegraded":"2024-03-21T14:23:08.177610+0000","last_fullsized":"2024-03-21T14:23:08.177610+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.477581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.477581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:04.538947+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'18","reported_seq":137,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.186352+0000","last_change":"2024-03-21T14:20:21.796765+0000","last_active":"2024-03-21T14:23:08.186352+0000","last_peered":"2024-03-21T14:23:08.186352+0000","last_clean":"2024-03-21T14:23:08.186352+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:23:08.186352+0000","last_undegraded":"2024-03-21T14:23:08.186352+0000","last_fullsized":"2024-03-21T14:23:08.186352+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:19:32.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:32.302574+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"446'9567","reported_seq":9648,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.775193+0000","last_change":"2024-03-21T14:20:59.387014+0000","last_active":"2024-03-21T14:23:17.775193+0000","last_peered":"2024-03-21T14:23:17.775193+0000","last_clean":"2024-03-21T14:23:17.775193+0000","last_became_active":"2024-03-21T14:20:59.386663+0000","last_became_peered":"2024-03-21T14:20:59.386663+0000","last_unstale":"2024-03-21T14:23:17.775193+0000","last_undegraded":"2024-03-21T14:23:17.775193+0000","last_fullsized":"2024-03-21T14:23:17.775193+0000","mapping_epoch":411,"log_start":"438'5500","ondisk_log_start":"438'5500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4067,"log_dups_size":0,"ondisk_log_size":4067,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":300351488,"num_objects":4583,"num_object_clones":0,"num_object_copies":13749,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4583,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9567,"num_write_kb":452800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":718,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189591+0000","last_change":"2024-03-21T14:22:52.940837+0000","last_active":"2024-03-21T14:23:08.189591+0000","last_peered":"2024-03-21T14:23:08.189591+0000","last_clean":"2024-03-21T14:23:08.189591+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:23:08.189591+0000","last_undegraded":"2024-03-21T14:23:08.189591+0000","last_fullsized":"2024-03-21T14:23:08.189591+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_clean_scrub_stamp":"2024-03-21T14:22:52.940702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:19.596185+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":891,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175411+0000","last_change":"2024-03-21T14:21:41.979701+0000","last_active":"2024-03-21T14:23:08.175411+0000","last_peered":"2024-03-21T14:23:08.175411+0000","last_clean":"2024-03-21T14:23:08.175411+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:23:08.175411+0000","last_undegraded":"2024-03-21T14:23:08.175411+0000","last_fullsized":"2024-03-21T14:23:08.175411+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:21:41.979582+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.979582+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":791,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.174654+0000","last_change":"2024-03-21T14:22:37.805371+0000","last_active":"2024-03-21T14:23:08.174654+0000","last_peered":"2024-03-21T14:23:08.174654+0000","last_clean":"2024-03-21T14:23:08.174654+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:23:08.174654+0000","last_undegraded":"2024-03-21T14:23:08.174654+0000","last_fullsized":"2024-03-21T14:23:08.174654+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:22:37.805239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:37.805239+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:56.435148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"372'10","reported_seq":1113,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.737374+0000","last_change":"2024-03-21T14:21:35.014048+0000","last_active":"2024-03-21T14:23:08.737374+0000","last_peered":"2024-03-21T14:23:08.737374+0000","last_clean":"2024-03-21T14:23:08.737374+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:23:08.737374+0000","last_undegraded":"2024-03-21T14:23:08.737374+0000","last_fullsized":"2024-03-21T14:23:08.737374+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:21:35.013952+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_clean_scrub_stamp":"2024-03-21T14:21:35.013952+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:14.661192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"446'9468","reported_seq":9548,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:18.161992+0000","last_change":"2024-03-21T14:20:59.242772+0000","last_active":"2024-03-21T14:23:18.161992+0000","last_peered":"2024-03-21T14:23:18.161992+0000","last_clean":"2024-03-21T14:23:18.161992+0000","last_became_active":"2024-03-21T14:20:59.241791+0000","last_became_peered":"2024-03-21T14:20:59.241791+0000","last_unstale":"2024-03-21T14:23:18.161992+0000","last_undegraded":"2024-03-21T14:23:18.161992+0000","last_fullsized":"2024-03-21T14:23:18.161992+0000","mapping_epoch":411,"log_start":"436'4400","ondisk_log_start":"436'4400","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5068,"log_dups_size":0,"ondisk_log_size":5068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294518784,"num_objects":4494,"num_object_clones":0,"num_object_copies":13482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9468,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":578,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.195235+0000","last_change":"2024-03-21T14:22:41.593754+0000","last_active":"2024-03-21T14:23:08.195235+0000","last_peered":"2024-03-21T14:23:08.195235+0000","last_clean":"2024-03-21T14:23:08.195235+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:23:08.195235+0000","last_undegraded":"2024-03-21T14:23:08.195235+0000","last_fullsized":"2024-03-21T14:23:08.195235+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:41.593644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:41.593644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:41.855080+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":869,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175758+0000","last_change":"2024-03-21T14:21:50.747507+0000","last_active":"2024-03-21T14:23:08.175758+0000","last_peered":"2024-03-21T14:23:08.175758+0000","last_clean":"2024-03-21T14:23:08.175758+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:23:08.175758+0000","last_undegraded":"2024-03-21T14:23:08.175758+0000","last_fullsized":"2024-03-21T14:23:08.175758+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.747341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":919,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170436+0000","last_change":"2024-03-21T14:20:21.797847+0000","last_active":"2024-03-21T14:23:08.170436+0000","last_peered":"2024-03-21T14:23:08.170436+0000","last_clean":"2024-03-21T14:23:08.170436+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:23:08.170436+0000","last_undegraded":"2024-03-21T14:23:08.170436+0000","last_fullsized":"2024-03-21T14:23:08.170436+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:18:29.432719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.432719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":685,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177028+0000","last_change":"2024-03-21T14:21:54.295031+0000","last_active":"2024-03-21T14:23:08.177028+0000","last_peered":"2024-03-21T14:23:08.177028+0000","last_clean":"2024-03-21T14:23:08.177028+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:23:08.177028+0000","last_undegraded":"2024-03-21T14:23:08.177028+0000","last_fullsized":"2024-03-21T14:23:08.177028+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"446'9159","reported_seq":9240,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.627837+0000","last_change":"2024-03-21T14:20:59.387129+0000","last_active":"2024-03-21T14:23:17.627837+0000","last_peered":"2024-03-21T14:23:17.627837+0000","last_clean":"2024-03-21T14:23:17.627837+0000","last_became_active":"2024-03-21T14:20:59.386704+0000","last_became_peered":"2024-03-21T14:20:59.386704+0000","last_unstale":"2024-03-21T14:23:17.627837+0000","last_undegraded":"2024-03-21T14:23:17.627837+0000","last_fullsized":"2024-03-21T14:23:17.627837+0000","mapping_epoch":411,"log_start":"436'4900","ondisk_log_start":"436'4900","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4259,"log_dups_size":0,"ondisk_log_size":4259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.189122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9159,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":206,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177745+0000","last_change":"2024-03-21T14:22:54.306435+0000","last_active":"2024-03-21T14:23:08.177745+0000","last_peered":"2024-03-21T14:23:08.177745+0000","last_clean":"2024-03-21T14:23:08.177745+0000","last_became_active":"2024-03-21T14:19:14.700785+0000","last_became_peered":"2024-03-21T14:19:14.700785+0000","last_unstale":"2024-03-21T14:23:08.177745+0000","last_undegraded":"2024-03-21T14:23:08.177745+0000","last_fullsized":"2024-03-21T14:23:08.177745+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.306329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.306329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.846326+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":646,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177327+0000","last_change":"2024-03-21T14:23:04.356857+0000","last_active":"2024-03-21T14:23:08.177327+0000","last_peered":"2024-03-21T14:23:08.177327+0000","last_clean":"2024-03-21T14:23:08.177327+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:23:08.177327+0000","last_undegraded":"2024-03-21T14:23:08.177327+0000","last_fullsized":"2024-03-21T14:23:08.177327+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:04.356730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.356730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.865958+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":71,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178095+0000","last_change":"2024-03-21T14:23:06.509103+0000","last_active":"2024-03-21T14:23:08.178095+0000","last_peered":"2024-03-21T14:23:08.178095+0000","last_clean":"2024-03-21T14:23:08.178095+0000","last_became_active":"2024-03-21T14:21:26.764088+0000","last_became_peered":"2024-03-21T14:21:26.764088+0000","last_unstale":"2024-03-21T14:23:08.178095+0000","last_undegraded":"2024-03-21T14:23:08.178095+0000","last_fullsized":"2024-03-21T14:23:08.178095+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:06.509028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:06.509028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.147435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"441'60","reported_seq":818,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189900+0000","last_change":"2024-03-21T14:20:49.145974+0000","last_active":"2024-03-21T14:23:08.189900+0000","last_peered":"2024-03-21T14:23:08.189900+0000","last_clean":"2024-03-21T14:23:08.189900+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:23:08.189900+0000","last_undegraded":"2024-03-21T14:23:08.189900+0000","last_fullsized":"2024-03-21T14:23:08.189900+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.145792+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"446'9219","reported_seq":12186,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.073325+0000","last_change":"2024-03-21T14:22:56.879969+0000","last_active":"2024-03-21T14:23:17.073325+0000","last_peered":"2024-03-21T14:23:17.073325+0000","last_clean":"2024-03-21T14:23:17.073325+0000","last_became_active":"2024-03-21T14:21:34.716304+0000","last_became_peered":"2024-03-21T14:21:34.716304+0000","last_unstale":"2024-03-21T14:23:17.073325+0000","last_undegraded":"2024-03-21T14:23:17.073325+0000","last_fullsized":"2024-03-21T14:23:17.073325+0000","mapping_epoch":427,"log_start":"436'4626","ondisk_log_start":"436'4626","created":411,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'7840","last_scrub_stamp":"2024-03-21T14:22:56.879850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:56.879850+0000","objects_scrubbed":6014,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:20.616700+0000","scrub_duration":1210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301137920,"num_objects":4595,"num_object_clones":0,"num_object_copies":13785,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4595,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9219,"num_write_kb":442048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1405,"num_bytes_recovered":92078080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":205,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178573+0000","last_change":"2024-03-21T14:22:29.603576+0000","last_active":"2024-03-21T14:23:08.178573+0000","last_peered":"2024-03-21T14:23:08.178573+0000","last_clean":"2024-03-21T14:23:08.178573+0000","last_became_active":"2024-03-21T14:19:34.690841+0000","last_became_peered":"2024-03-21T14:19:34.690841+0000","last_unstale":"2024-03-21T14:23:08.178573+0000","last_undegraded":"2024-03-21T14:23:08.178573+0000","last_fullsized":"2024-03-21T14:23:08.178573+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:29.603450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.603450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.754649+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":685,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178023+0000","last_change":"2024-03-21T14:21:46.343726+0000","last_active":"2024-03-21T14:23:08.178023+0000","last_peered":"2024-03-21T14:23:08.178023+0000","last_clean":"2024-03-21T14:23:08.178023+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:23:08.178023+0000","last_undegraded":"2024-03-21T14:23:08.178023+0000","last_fullsized":"2024-03-21T14:23:08.178023+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.343600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.343600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:11.677559+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177670+0000","last_change":"2024-03-21T14:22:50.084105+0000","last_active":"2024-03-21T14:23:08.177670+0000","last_peered":"2024-03-21T14:23:08.177670+0000","last_clean":"2024-03-21T14:23:08.177670+0000","last_became_active":"2024-03-21T14:22:50.083698+0000","last_became_peered":"2024-03-21T14:22:50.083698+0000","last_unstale":"2024-03-21T14:23:08.177670+0000","last_undegraded":"2024-03-21T14:23:08.177670+0000","last_fullsized":"2024-03-21T14:23:08.177670+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:47.337388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:47.337388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.949906+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"441'21","reported_seq":349,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177601+0000","last_change":"2024-03-21T14:21:20.291857+0000","last_active":"2024-03-21T14:23:08.177601+0000","last_peered":"2024-03-21T14:23:08.177601+0000","last_clean":"2024-03-21T14:23:08.177601+0000","last_became_active":"2024-03-21T14:19:14.692456+0000","last_became_peered":"2024-03-21T14:19:14.692456+0000","last_unstale":"2024-03-21T14:23:08.177601+0000","last_undegraded":"2024-03-21T14:23:08.177601+0000","last_fullsized":"2024-03-21T14:23:08.177601+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:21:20.291754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.291754+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"446'8957","reported_seq":9037,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.636762+0000","last_change":"2024-03-21T14:20:59.241783+0000","last_active":"2024-03-21T14:23:17.636762+0000","last_peered":"2024-03-21T14:23:17.636762+0000","last_clean":"2024-03-21T14:23:17.636762+0000","last_became_active":"2024-03-21T14:20:59.241477+0000","last_became_peered":"2024-03-21T14:20:59.241477+0000","last_unstale":"2024-03-21T14:23:17.636762+0000","last_undegraded":"2024-03-21T14:23:17.636762+0000","last_fullsized":"2024-03-21T14:23:17.636762+0000","mapping_epoch":411,"log_start":"436'4700","ondisk_log_start":"436'4700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.175465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289472512,"num_objects":4417,"num_object_clones":0,"num_object_copies":13251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8957,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":821,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178887+0000","last_change":"2024-03-21T14:22:55.310949+0000","last_active":"2024-03-21T14:23:08.178887+0000","last_peered":"2024-03-21T14:23:08.178887+0000","last_clean":"2024-03-21T14:23:08.178887+0000","last_became_active":"2024-03-21T14:19:14.689666+0000","last_became_peered":"2024-03-21T14:19:14.689666+0000","last_unstale":"2024-03-21T14:23:08.178887+0000","last_undegraded":"2024-03-21T14:23:08.178887+0000","last_fullsized":"2024-03-21T14:23:08.178887+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:55.310815+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:55.310815+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:12.562170+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":825,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187730+0000","last_change":"2024-03-21T14:20:50.816435+0000","last_active":"2024-03-21T14:23:08.187730+0000","last_peered":"2024-03-21T14:23:08.187730+0000","last_clean":"2024-03-21T14:23:08.187730+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:23:08.187730+0000","last_undegraded":"2024-03-21T14:23:08.187730+0000","last_fullsized":"2024-03-21T14:23:08.187730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:50.816334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:50.816334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":345,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189748+0000","last_change":"2024-03-21T14:22:53.863644+0000","last_active":"2024-03-21T14:23:08.189748+0000","last_peered":"2024-03-21T14:23:08.189748+0000","last_clean":"2024-03-21T14:23:08.189748+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:23:08.189748+0000","last_undegraded":"2024-03-21T14:23:08.189748+0000","last_fullsized":"2024-03-21T14:23:08.189748+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:22:53.863327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:22:53.863327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.416207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":834,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170665+0000","last_change":"2024-03-21T14:22:33.543481+0000","last_active":"2024-03-21T14:23:08.170665+0000","last_peered":"2024-03-21T14:23:08.170665+0000","last_clean":"2024-03-21T14:23:08.170665+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:23:08.170665+0000","last_undegraded":"2024-03-21T14:23:08.170665+0000","last_fullsized":"2024-03-21T14:23:08.170665+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:33.543370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:33.543370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:57.805975+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"441'66","reported_seq":805,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.740348+0000","last_change":"2024-03-21T14:22:45.361704+0000","last_active":"2024-03-21T14:23:08.740348+0000","last_peered":"2024-03-21T14:23:08.740348+0000","last_clean":"2024-03-21T14:23:08.740348+0000","last_became_active":"2024-03-21T14:19:14.693616+0000","last_became_peered":"2024-03-21T14:19:14.693616+0000","last_unstale":"2024-03-21T14:23:08.740348+0000","last_undegraded":"2024-03-21T14:23:08.740348+0000","last_fullsized":"2024-03-21T14:23:08.740348+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'66","last_scrub_stamp":"2024-03-21T14:22:45.361551+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:22:45.361551+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:59.222326+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":711,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191984+0000","last_change":"2024-03-21T14:21:32.597365+0000","last_active":"2024-03-21T14:23:08.191984+0000","last_peered":"2024-03-21T14:23:08.191984+0000","last_clean":"2024-03-21T14:23:08.191984+0000","last_became_active":"2024-03-21T14:19:34.690636+0000","last_became_peered":"2024-03-21T14:19:34.690636+0000","last_unstale":"2024-03-21T14:23:08.191984+0000","last_undegraded":"2024-03-21T14:23:08.191984+0000","last_fullsized":"2024-03-21T14:23:08.191984+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:32.597257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:32.597257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"446'9474","reported_seq":9555,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.561794+0000","last_change":"2024-03-21T14:20:59.387006+0000","last_active":"2024-03-21T14:23:17.561794+0000","last_peered":"2024-03-21T14:23:17.561794+0000","last_clean":"2024-03-21T14:23:17.561794+0000","last_became_active":"2024-03-21T14:20:59.386643+0000","last_became_peered":"2024-03-21T14:20:59.386643+0000","last_unstale":"2024-03-21T14:23:17.561794+0000","last_undegraded":"2024-03-21T14:23:17.561794+0000","last_fullsized":"2024-03-21T14:23:17.561794+0000","mapping_epoch":411,"log_start":"437'5300","ondisk_log_start":"437'5300","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.186773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288489472,"num_objects":4402,"num_object_clones":0,"num_object_copies":13206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9474,"num_write_kb":444032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":1059,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170214+0000","last_change":"2024-03-21T14:21:52.357400+0000","last_active":"2024-03-21T14:23:08.170214+0000","last_peered":"2024-03-21T14:23:08.170214+0000","last_clean":"2024-03-21T14:23:08.170214+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:23:08.170214+0000","last_undegraded":"2024-03-21T14:23:08.170214+0000","last_fullsized":"2024-03-21T14:23:08.170214+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:52.357256+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:52.357256+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:09.449933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":848,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.175514+0000","last_change":"2024-03-21T14:23:03.702113+0000","last_active":"2024-03-21T14:23:08.175514+0000","last_peered":"2024-03-21T14:23:08.175514+0000","last_clean":"2024-03-21T14:23:08.175514+0000","last_became_active":"2024-03-21T14:19:14.685026+0000","last_became_peered":"2024-03-21T14:19:14.685026+0000","last_unstale":"2024-03-21T14:23:08.175514+0000","last_undegraded":"2024-03-21T14:23:08.175514+0000","last_fullsized":"2024-03-21T14:23:08.175514+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:03.702034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:03.702034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.563384+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":853,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.188092+0000","last_change":"2024-03-21T14:22:45.828480+0000","last_active":"2024-03-21T14:23:08.188092+0000","last_peered":"2024-03-21T14:23:08.188092+0000","last_clean":"2024-03-21T14:23:08.188092+0000","last_became_active":"2024-03-21T14:19:14.697007+0000","last_became_peered":"2024-03-21T14:19:14.697007+0000","last_unstale":"2024-03-21T14:23:08.188092+0000","last_undegraded":"2024-03-21T14:23:08.188092+0000","last_fullsized":"2024-03-21T14:23:08.188092+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:45.828270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:22:45.828270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:57.425654+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":680,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177032+0000","last_change":"2024-03-21T14:20:48.551092+0000","last_active":"2024-03-21T14:23:08.177032+0000","last_peered":"2024-03-21T14:23:08.177032+0000","last_clean":"2024-03-21T14:23:08.177032+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:23:08.177032+0000","last_undegraded":"2024-03-21T14:23:08.177032+0000","last_fullsized":"2024-03-21T14:23:08.177032+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.550975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.550975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":937,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190503+0000","last_change":"2024-03-21T14:21:47.628148+0000","last_active":"2024-03-21T14:23:08.190503+0000","last_peered":"2024-03-21T14:23:08.190503+0000","last_clean":"2024-03-21T14:23:08.190503+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:23:08.190503+0000","last_undegraded":"2024-03-21T14:23:08.190503+0000","last_fullsized":"2024-03-21T14:23:08.190503+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_deep_scrub":"236'22","last_deep_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.628002+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":663,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.182177+0000","last_change":"2024-03-21T14:21:42.301835+0000","last_active":"2024-03-21T14:23:08.182177+0000","last_peered":"2024-03-21T14:23:08.182177+0000","last_clean":"2024-03-21T14:23:08.182177+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:23:08.182177+0000","last_undegraded":"2024-03-21T14:23:08.182177+0000","last_fullsized":"2024-03-21T14:23:08.182177+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:42.301114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:21:42.301114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"446'9371","reported_seq":9452,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.618410+0000","last_change":"2024-03-21T14:20:59.387064+0000","last_active":"2024-03-21T14:23:17.618410+0000","last_peered":"2024-03-21T14:23:17.618410+0000","last_clean":"2024-03-21T14:23:17.618410+0000","last_became_active":"2024-03-21T14:20:59.386748+0000","last_became_peered":"2024-03-21T14:20:59.386748+0000","last_unstale":"2024-03-21T14:23:17.618410+0000","last_undegraded":"2024-03-21T14:23:17.618410+0000","last_fullsized":"2024-03-21T14:23:17.618410+0000","mapping_epoch":411,"log_start":"437'5200","ondisk_log_start":"437'5200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4171,"log_dups_size":0,"ondisk_log_size":4171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.179617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292880384,"num_objects":4469,"num_object_clones":0,"num_object_copies":13407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9371,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":310,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187055+0000","last_change":"2024-03-21T14:21:23.819765+0000","last_active":"2024-03-21T14:23:08.187055+0000","last_peered":"2024-03-21T14:23:08.187055+0000","last_clean":"2024-03-21T14:23:08.187055+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:23:08.187055+0000","last_undegraded":"2024-03-21T14:23:08.187055+0000","last_fullsized":"2024-03-21T14:23:08.187055+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:23.819644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:23.819644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1159,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.174900+0000","last_change":"2024-03-21T14:21:44.886680+0000","last_active":"2024-03-21T14:23:08.174900+0000","last_peered":"2024-03-21T14:23:08.174900+0000","last_clean":"2024-03-21T14:23:08.174900+0000","last_became_active":"2024-03-21T14:19:37.727165+0000","last_became_peered":"2024-03-21T14:19:37.727165+0000","last_unstale":"2024-03-21T14:23:08.174900+0000","last_undegraded":"2024-03-21T14:23:08.174900+0000","last_fullsized":"2024-03-21T14:23:08.174900+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:44.886598+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:44.886598+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":859,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.196721+0000","last_change":"2024-03-21T14:22:30.336811+0000","last_active":"2024-03-21T14:23:08.196721+0000","last_peered":"2024-03-21T14:23:08.196721+0000","last_clean":"2024-03-21T14:23:08.196721+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:23:08.196721+0000","last_undegraded":"2024-03-21T14:23:08.196721+0000","last_fullsized":"2024-03-21T14:23:08.196721+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:30.335992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:22:30.335992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:48.200343+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"441'41","reported_seq":742,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.737868+0000","last_change":"2024-03-21T14:22:58.255728+0000","last_active":"2024-03-21T14:23:08.737868+0000","last_peered":"2024-03-21T14:23:08.737868+0000","last_clean":"2024-03-21T14:23:08.737868+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:23:08.737868+0000","last_undegraded":"2024-03-21T14:23:08.737868+0000","last_fullsized":"2024-03-21T14:23:08.737868+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'41","last_scrub_stamp":"2024-03-21T14:22:58.255560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:58.255560+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.445411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":205,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.180260+0000","last_change":"2024-03-21T14:23:07.337422+0000","last_active":"2024-03-21T14:23:08.180260+0000","last_peered":"2024-03-21T14:23:08.180260+0000","last_clean":"2024-03-21T14:23:08.180260+0000","last_became_active":"2024-03-21T14:19:14.690145+0000","last_became_peered":"2024-03-21T14:19:14.690145+0000","last_unstale":"2024-03-21T14:23:08.180260+0000","last_undegraded":"2024-03-21T14:23:08.180260+0000","last_fullsized":"2024-03-21T14:23:08.180260+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:07.333488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:07.333488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:12.814063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"446'9485","reported_seq":9566,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.632450+0000","last_change":"2024-03-21T14:20:59.239970+0000","last_active":"2024-03-21T14:23:17.632450+0000","last_peered":"2024-03-21T14:23:17.632450+0000","last_clean":"2024-03-21T14:23:17.632450+0000","last_became_active":"2024-03-21T14:20:59.239659+0000","last_became_peered":"2024-03-21T14:20:59.239659+0000","last_unstale":"2024-03-21T14:23:17.632450+0000","last_undegraded":"2024-03-21T14:23:17.632450+0000","last_fullsized":"2024-03-21T14:23:17.632450+0000","mapping_epoch":411,"log_start":"437'5300","ondisk_log_start":"437'5300","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.186728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299171840,"num_objects":4565,"num_object_clones":0,"num_object_copies":13695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4565,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9485,"num_write_kb":449600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":915,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190453+0000","last_change":"2024-03-21T14:22:32.735774+0000","last_active":"2024-03-21T14:23:08.190453+0000","last_peered":"2024-03-21T14:23:08.190453+0000","last_clean":"2024-03-21T14:23:08.190453+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:23:08.190453+0000","last_undegraded":"2024-03-21T14:23:08.190453+0000","last_fullsized":"2024-03-21T14:23:08.190453+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:32.735646+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:32.735646+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:53.418603+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":851,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190253+0000","last_change":"2024-03-21T14:20:48.131092+0000","last_active":"2024-03-21T14:23:08.190253+0000","last_peered":"2024-03-21T14:23:08.190253+0000","last_clean":"2024-03-21T14:23:08.190253+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:23:08.190253+0000","last_undegraded":"2024-03-21T14:23:08.190253+0000","last_fullsized":"2024-03-21T14:23:08.190253+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.130989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.130989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":644,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170354+0000","last_change":"2024-03-21T14:22:51.407734+0000","last_active":"2024-03-21T14:23:08.170354+0000","last_peered":"2024-03-21T14:23:08.170354+0000","last_clean":"2024-03-21T14:23:08.170354+0000","last_became_active":"2024-03-21T14:20:21.778326+0000","last_became_peered":"2024-03-21T14:20:21.778326+0000","last_unstale":"2024-03-21T14:23:08.170354+0000","last_undegraded":"2024-03-21T14:23:08.170354+0000","last_fullsized":"2024-03-21T14:23:08.170354+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:51.407608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.407608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.985048+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":2000,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:16.761557+0000","last_change":"2024-03-21T14:21:41.487654+0000","last_active":"2024-03-21T14:23:16.761557+0000","last_peered":"2024-03-21T14:23:16.761557+0000","last_clean":"2024-03-21T14:23:16.761557+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:23:16.761557+0000","last_undegraded":"2024-03-21T14:23:16.761557+0000","last_fullsized":"2024-03-21T14:23:16.761557+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:21:41.487571+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.487571+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1178,"num_read_kb":1173,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":806,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191882+0000","last_change":"2024-03-21T14:22:51.894009+0000","last_active":"2024-03-21T14:23:08.191882+0000","last_peered":"2024-03-21T14:23:08.191882+0000","last_clean":"2024-03-21T14:23:08.191882+0000","last_became_active":"2024-03-21T14:19:34.692301+0000","last_became_peered":"2024-03-21T14:19:34.692301+0000","last_unstale":"2024-03-21T14:23:08.191882+0000","last_undegraded":"2024-03-21T14:23:08.191882+0000","last_fullsized":"2024-03-21T14:23:08.191882+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_deep_scrub":"441'36","last_deep_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.893561+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.310816+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":312,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190910+0000","last_change":"2024-03-21T14:21:40.525563+0000","last_active":"2024-03-21T14:23:08.190910+0000","last_peered":"2024-03-21T14:23:08.190910+0000","last_clean":"2024-03-21T14:23:08.190910+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:23:08.190910+0000","last_undegraded":"2024-03-21T14:23:08.190910+0000","last_fullsized":"2024-03-21T14:23:08.190910+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:40.525392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:21:40.525392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"446'9379","reported_seq":9460,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.802824+0000","last_change":"2024-03-21T14:20:59.110120+0000","last_active":"2024-03-21T14:23:17.802824+0000","last_peered":"2024-03-21T14:23:17.802824+0000","last_clean":"2024-03-21T14:23:17.802824+0000","last_became_active":"2024-03-21T14:20:59.109825+0000","last_became_peered":"2024-03-21T14:20:59.109825+0000","last_unstale":"2024-03-21T14:23:17.802824+0000","last_undegraded":"2024-03-21T14:23:17.802824+0000","last_fullsized":"2024-03-21T14:23:17.802824+0000","mapping_epoch":411,"log_start":"437'5300","ondisk_log_start":"437'5300","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4079,"log_dups_size":0,"ondisk_log_size":4079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9379,"num_write_kb":442816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":643,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.187338+0000","last_change":"2024-03-21T14:20:30.945601+0000","last_active":"2024-03-21T14:23:08.187338+0000","last_peered":"2024-03-21T14:23:08.187338+0000","last_clean":"2024-03-21T14:23:08.187338+0000","last_became_active":"2024-03-21T14:20:24.825637+0000","last_became_peered":"2024-03-21T14:20:24.825637+0000","last_unstale":"2024-03-21T14:23:08.187338+0000","last_undegraded":"2024-03-21T14:23:08.187338+0000","last_fullsized":"2024-03-21T14:23:08.187338+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:30.945471+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.945471+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":848,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.189728+0000","last_change":"2024-03-21T14:21:15.242152+0000","last_active":"2024-03-21T14:23:08.189728+0000","last_peered":"2024-03-21T14:23:08.189728+0000","last_clean":"2024-03-21T14:23:08.189728+0000","last_became_active":"2024-03-21T14:19:14.697998+0000","last_became_peered":"2024-03-21T14:19:14.697998+0000","last_unstale":"2024-03-21T14:23:08.189728+0000","last_undegraded":"2024-03-21T14:23:08.189728+0000","last_fullsized":"2024-03-21T14:23:08.189728+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:15.242057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:21:15.242057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":854,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190578+0000","last_change":"2024-03-21T14:22:48.925469+0000","last_active":"2024-03-21T14:23:08.190578+0000","last_peered":"2024-03-21T14:23:08.190578+0000","last_clean":"2024-03-21T14:23:08.190578+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:23:08.190578+0000","last_undegraded":"2024-03-21T14:23:08.190578+0000","last_fullsized":"2024-03-21T14:23:08.190578+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:48.925350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.925350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:14.372873+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'39","reported_seq":591,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.179131+0000","last_change":"2024-03-21T14:22:29.489662+0000","last_active":"2024-03-21T14:23:08.179131+0000","last_peered":"2024-03-21T14:23:08.179131+0000","last_clean":"2024-03-21T14:23:08.179131+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:23:08.179131+0000","last_undegraded":"2024-03-21T14:23:08.179131+0000","last_fullsized":"2024-03-21T14:23:08.179131+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:22:29.489522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.489522+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:31.892959+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":209,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177608+0000","last_change":"2024-03-21T14:22:50.356522+0000","last_active":"2024-03-21T14:23:08.177608+0000","last_peered":"2024-03-21T14:23:08.177608+0000","last_clean":"2024-03-21T14:23:08.177608+0000","last_became_active":"2024-03-21T14:19:14.675538+0000","last_became_peered":"2024-03-21T14:19:14.675538+0000","last_unstale":"2024-03-21T14:23:08.177608+0000","last_undegraded":"2024-03-21T14:23:08.177608+0000","last_fullsized":"2024-03-21T14:23:08.177608+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:50.356413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:22:50.356413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:56.994527+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"446'9475","reported_seq":12478,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:18.182825+0000","last_change":"2024-03-21T14:21:41.926494+0000","last_active":"2024-03-21T14:23:18.182825+0000","last_peered":"2024-03-21T14:23:18.182825+0000","last_clean":"2024-03-21T14:23:18.182825+0000","last_became_active":"2024-03-21T14:21:41.923763+0000","last_became_peered":"2024-03-21T14:21:41.923763+0000","last_unstale":"2024-03-21T14:23:18.182825+0000","last_undegraded":"2024-03-21T14:23:18.182825+0000","last_fullsized":"2024-03-21T14:23:18.182825+0000","mapping_epoch":431,"log_start":"436'4400","ondisk_log_start":"436'4400","created":411,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298844184,"num_objects":4561,"num_object_clones":0,"num_object_copies":13683,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4561,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9475,"num_write_kb":449089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":95420416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":817,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.186465+0000","last_change":"2024-03-21T14:20:24.821031+0000","last_active":"2024-03-21T14:23:08.186465+0000","last_peered":"2024-03-21T14:23:08.186465+0000","last_clean":"2024-03-21T14:23:08.186465+0000","last_became_active":"2024-03-21T14:20:24.819844+0000","last_became_peered":"2024-03-21T14:20:24.819844+0000","last_unstale":"2024-03-21T14:23:08.186465+0000","last_undegraded":"2024-03-21T14:23:08.186465+0000","last_fullsized":"2024-03-21T14:23:08.186465+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:15.665015+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.665015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":348,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.198828+0000","last_change":"2024-03-21T14:22:56.247550+0000","last_active":"2024-03-21T14:23:08.198828+0000","last_peered":"2024-03-21T14:23:08.198828+0000","last_clean":"2024-03-21T14:23:08.198828+0000","last_became_active":"2024-03-21T14:19:15.688238+0000","last_became_peered":"2024-03-21T14:19:15.688238+0000","last_unstale":"2024-03-21T14:23:08.198828+0000","last_undegraded":"2024-03-21T14:23:08.198828+0000","last_fullsized":"2024-03-21T14:23:08.198828+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:56.247428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:56.247428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:16.119271+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":777,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.188174+0000","last_change":"2024-03-21T14:20:32.884624+0000","last_active":"2024-03-21T14:23:08.188174+0000","last_peered":"2024-03-21T14:23:08.188174+0000","last_clean":"2024-03-21T14:23:08.188174+0000","last_became_active":"2024-03-21T14:19:14.695943+0000","last_became_peered":"2024-03-21T14:19:14.695943+0000","last_unstale":"2024-03-21T14:23:08.188174+0000","last_undegraded":"2024-03-21T14:23:08.188174+0000","last_fullsized":"2024-03-21T14:23:08.188174+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:32.884514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:32.884514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":899,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190074+0000","last_change":"2024-03-21T14:21:43.496288+0000","last_active":"2024-03-21T14:23:08.190074+0000","last_peered":"2024-03-21T14:23:08.190074+0000","last_clean":"2024-03-21T14:23:08.190074+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:23:08.190074+0000","last_undegraded":"2024-03-21T14:23:08.190074+0000","last_fullsized":"2024-03-21T14:23:08.190074+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:43.496137+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:21:43.496137+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":851,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.738187+0000","last_change":"2024-03-21T14:22:55.163828+0000","last_active":"2024-03-21T14:23:08.738187+0000","last_peered":"2024-03-21T14:23:08.738187+0000","last_clean":"2024-03-21T14:23:08.738187+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:23:08.738187+0000","last_undegraded":"2024-03-21T14:23:08.738187+0000","last_fullsized":"2024-03-21T14:23:08.738187+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'11","last_scrub_stamp":"2024-03-21T14:22:55.163691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:55.163691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:14.088158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"446'9434","reported_seq":9514,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:16.884213+0000","last_change":"2024-03-21T14:20:59.110795+0000","last_active":"2024-03-21T14:23:16.884213+0000","last_peered":"2024-03-21T14:23:16.884213+0000","last_clean":"2024-03-21T14:23:16.884213+0000","last_became_active":"2024-03-21T14:20:59.110468+0000","last_became_peered":"2024-03-21T14:20:59.110468+0000","last_unstale":"2024-03-21T14:23:16.884213+0000","last_undegraded":"2024-03-21T14:23:16.884213+0000","last_fullsized":"2024-03-21T14:23:16.884213+0000","mapping_epoch":411,"log_start":"436'5000","ondisk_log_start":"436'5000","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.398016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298713088,"num_objects":4558,"num_object_clones":0,"num_object_copies":13674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9434,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":899,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.185064+0000","last_change":"2024-03-21T14:20:52.764101+0000","last_active":"2024-03-21T14:23:08.185064+0000","last_peered":"2024-03-21T14:23:08.185064+0000","last_clean":"2024-03-21T14:23:08.185064+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:23:08.185064+0000","last_undegraded":"2024-03-21T14:23:08.185064+0000","last_fullsized":"2024-03-21T14:23:08.185064+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:52.763988+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.763988+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":848,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.179319+0000","last_change":"2024-03-21T14:22:49.915554+0000","last_active":"2024-03-21T14:23:08.179319+0000","last_peered":"2024-03-21T14:23:08.179319+0000","last_clean":"2024-03-21T14:23:08.179319+0000","last_became_active":"2024-03-21T14:19:14.684876+0000","last_became_peered":"2024-03-21T14:19:14.684876+0000","last_unstale":"2024-03-21T14:23:08.179319+0000","last_undegraded":"2024-03-21T14:23:08.179319+0000","last_fullsized":"2024-03-21T14:23:08.179319+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:49.915445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:27.609702+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.915445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:04.268044+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":713,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191728+0000","last_change":"2024-03-21T14:20:51.116248+0000","last_active":"2024-03-21T14:23:08.191728+0000","last_peered":"2024-03-21T14:23:08.191728+0000","last_clean":"2024-03-21T14:23:08.191728+0000","last_became_active":"2024-03-21T14:19:34.692258+0000","last_became_peered":"2024-03-21T14:19:34.692258+0000","last_unstale":"2024-03-21T14:23:08.191728+0000","last_undegraded":"2024-03-21T14:23:08.191728+0000","last_fullsized":"2024-03-21T14:23:08.191728+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:20:51.116139+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.116139+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"441'42","reported_seq":939,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170329+0000","last_change":"2024-03-21T14:22:49.399821+0000","last_active":"2024-03-21T14:23:08.170329+0000","last_peered":"2024-03-21T14:23:08.170329+0000","last_clean":"2024-03-21T14:23:08.170329+0000","last_became_active":"2024-03-21T14:21:26.768622+0000","last_became_peered":"2024-03-21T14:21:26.768622+0000","last_unstale":"2024-03-21T14:23:08.170329+0000","last_undegraded":"2024-03-21T14:23:08.170329+0000","last_fullsized":"2024-03-21T14:23:08.170329+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-21T14:22:49.399707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.399707+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:00.879647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":1095,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.737889+0000","last_change":"2024-03-21T14:22:38.443967+0000","last_active":"2024-03-21T14:23:08.737889+0000","last_peered":"2024-03-21T14:23:08.737889+0000","last_clean":"2024-03-21T14:23:08.737889+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:23:08.737889+0000","last_undegraded":"2024-03-21T14:23:08.737889+0000","last_fullsized":"2024-03-21T14:23:08.737889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:22:38.443825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:38.443825+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:43.527309+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"446'9328","reported_seq":10194,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:16.880957+0000","last_change":"2024-03-21T14:21:28.769176+0000","last_active":"2024-03-21T14:23:16.880957+0000","last_peered":"2024-03-21T14:23:16.880957+0000","last_clean":"2024-03-21T14:23:16.880957+0000","last_became_active":"2024-03-21T14:21:28.766274+0000","last_became_peered":"2024-03-21T14:21:28.766274+0000","last_unstale":"2024-03-21T14:23:16.880957+0000","last_undegraded":"2024-03-21T14:23:16.880957+0000","last_fullsized":"2024-03-21T14:23:16.880957+0000","mapping_epoch":425,"log_start":"436'4900","ondisk_log_start":"436'4900","created":411,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.397960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301727744,"num_objects":4604,"num_object_clones":0,"num_object_copies":13812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9328,"num_write_kb":445824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":59375616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":901,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170784+0000","last_change":"2024-03-21T14:21:19.302873+0000","last_active":"2024-03-21T14:23:08.170784+0000","last_peered":"2024-03-21T14:23:08.170784+0000","last_clean":"2024-03-21T14:23:08.170784+0000","last_became_active":"2024-03-21T14:19:14.682251+0000","last_became_peered":"2024-03-21T14:19:14.682251+0000","last_unstale":"2024-03-21T14:23:08.170784+0000","last_undegraded":"2024-03-21T14:23:08.170784+0000","last_fullsized":"2024-03-21T14:23:08.170784+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:19.302784+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.302784+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":848,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170356+0000","last_change":"2024-03-21T14:22:35.628602+0000","last_active":"2024-03-21T14:23:08.170356+0000","last_peered":"2024-03-21T14:23:08.170356+0000","last_clean":"2024-03-21T14:23:08.170356+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:23:08.170356+0000","last_undegraded":"2024-03-21T14:23:08.170356+0000","last_fullsized":"2024-03-21T14:23:08.170356+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:35.628491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.628491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:49.293615+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":845,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.186882+0000","last_change":"2024-03-21T14:21:39.060809+0000","last_active":"2024-03-21T14:23:08.186882+0000","last_peered":"2024-03-21T14:23:08.186882+0000","last_clean":"2024-03-21T14:23:08.186882+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:23:08.186882+0000","last_undegraded":"2024-03-21T14:23:08.186882+0000","last_fullsized":"2024-03-21T14:23:08.186882+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:39.060697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:21:39.060697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'50","reported_seq":164,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.198493+0000","last_change":"2024-03-21T14:23:04.313764+0000","last_active":"2024-03-21T14:23:08.198493+0000","last_peered":"2024-03-21T14:23:08.198493+0000","last_clean":"2024-03-21T14:23:08.198493+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:23:08.198493+0000","last_undegraded":"2024-03-21T14:23:08.198493+0000","last_fullsized":"2024-03-21T14:23:08.198493+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'50","last_scrub_stamp":"2024-03-21T14:23:04.313411+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.313411+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:11.545037+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":820,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170205+0000","last_change":"2024-03-21T14:22:43.461461+0000","last_active":"2024-03-21T14:23:08.170205+0000","last_peered":"2024-03-21T14:23:08.170205+0000","last_clean":"2024-03-21T14:23:08.170205+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:23:08.170205+0000","last_undegraded":"2024-03-21T14:23:08.170205+0000","last_fullsized":"2024-03-21T14:23:08.170205+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.461338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.461338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:02.086996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"446'9292","reported_seq":9442,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.786437+0000","last_change":"2024-03-21T14:22:33.276757+0000","last_active":"2024-03-21T14:23:17.786437+0000","last_peered":"2024-03-21T14:23:17.786437+0000","last_clean":"2024-03-21T14:23:17.786437+0000","last_became_active":"2024-03-21T14:20:59.109191+0000","last_became_peered":"2024-03-21T14:20:59.109191+0000","last_unstale":"2024-03-21T14:23:17.786437+0000","last_undegraded":"2024-03-21T14:23:17.786437+0000","last_fullsized":"2024-03-21T14:23:17.786437+0000","mapping_epoch":411,"log_start":"437'5200","ondisk_log_start":"437'5200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'6212","last_scrub_stamp":"2024-03-21T14:22:33.276373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:33.276373+0000","objects_scrubbed":6148,"log_size":4092,"log_dups_size":0,"ondisk_log_size":4092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:35.926896+0000","scrub_duration":2601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291897344,"num_objects":4454,"num_object_clones":0,"num_object_copies":13362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9292,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":813,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177909+0000","last_change":"2024-03-21T14:20:36.503605+0000","last_active":"2024-03-21T14:23:08.177909+0000","last_peered":"2024-03-21T14:23:08.177909+0000","last_clean":"2024-03-21T14:23:08.177909+0000","last_became_active":"2024-03-21T14:19:14.692919+0000","last_became_peered":"2024-03-21T14:19:14.692919+0000","last_unstale":"2024-03-21T14:23:08.177909+0000","last_undegraded":"2024-03-21T14:23:08.177909+0000","last_fullsized":"2024-03-21T14:23:08.177909+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:36.503498+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:36.503498+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":854,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.191507+0000","last_change":"2024-03-21T14:20:43.142589+0000","last_active":"2024-03-21T14:23:08.191507+0000","last_peered":"2024-03-21T14:23:08.191507+0000","last_clean":"2024-03-21T14:23:08.191507+0000","last_became_active":"2024-03-21T14:19:34.691121+0000","last_became_peered":"2024-03-21T14:19:34.691121+0000","last_unstale":"2024-03-21T14:23:08.191507+0000","last_undegraded":"2024-03-21T14:23:08.191507+0000","last_fullsized":"2024-03-21T14:23:08.191507+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:43.142495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:20:43.142495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1085,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.737554+0000","last_change":"2024-03-21T14:22:40.498837+0000","last_active":"2024-03-21T14:23:08.737554+0000","last_peered":"2024-03-21T14:23:08.737554+0000","last_clean":"2024-03-21T14:23:08.737554+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:23:08.737554+0000","last_undegraded":"2024-03-21T14:23:08.737554+0000","last_fullsized":"2024-03-21T14:23:08.737554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:22:40.498694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.498694+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:49.467479+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":128,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.198735+0000","last_change":"2024-03-21T14:21:54.066071+0000","last_active":"2024-03-21T14:23:08.198735+0000","last_peered":"2024-03-21T14:23:08.198735+0000","last_clean":"2024-03-21T14:23:08.198735+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:23:08.198735+0000","last_undegraded":"2024-03-21T14:23:08.198735+0000","last_fullsized":"2024-03-21T14:23:08.198735+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.065922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.065922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:13.357461+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'39","reported_seq":924,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190598+0000","last_change":"2024-03-21T14:21:19.723743+0000","last_active":"2024-03-21T14:23:08.190598+0000","last_peered":"2024-03-21T14:23:08.190598+0000","last_clean":"2024-03-21T14:23:08.190598+0000","last_became_active":"2024-03-21T14:21:19.713483+0000","last_became_peered":"2024-03-21T14:21:19.713483+0000","last_unstale":"2024-03-21T14:23:08.190598+0000","last_undegraded":"2024-03-21T14:23:08.190598+0000","last_fullsized":"2024-03-21T14:23:08.190598+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:20:42.035180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.035180+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"446'9419","reported_seq":9500,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.627167+0000","last_change":"2024-03-21T14:20:59.240921+0000","last_active":"2024-03-21T14:23:17.627167+0000","last_peered":"2024-03-21T14:23:17.627167+0000","last_clean":"2024-03-21T14:23:17.627167+0000","last_became_active":"2024-03-21T14:20:59.240641+0000","last_became_peered":"2024-03-21T14:20:59.240641+0000","last_unstale":"2024-03-21T14:23:17.627167+0000","last_undegraded":"2024-03-21T14:23:17.627167+0000","last_fullsized":"2024-03-21T14:23:17.627167+0000","mapping_epoch":411,"log_start":"437'5200","ondisk_log_start":"437'5200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4219,"log_dups_size":0,"ondisk_log_size":4219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:19.179573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297336832,"num_objects":4537,"num_object_clones":0,"num_object_copies":13611,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9419,"num_write_kb":446592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":839,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.185164+0000","last_change":"2024-03-21T14:22:39.460174+0000","last_active":"2024-03-21T14:23:08.185164+0000","last_peered":"2024-03-21T14:23:08.185164+0000","last_clean":"2024-03-21T14:23:08.185164+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:23:08.185164+0000","last_undegraded":"2024-03-21T14:23:08.185164+0000","last_fullsized":"2024-03-21T14:23:08.185164+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:39.460024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:39.460024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:02.211969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":829,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171559+0000","last_change":"2024-03-21T14:21:51.322736+0000","last_active":"2024-03-21T14:23:08.171559+0000","last_peered":"2024-03-21T14:23:08.171559+0000","last_clean":"2024-03-21T14:23:08.171559+0000","last_became_active":"2024-03-21T14:21:26.767687+0000","last_became_peered":"2024-03-21T14:21:26.767687+0000","last_unstale":"2024-03-21T14:23:08.171559+0000","last_undegraded":"2024-03-21T14:23:08.171559+0000","last_fullsized":"2024-03-21T14:23:08.171559+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.322658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.322658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:12.324516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":845,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170235+0000","last_change":"2024-03-21T14:22:31.537067+0000","last_active":"2024-03-21T14:23:08.170235+0000","last_peered":"2024-03-21T14:23:08.170235+0000","last_clean":"2024-03-21T14:23:08.170235+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:23:08.170235+0000","last_undegraded":"2024-03-21T14:23:08.170235+0000","last_fullsized":"2024-03-21T14:23:08.170235+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:31.536957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:31.536957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:41.005949+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'27","reported_seq":885,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171114+0000","last_change":"2024-03-21T14:18:32.414098+0000","last_active":"2024-03-21T14:23:08.171114+0000","last_peered":"2024-03-21T14:23:08.171114+0000","last_clean":"2024-03-21T14:23:08.171114+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:23:08.171114+0000","last_undegraded":"2024-03-21T14:23:08.171114+0000","last_fullsized":"2024-03-21T14:23:08.171114+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:18:32.413975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:32.413975+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"446'9246","reported_seq":9327,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:18.148232+0000","last_change":"2024-03-21T14:20:59.239976+0000","last_active":"2024-03-21T14:23:18.148232+0000","last_peered":"2024-03-21T14:23:18.148232+0000","last_clean":"2024-03-21T14:23:18.148232+0000","last_became_active":"2024-03-21T14:20:59.239396+0000","last_became_peered":"2024-03-21T14:20:59.239396+0000","last_unstale":"2024-03-21T14:23:18.148232+0000","last_undegraded":"2024-03-21T14:23:18.148232+0000","last_fullsized":"2024-03-21T14:23:18.148232+0000","mapping_epoch":411,"log_start":"436'4200","ondisk_log_start":"436'4200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5046,"log_dups_size":0,"ondisk_log_size":5046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291897344,"num_objects":4454,"num_object_clones":0,"num_object_copies":13362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9246,"num_write_kb":438400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":821,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.176779+0000","last_change":"2024-03-21T14:22:57.334699+0000","last_active":"2024-03-21T14:23:08.176779+0000","last_peered":"2024-03-21T14:23:08.176779+0000","last_clean":"2024-03-21T14:23:08.176779+0000","last_became_active":"2024-03-21T14:19:14.700353+0000","last_became_peered":"2024-03-21T14:19:14.700353+0000","last_unstale":"2024-03-21T14:23:08.176779+0000","last_undegraded":"2024-03-21T14:23:08.176779+0000","last_fullsized":"2024-03-21T14:23:08.176779+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:57.334563+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.334563+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.721763+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":632,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.178409+0000","last_change":"2024-03-21T14:20:45.611947+0000","last_active":"2024-03-21T14:23:08.178409+0000","last_peered":"2024-03-21T14:23:08.178409+0000","last_clean":"2024-03-21T14:23:08.178409+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:23:08.178409+0000","last_undegraded":"2024-03-21T14:23:08.178409+0000","last_fullsized":"2024-03-21T14:23:08.178409+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.611844+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":845,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.190735+0000","last_change":"2024-03-21T14:21:54.619575+0000","last_active":"2024-03-21T14:23:08.190735+0000","last_peered":"2024-03-21T14:23:08.190735+0000","last_clean":"2024-03-21T14:23:08.190735+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:23:08.190735+0000","last_undegraded":"2024-03-21T14:23:08.190735+0000","last_fullsized":"2024-03-21T14:23:08.190735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.617817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:08.722294+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":132,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.177505+0000","last_change":"2024-03-21T14:22:41.364253+0000","last_active":"2024-03-21T14:23:08.177505+0000","last_peered":"2024-03-21T14:23:08.177505+0000","last_clean":"2024-03-21T14:23:08.177505+0000","last_became_active":"2024-03-21T14:20:19.703587+0000","last_became_peered":"2024-03-21T14:20:19.703587+0000","last_unstale":"2024-03-21T14:23:08.177505+0000","last_undegraded":"2024-03-21T14:23:08.177505+0000","last_fullsized":"2024-03-21T14:23:08.177505+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:41.363995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:41.363995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:51.499840+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":895,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171163+0000","last_change":"2024-03-21T14:20:46.480561+0000","last_active":"2024-03-21T14:23:08.171163+0000","last_peered":"2024-03-21T14:23:08.171163+0000","last_clean":"2024-03-21T14:23:08.171163+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:23:08.171163+0000","last_undegraded":"2024-03-21T14:23:08.171163+0000","last_fullsized":"2024-03-21T14:23:08.171163+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:20:46.480457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.480457+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"446'9303","reported_seq":9450,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:17.108583+0000","last_change":"2024-03-21T14:22:59.010537+0000","last_active":"2024-03-21T14:23:17.108583+0000","last_peered":"2024-03-21T14:23:17.108583+0000","last_clean":"2024-03-21T14:23:17.108583+0000","last_became_active":"2024-03-21T14:20:59.386715+0000","last_became_peered":"2024-03-21T14:20:59.386715+0000","last_unstale":"2024-03-21T14:23:17.108583+0000","last_undegraded":"2024-03-21T14:23:17.108583+0000","last_fullsized":"2024-03-21T14:23:17.108583+0000","mapping_epoch":411,"log_start":"436'4716","ondisk_log_start":"436'4716","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'8088","last_scrub_stamp":"2024-03-21T14:22:59.010282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:59.010282+0000","objects_scrubbed":5843,"log_size":4587,"log_dups_size":0,"ondisk_log_size":4587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.160780+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299696128,"num_objects":4573,"num_object_clones":0,"num_object_copies":13719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9303,"num_write_kb":444032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1166,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.174639+0000","last_change":"2024-03-21T14:23:04.602240+0000","last_active":"2024-03-21T14:23:08.174639+0000","last_peered":"2024-03-21T14:23:08.174639+0000","last_clean":"2024-03-21T14:23:08.174639+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:23:08.174639+0000","last_undegraded":"2024-03-21T14:23:08.174639+0000","last_fullsized":"2024-03-21T14:23:08.174639+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:23:04.602105+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.602105+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:32.873082+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":841,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.171639+0000","last_change":"2024-03-21T14:19:15.689377+0000","last_active":"2024-03-21T14:23:08.171639+0000","last_peered":"2024-03-21T14:23:08.171639+0000","last_clean":"2024-03-21T14:23:08.171639+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:23:08.171639+0000","last_undegraded":"2024-03-21T14:23:08.171639+0000","last_fullsized":"2024-03-21T14:23:08.171639+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:13.493205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:13.493205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":851,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.740592+0000","last_change":"2024-03-21T14:20:49.535756+0000","last_active":"2024-03-21T14:23:08.740592+0000","last_peered":"2024-03-21T14:23:08.740592+0000","last_clean":"2024-03-21T14:23:08.740592+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:23:08.740592+0000","last_undegraded":"2024-03-21T14:23:08.740592+0000","last_fullsized":"2024-03-21T14:23:08.740592+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.535652+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:14.252178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":857,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.195394+0000","last_change":"2024-03-21T14:21:46.110127+0000","last_active":"2024-03-21T14:23:08.195394+0000","last_peered":"2024-03-21T14:23:08.195394+0000","last_clean":"2024-03-21T14:23:08.195394+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:23:08.195394+0000","last_undegraded":"2024-03-21T14:23:08.195394+0000","last_fullsized":"2024-03-21T14:23:08.195394+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.110008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.110008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:13.331279+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":900,"reported_epoch":446,"state":"active+clean","last_fresh":"2024-03-21T14:23:08.170882+0000","last_change":"2024-03-21T14:23:02.231025+0000","last_active":"2024-03-21T14:23:08.170882+0000","last_peered":"2024-03-21T14:23:08.170882+0000","last_clean":"2024-03-21T14:23:08.170882+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:23:08.170882+0000","last_undegraded":"2024-03-21T14:23:08.170882+0000","last_fullsized":"2024-03-21T14:23:08.170882+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:23:02.230890+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:23:02.230890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.516151+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4730585112,"num_objects":72184,"num_object_clones":0,"num_object_copies":216552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72184,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":149576,"num_write_kb":7096257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3767,"num_bytes_recovered":246874112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14241574912,"data_stored":14241562696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70734,"ondisk_log_size":70734,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1178,"num_read_kb":1173,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1542,"num_read_kb":1299,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":3812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1057,"ondisk_log_size":1057,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103421,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2646264,"kb_used_data":1762672,"kb_used_omap":1,"kb_used_meta":883582,"kb_avail":91098888,"statfs":{"total":95995035648,"available":93285261312,"internally_reserved":0,"allocated":1804976128,"data_stored":1804470884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":904788426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":351,"seq":1507533520948,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":693120,"kb_used_data":619768,"kb_used_omap":4,"kb_used_meta":73339,"kb_avail":93052032,"statfs":{"total":95995035648,"available":95285280768,"internally_reserved":0,"allocated":634642432,"data_stored":634191650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":75099812},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575133,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3481040,"kb_used_data":2593224,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":90264112,"statfs":{"total":95995035648,"available":92430450688,"internally_reserved":0,"allocated":2655461376,"data_stored":2654932147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738658,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2603848,"kb_used_data":1752192,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91141304,"statfs":{"total":95995035648,"available":93328695296,"internally_reserved":0,"allocated":1794244608,"data_stored":1793722204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598292018,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2706996,"kb_used_data":2371820,"kb_used_omap":7,"kb_used_meta":335160,"kb_avail":91038156,"statfs":{"total":95995035648,"available":93223071744,"internally_reserved":0,"allocated":2428743680,"data_stored":2428241595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":343203972},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182673,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3126996,"kb_used_data":2311116,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":90618156,"statfs":{"total":95995035648,"available":92792991744,"internally_reserved":0,"allocated":2366582784,"data_stored":2366048899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":835123655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019149,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2386444,"kb_used_data":1484804,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91358708,"statfs":{"total":95995035648,"available":93551316992,"internally_reserved":0,"allocated":1520439296,"data_stored":1519912797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692104,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2355524,"kb_used_data":1467772,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91389628,"statfs":{"total":95995035648,"available":93582979072,"internally_reserved":0,"allocated":1502998528,"data_stored":1502481392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1771110400,"data_stored":1771110400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2650341376,"data_stored":2650341376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2061897728,"data_stored":2061893656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2346778624,"data_stored":2346778624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1506742272,"data_stored":1506738200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1486684160,"data_stored":1486684160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":629805056,"data_stored":629800984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1788215296,"data_stored":1788215296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:23:26.499 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-21T14:23:26.520 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:26.622 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:26.668 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:26.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:26.755 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:23:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:23:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:23:26.755 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:26 smithi003 ceph-mon[29576]: pgmap v984: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T14:23:26.755 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:26 smithi003 ceph-mon[40510]: pgmap v984: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T14:23:26.770 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:26.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:26.924 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:26.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:26 smithi118 ceph-mon[33862]: pgmap v984: 158 pgs: 158 active+clean; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T14:23:26.961 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:26.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:27.063 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:27.106 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:27.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:27.236 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:27.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:27.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:27.365 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:27.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:27.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.496 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:27.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:27.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:27.636 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:27.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:27.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.790 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:27.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:27.817 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:23:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:23:27.891 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:27.910 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:27.919 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:27.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:27.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:28.021 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:28.051 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:28.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:28.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:28.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:28.284 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:28.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:28.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:28.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:28.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:28.560 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:28.588 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:23:28.590 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:28.632 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:23:28.678 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:23:28.692 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:28.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.727 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:28.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:28.730 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:23:28.768 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:23:28.808 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:23:28.827 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:28.849 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:23:28.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:28.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:28.892 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[29576]: pgmap v985: 158 pgs: 158 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[29576]: osdmap e449: 8 total, 8 up, 8 in 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/609691401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[40510]: pgmap v985: 158 pgs: 158 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[40510]: osdmap e449: 8 total, 8 up, 8 in 2024-03-21T14:23:28.893 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:28 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/609691401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:28.895 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:23:28.906 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:23:28.906 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:23:28.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:23:28.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:23:28.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:23:28.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:23:28.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:23:28.907 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:28 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:28 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:28 smithi118 ceph-mon[33862]: pgmap v985: 158 pgs: 158 active+clean; 4.0 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s 2024-03-21T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:28 smithi118 ceph-mon[33862]: osdmap e449: 8 total, 8 up, 8 in 2024-03-21T14:23:28.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:28 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/609691401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:28.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:29.012 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:29.114 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:29.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:29.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:29.264 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:23:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:29.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:29.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:29.410 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:29.512 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:29.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.559 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:29.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:29.660 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:29.692 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:29.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.837 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:29.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:29.939 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:29.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:30.070 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:30.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:30.182 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:30.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:30.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:30.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:30.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:30.431 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:30.486 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:30.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:30.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:30.631 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:30.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:30.732 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:30.768 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:30.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:30.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:30.870 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:30.902 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:30.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:31.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:31.031 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:31.110 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:30 smithi003 ceph-mon[29576]: pgmap v987: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-21T14:23:31.111 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:30 smithi003 ceph-mon[40510]: pgmap v987: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-21T14:23:31.133 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:31.161 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:30 smithi118 ceph-mon[33862]: pgmap v987: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s 2024-03-21T14:23:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:31.263 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:31.296 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:31.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:31.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:31.439 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:31.499 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:23:31.499 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:23:31.500 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-21T14:23:31.500 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:23:31.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:31.567 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:31.669 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:31.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:31.798 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:31.831 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:31.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:31.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:31.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:32.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:32.100 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:32.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.251 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:32.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:32.352 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:32.381 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:32.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:32.452 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:23:32.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:32.542 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:32.645 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:32.695 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:32.798 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.828 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:32.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:32.929 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:32.952 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:32.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:33.054 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:33.081 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:32 smithi118 ceph-mon[33862]: pgmap v988: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 683 op/s 2024-03-21T14:23:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:32 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:32 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:33.183 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.208 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:33.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[29576]: pgmap v988: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 683 op/s 2024-03-21T14:23:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[40510]: pgmap v988: 158 pgs: 158 active+clean; 3.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 683 op/s 2024-03-21T14:23:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:33.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.340 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:33.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:33.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:33.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:33.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:33.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:33.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:33.714 INFO:teuthology.orchestra.run.smithi003.stdout:ERROR: (22) Invalid argument 2024-03-21T14:23:33.715 INFO:teuthology.orchestra.run.smithi003.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:23:33.735 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:33.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:33.837 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:33.868 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:33.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:33.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:33.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:33 smithi003 ceph-mon[29576]: osdmap e450: 8 total, 8 up, 8 in 2024-03-21T14:23:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:33 smithi003 ceph-mon[40510]: osdmap e450: 8 total, 8 up, 8 in 2024-03-21T14:23:34.020 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:34.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:34.170 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:33 smithi118 ceph-mon[33862]: osdmap e450: 8 total, 8 up, 8 in 2024-03-21T14:23:34.272 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:34.309 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:34.411 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:34.450 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:34.552 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:34.591 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:34.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-21T14:23:34.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:34.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:34.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:34.850 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:34.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:34.872 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:34 smithi003 ceph-mon[29576]: pgmap v990: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-21T14:23:34.872 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:34 smithi003 ceph-mon[40510]: pgmap v990: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-21T14:23:34.952 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:35.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:35.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:35.129 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:34 smithi118 ceph-mon[33862]: pgmap v990: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s 2024-03-21T14:23:35.231 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:35.272 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:35.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:35.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.424 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:35.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:35.425 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:35.428 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:35.525 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:35.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:35.673 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:35.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:35.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:35.821 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:35.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.863 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:35.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:35.965 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:35.992 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:35.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:35.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:36.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:36.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:36.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:36.235 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.281 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:36.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:36.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:36.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:36.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:36.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.540 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:36.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:36.588 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:23:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:23:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:23:36.641 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:36.670 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:36.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:36.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:36.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:36.940 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:36.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:36.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:36 smithi003 ceph-mon[29576]: pgmap v991: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T14:23:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:36 smithi003 ceph-mon[40510]: pgmap v991: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T14:23:37.042 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:37.072 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:37.174 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:36 smithi118 ceph-mon[33862]: pgmap v991: 158 pgs: 158 active+clean; 3.4 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T14:23:37.212 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:37.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:37.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:37.353 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:37.455 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:37.498 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:37.600 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:37.630 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:37.732 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:37.759 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:37.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:37.860 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:37.890 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:37.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:37.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:37.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:23:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:23:37.993 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:38.023 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:38.125 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:38.153 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:38.255 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:38.285 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:38.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:38.386 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:38.429 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:38.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:38.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.563 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:38.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:38.664 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:38.701 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:38.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:38.850 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:38.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:38.983 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:38.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:38.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:39.085 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:39.115 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:38 smithi118 ceph-mon[33862]: pgmap v992: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:39.217 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:39.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:38 smithi003 ceph-mon[29576]: pgmap v992: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:39.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:39.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:38 smithi003 ceph-mon[40510]: pgmap v992: 158 pgs: 158 active+clean; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-21T14:23:39.252 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:39.256 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:39.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:39.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:39.387 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:39.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:39.514 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:39.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:39.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:39.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:39.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.813 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:39.914 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:39.944 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:39.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:40.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:40.083 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:40.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:40.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:40.334 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:40.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:40.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:40.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:40.506 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:40.608 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:40.637 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:40.739 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:40.765 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:40.867 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:40.895 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:40.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:40.996 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:41.021 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:40 smithi003 ceph-mon[40510]: pgmap v993: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:41.022 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:40 smithi003 ceph-mon[29576]: pgmap v993: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:41.026 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:41.128 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:41.168 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:41.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:40 smithi118 ceph-mon[33862]: pgmap v993: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:41.270 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:41.313 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:41.415 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:41.444 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:41.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:41.545 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:41.584 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:41.685 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:41.711 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:41.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:41.813 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:41.849 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:41.865 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:41.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:41.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:42.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:42.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:42.141 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:42.243 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:42.279 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:42.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:42.381 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:42.422 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:42.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:42.524 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:42.571 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:42.581 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:23:42.588 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:23:42.674 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:42.701 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:42.802 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:42.829 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:42.854 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:23:42.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:42.960 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:42.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:43.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:43.116 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:42 smithi118 ceph-mon[33862]: pgmap v994: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:42 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1002966749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:23:43.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:42 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:23:43.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:42 smithi003 ceph-mon[29576]: pgmap v994: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:43.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:42 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1002966749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:23:43.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:42 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:23:43.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:42 smithi003 ceph-mon[40510]: pgmap v994: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-21T14:23:43.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:42 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1002966749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:23:43.201 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:42 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:23:43.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:43.249 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:43.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:43.382 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:43.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:43.510 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:43.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:43.612 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:43.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:43.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:43.784 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:43.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:43.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:43.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[29576]: from='client.16071 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:23:44.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:23:44.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[29576]: osdmap e451: 8 total, 8 up, 8 in 2024-03-21T14:23:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[40510]: from='client.16071 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:23:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:23:44.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:43 smithi003 ceph-mon[40510]: osdmap e451: 8 total, 8 up, 8 in 2024-03-21T14:23:44.017 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:44.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:44.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:44.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:44.179 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:44.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:43 smithi118 ceph-mon[33862]: from='client.16071 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:43 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:23:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:43 smithi118 ceph-mon[33862]: osdmap e451: 8 total, 8 up, 8 in 2024-03-21T14:23:44.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:44.324 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:44.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:44.426 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:44.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.454 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:44.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:44.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:44.581 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:44.682 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:44.715 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:44.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:44.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:44.843 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:44.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:44.936 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":994,"stamp":"2024-03-21T14:23:41.361416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3249084864,"num_objects":49767,"num_object_clones":0,"num_object_copies":149301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49767,"num_whiteouts":0,"num_read":2758,"num_read_kb":2510,"num_write":173199,"num_write_kb":7096302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3952,"num_bytes_recovered":246878200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73245,"ondisk_log_size":73245,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":479,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15468680,"kb_used_data":9774856,"kb_used_omap":21,"kb_used_meta":5693354,"kb_avail":734492536,"statfs":{"total":767960285184,"available":752120356864,"internally_reserved":0,"allocated":10009452544,"data_stored":10006449616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22067,"internal_metadata":5829994957},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-747372544,"num_objects":-11404,"num_object_clones":0,"num_object_copies":-34212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11404,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11404,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.005466"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":831,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786833+0000","last_change":"2024-03-21T14:23:04.359116+0000","last_active":"2024-03-21T14:23:32.786833+0000","last_peered":"2024-03-21T14:23:32.786833+0000","last_clean":"2024-03-21T14:23:32.786833+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:23:32.786833+0000","last_undegraded":"2024-03-21T14:23:32.786833+0000","last_fullsized":"2024-03-21T14:23:32.786833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:04.358989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.358989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:15.061861+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":852,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790164+0000","last_change":"2024-03-21T14:22:40.829274+0000","last_active":"2024-03-21T14:23:32.790164+0000","last_peered":"2024-03-21T14:23:32.790164+0000","last_clean":"2024-03-21T14:23:32.790164+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:23:32.790164+0000","last_undegraded":"2024-03-21T14:23:32.790164+0000","last_fullsized":"2024-03-21T14:23:32.790164+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:40.829154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.829154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:43.086824+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":408,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794292+0000","last_change":"2024-03-21T14:23:06.316701+0000","last_active":"2024-03-21T14:23:32.794292+0000","last_peered":"2024-03-21T14:23:32.794292+0000","last_clean":"2024-03-21T14:23:32.794292+0000","last_became_active":"2024-03-21T14:19:34.695760+0000","last_became_peered":"2024-03-21T14:19:34.695760+0000","last_unstale":"2024-03-21T14:23:32.794292+0000","last_undegraded":"2024-03-21T14:23:32.794292+0000","last_fullsized":"2024-03-21T14:23:32.794292+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:06.316132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:06.316132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:29.219900+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":898,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786698+0000","last_change":"2024-03-21T14:20:41.426448+0000","last_active":"2024-03-21T14:23:32.786698+0000","last_peered":"2024-03-21T14:23:32.786698+0000","last_clean":"2024-03-21T14:23:32.786698+0000","last_became_active":"2024-03-21T14:20:21.782187+0000","last_became_peered":"2024-03-21T14:20:21.782187+0000","last_unstale":"2024-03-21T14:23:32.786698+0000","last_undegraded":"2024-03-21T14:23:32.786698+0000","last_fullsized":"2024-03-21T14:23:32.786698+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:20:41.426344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:41.426344+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":136,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793788+0000","last_change":"2024-03-21T14:23:07.529805+0000","last_active":"2024-03-21T14:23:32.793788+0000","last_peered":"2024-03-21T14:23:32.793788+0000","last_clean":"2024-03-21T14:23:32.793788+0000","last_became_active":"2024-03-21T14:20:21.782401+0000","last_became_peered":"2024-03-21T14:20:21.782401+0000","last_unstale":"2024-03-21T14:23:32.793788+0000","last_undegraded":"2024-03-21T14:23:32.793788+0000","last_fullsized":"2024-03-21T14:23:32.793788+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:07.529663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:07.529663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:22.958204+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":842,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786583+0000","last_change":"2024-03-21T14:20:52.477097+0000","last_active":"2024-03-21T14:23:32.786583+0000","last_peered":"2024-03-21T14:23:32.786583+0000","last_clean":"2024-03-21T14:23:32.786583+0000","last_became_active":"2024-03-21T14:19:14.689459+0000","last_became_peered":"2024-03-21T14:19:14.689459+0000","last_unstale":"2024-03-21T14:23:32.786583+0000","last_undegraded":"2024-03-21T14:23:32.786583+0000","last_fullsized":"2024-03-21T14:23:32.786583+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:52.477009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.477009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":853,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791869+0000","last_change":"2024-03-21T14:20:37.939217+0000","last_active":"2024-03-21T14:23:32.791869+0000","last_peered":"2024-03-21T14:23:32.791869+0000","last_clean":"2024-03-21T14:23:32.791869+0000","last_became_active":"2024-03-21T14:19:14.695703+0000","last_became_peered":"2024-03-21T14:19:14.695703+0000","last_unstale":"2024-03-21T14:23:32.791869+0000","last_undegraded":"2024-03-21T14:23:32.791869+0000","last_fullsized":"2024-03-21T14:23:32.791869+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:37.939080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.939080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.103010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'46","reported_seq":909,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793317+0000","last_change":"2024-03-21T14:20:40.919011+0000","last_active":"2024-03-21T14:23:32.793317+0000","last_peered":"2024-03-21T14:23:32.793317+0000","last_clean":"2024-03-21T14:23:32.793317+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:23:32.793317+0000","last_undegraded":"2024-03-21T14:23:32.793317+0000","last_fullsized":"2024-03-21T14:23:32.793317+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:20:40.918895+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:20:40.918895+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.104247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":712,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789816+0000","last_change":"2024-03-21T14:22:57.984639+0000","last_active":"2024-03-21T14:23:32.789816+0000","last_peered":"2024-03-21T14:23:32.789816+0000","last_clean":"2024-03-21T14:23:32.789816+0000","last_became_active":"2024-03-21T14:19:34.690300+0000","last_became_peered":"2024-03-21T14:19:34.690300+0000","last_unstale":"2024-03-21T14:23:32.789816+0000","last_undegraded":"2024-03-21T14:23:32.789816+0000","last_fullsized":"2024-03-21T14:23:32.789816+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:57.984394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.984394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.446523+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.783456+0000","last_change":"2024-03-21T14:22:48.510243+0000","last_active":"2024-03-21T14:23:32.783456+0000","last_peered":"2024-03-21T14:23:32.783456+0000","last_clean":"2024-03-21T14:23:32.783456+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:23:32.783456+0000","last_undegraded":"2024-03-21T14:23:32.783456+0000","last_fullsized":"2024-03-21T14:23:32.783456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:48.510139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.510139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:13.480887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":651,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787422+0000","last_change":"2024-03-21T14:22:54.329383+0000","last_active":"2024-03-21T14:23:32.787422+0000","last_peered":"2024-03-21T14:23:32.787422+0000","last_clean":"2024-03-21T14:23:32.787422+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:23:32.787422+0000","last_undegraded":"2024-03-21T14:23:32.787422+0000","last_fullsized":"2024-03-21T14:23:32.787422+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.329270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.329270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:10.420864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"441'21","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786939+0000","last_change":"2024-03-21T14:22:48.380073+0000","last_active":"2024-03-21T14:23:32.786939+0000","last_peered":"2024-03-21T14:23:32.786939+0000","last_clean":"2024-03-21T14:23:32.786939+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:23:32.786939+0000","last_undegraded":"2024-03-21T14:23:32.786939+0000","last_fullsized":"2024-03-21T14:23:32.786939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:22:48.379990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.379990+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:05.754736+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":802,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790405+0000","last_change":"2024-03-21T14:22:57.054038+0000","last_active":"2024-03-21T14:23:32.790405+0000","last_peered":"2024-03-21T14:23:32.790405+0000","last_clean":"2024-03-21T14:23:32.790405+0000","last_became_active":"2024-03-21T14:20:21.776360+0000","last_became_peered":"2024-03-21T14:20:21.776360+0000","last_unstale":"2024-03-21T14:23:32.790405+0000","last_undegraded":"2024-03-21T14:23:32.790405+0000","last_fullsized":"2024-03-21T14:23:32.790405+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:57.053921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.053921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.452840+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":852,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790641+0000","last_change":"2024-03-21T14:21:47.782509+0000","last_active":"2024-03-21T14:23:32.790641+0000","last_peered":"2024-03-21T14:23:32.790641+0000","last_clean":"2024-03-21T14:23:32.790641+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:23:32.790641+0000","last_undegraded":"2024-03-21T14:23:32.790641+0000","last_fullsized":"2024-03-21T14:23:32.790641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:47.782401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.782401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":317,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787623+0000","last_change":"2024-03-21T14:21:53.335510+0000","last_active":"2024-03-21T14:23:32.787623+0000","last_peered":"2024-03-21T14:23:32.787623+0000","last_clean":"2024-03-21T14:23:32.787623+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:23:32.787623+0000","last_undegraded":"2024-03-21T14:23:32.787623+0000","last_fullsized":"2024-03-21T14:23:32.787623+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_clean_scrub_stamp":"2024-03-21T14:21:53.335395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"445'10","reported_seq":856,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787015+0000","last_change":"2024-03-21T14:22:57.290034+0000","last_active":"2024-03-21T14:23:32.787015+0000","last_peered":"2024-03-21T14:23:32.787015+0000","last_clean":"2024-03-21T14:23:32.787015+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:23:32.787015+0000","last_undegraded":"2024-03-21T14:23:32.787015+0000","last_fullsized":"2024-03-21T14:23:32.787015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'10","last_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_deep_scrub":"445'10","last_deep_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.289894+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.557256+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":831,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787419+0000","last_change":"2024-03-21T14:22:32.629084+0000","last_active":"2024-03-21T14:23:32.787419+0000","last_peered":"2024-03-21T14:23:32.787419+0000","last_clean":"2024-03-21T14:23:32.787419+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:23:32.787419+0000","last_undegraded":"2024-03-21T14:23:32.787419+0000","last_fullsized":"2024-03-21T14:23:32.787419+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:32.628950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:32.628950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:35.170470+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":844,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789762+0000","last_change":"2024-03-21T14:22:35.833709+0000","last_active":"2024-03-21T14:23:32.789762+0000","last_peered":"2024-03-21T14:23:32.789762+0000","last_clean":"2024-03-21T14:23:32.789762+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:23:32.789762+0000","last_undegraded":"2024-03-21T14:23:32.789762+0000","last_fullsized":"2024-03-21T14:23:32.789762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:35.833457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.833457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:03.246243+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":694,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787002+0000","last_change":"2024-03-21T14:19:34.686397+0000","last_active":"2024-03-21T14:23:32.787002+0000","last_peered":"2024-03-21T14:23:32.787002+0000","last_clean":"2024-03-21T14:23:32.787002+0000","last_became_active":"2024-03-21T14:19:34.686106+0000","last_became_peered":"2024-03-21T14:19:34.686106+0000","last_unstale":"2024-03-21T14:23:32.787002+0000","last_undegraded":"2024-03-21T14:23:32.787002+0000","last_fullsized":"2024-03-21T14:23:32.787002+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:33.422232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:18:33.422232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":775,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793595+0000","last_change":"2024-03-21T14:21:50.323084+0000","last_active":"2024-03-21T14:23:32.793595+0000","last_peered":"2024-03-21T14:23:32.793595+0000","last_clean":"2024-03-21T14:23:32.793595+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:23:32.793595+0000","last_undegraded":"2024-03-21T14:23:32.793595+0000","last_fullsized":"2024-03-21T14:23:32.793595+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'34","last_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_deep_scrub":"432'34","last_deep_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.322580+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139575+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":848,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791083+0000","last_change":"2024-03-21T14:21:33.288900+0000","last_active":"2024-03-21T14:23:32.791083+0000","last_peered":"2024-03-21T14:23:32.791083+0000","last_clean":"2024-03-21T14:23:32.791083+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:23:32.791083+0000","last_undegraded":"2024-03-21T14:23:32.791083+0000","last_fullsized":"2024-03-21T14:23:32.791083+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:33.288792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:33.288792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105174+0000","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":836,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787830+0000","last_change":"2024-03-21T14:22:29.551806+0000","last_active":"2024-03-21T14:23:32.787830+0000","last_peered":"2024-03-21T14:23:32.787830+0000","last_clean":"2024-03-21T14:23:32.787830+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:23:32.787830+0000","last_undegraded":"2024-03-21T14:23:32.787830+0000","last_fullsized":"2024-03-21T14:23:32.787830+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:29.551666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.551666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:59.532498+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":836,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787011+0000","last_change":"2024-03-21T14:20:31.407178+0000","last_active":"2024-03-21T14:23:32.787011+0000","last_peered":"2024-03-21T14:23:32.787011+0000","last_clean":"2024-03-21T14:23:32.787011+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:23:32.787011+0000","last_undegraded":"2024-03-21T14:23:32.787011+0000","last_fullsized":"2024-03-21T14:23:32.787011+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:31.407065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:20:31.407065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":889,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792311+0000","last_change":"2024-03-21T14:22:35.357291+0000","last_active":"2024-03-21T14:23:32.792311+0000","last_peered":"2024-03-21T14:23:32.792311+0000","last_clean":"2024-03-21T14:23:32.792311+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:23:32.792311+0000","last_undegraded":"2024-03-21T14:23:32.792311+0000","last_fullsized":"2024-03-21T14:23:32.792311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:22:35.357134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.357134+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:38.436579+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":351,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793259+0000","last_change":"2024-03-21T14:21:19.290180+0000","last_active":"2024-03-21T14:23:32.793259+0000","last_peered":"2024-03-21T14:23:32.793259+0000","last_clean":"2024-03-21T14:23:32.793259+0000","last_became_active":"2024-03-21T14:19:14.686523+0000","last_became_peered":"2024-03-21T14:19:14.686523+0000","last_unstale":"2024-03-21T14:23:32.793259+0000","last_undegraded":"2024-03-21T14:23:32.793259+0000","last_fullsized":"2024-03-21T14:23:32.793259+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:21:19.289804+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.289804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139466+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"425'15","reported_seq":111,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:34.714451+0000","last_change":"2024-03-21T14:22:44.412788+0000","last_active":"2024-03-21T14:23:34.714451+0000","last_peered":"2024-03-21T14:23:34.714451+0000","last_clean":"2024-03-21T14:23:34.714451+0000","last_became_active":"2024-03-21T14:21:26.767116+0000","last_became_peered":"2024-03-21T14:21:26.767116+0000","last_unstale":"2024-03-21T14:23:34.714451+0000","last_undegraded":"2024-03-21T14:23:34.714451+0000","last_fullsized":"2024-03-21T14:23:34.714451+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'15","last_scrub_stamp":"2024-03-21T14:22:44.412659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:44.412659+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:54.694196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789998+0000","last_change":"2024-03-21T14:20:37.151817+0000","last_active":"2024-03-21T14:23:32.789998+0000","last_peered":"2024-03-21T14:23:32.789998+0000","last_clean":"2024-03-21T14:23:32.789998+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:23:32.789998+0000","last_undegraded":"2024-03-21T14:23:32.789998+0000","last_fullsized":"2024-03-21T14:23:32.789998+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:20:37.151721+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.151721+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":581,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792189+0000","last_change":"2024-03-21T14:21:19.102404+0000","last_active":"2024-03-21T14:23:32.792189+0000","last_peered":"2024-03-21T14:23:32.792189+0000","last_clean":"2024-03-21T14:23:32.792189+0000","last_became_active":"2024-03-21T14:20:21.790051+0000","last_became_peered":"2024-03-21T14:20:21.790051+0000","last_unstale":"2024-03-21T14:23:32.792189+0000","last_undegraded":"2024-03-21T14:23:32.792189+0000","last_fullsized":"2024-03-21T14:23:32.792189+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-03-21T14:21:19.102261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.102261+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":840,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789437+0000","last_change":"2024-03-21T14:21:19.457230+0000","last_active":"2024-03-21T14:23:32.789437+0000","last_peered":"2024-03-21T14:23:32.789437+0000","last_clean":"2024-03-21T14:23:32.789437+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:23:32.789437+0000","last_undegraded":"2024-03-21T14:23:32.789437+0000","last_fullsized":"2024-03-21T14:23:32.789437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:19.457101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.457101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":134,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":508,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792104+0000","last_change":"2024-03-21T14:22:58.625867+0000","last_active":"2024-03-21T14:23:32.792104+0000","last_peered":"2024-03-21T14:23:32.792104+0000","last_clean":"2024-03-21T14:23:32.792104+0000","last_became_active":"2024-03-21T14:19:14.675463+0000","last_became_peered":"2024-03-21T14:19:14.675463+0000","last_unstale":"2024-03-21T14:23:32.792104+0000","last_undegraded":"2024-03-21T14:23:32.792104+0000","last_fullsized":"2024-03-21T14:23:32.792104+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:58.625708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:22:58.625708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.285980+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791915+0000","last_change":"2024-03-21T14:20:45.936416+0000","last_active":"2024-03-21T14:23:32.791915+0000","last_peered":"2024-03-21T14:23:32.791915+0000","last_clean":"2024-03-21T14:23:32.791915+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:23:32.791915+0000","last_undegraded":"2024-03-21T14:23:32.791915+0000","last_fullsized":"2024-03-21T14:23:32.791915+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.936318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.936318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.104302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'32","reported_seq":588,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790467+0000","last_change":"2024-03-21T14:20:46.036522+0000","last_active":"2024-03-21T14:23:32.790467+0000","last_peered":"2024-03-21T14:23:32.790467+0000","last_clean":"2024-03-21T14:23:32.790467+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:23:32.790467+0000","last_undegraded":"2024-03-21T14:23:32.790467+0000","last_fullsized":"2024-03-21T14:23:32.790467+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_deep_scrub":"237'24","last_deep_scrub_stamp":"2024-03-21T14:20:46.036377+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.036377+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":206,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789254+0000","last_change":"2024-03-21T14:22:40.390363+0000","last_active":"2024-03-21T14:23:32.789254+0000","last_peered":"2024-03-21T14:23:32.789254+0000","last_clean":"2024-03-21T14:23:32.789254+0000","last_became_active":"2024-03-21T14:19:14.679790+0000","last_became_peered":"2024-03-21T14:19:14.679790+0000","last_unstale":"2024-03-21T14:23:32.789254+0000","last_undegraded":"2024-03-21T14:23:32.789254+0000","last_fullsized":"2024-03-21T14:23:32.789254+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:40.390274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.390274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:05.109596+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":517,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786587+0000","last_change":"2024-03-21T14:18:42.513377+0000","last_active":"2024-03-21T14:23:32.786587+0000","last_peered":"2024-03-21T14:23:32.786587+0000","last_clean":"2024-03-21T14:23:32.786587+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:23:32.786587+0000","last_undegraded":"2024-03-21T14:23:32.786587+0000","last_fullsized":"2024-03-21T14:23:32.786587+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:18:42.513167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":320,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791265+0000","last_change":"2024-03-21T14:21:21.656390+0000","last_active":"2024-03-21T14:23:32.791265+0000","last_peered":"2024-03-21T14:23:32.791265+0000","last_clean":"2024-03-21T14:23:32.791265+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:23:32.791265+0000","last_undegraded":"2024-03-21T14:23:32.791265+0000","last_fullsized":"2024-03-21T14:23:32.791265+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:21.656278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:21.656278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.196256+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"441'14","reported_seq":889,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792168+0000","last_change":"2024-03-21T14:22:49.297359+0000","last_active":"2024-03-21T14:23:32.792168+0000","last_peered":"2024-03-21T14:23:32.792168+0000","last_clean":"2024-03-21T14:23:32.792168+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:23:32.792168+0000","last_undegraded":"2024-03-21T14:23:32.792168+0000","last_fullsized":"2024-03-21T14:23:32.792168+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'14","last_scrub_stamp":"2024-03-21T14:22:49.297239+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.297239+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:58.065833+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":864,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794376+0000","last_change":"2024-03-21T14:21:34.144597+0000","last_active":"2024-03-21T14:23:32.794376+0000","last_peered":"2024-03-21T14:23:32.794376+0000","last_clean":"2024-03-21T14:23:32.794376+0000","last_became_active":"2024-03-21T14:19:34.693279+0000","last_became_peered":"2024-03-21T14:19:34.693279+0000","last_unstale":"2024-03-21T14:23:32.794376+0000","last_undegraded":"2024-03-21T14:23:32.794376+0000","last_fullsized":"2024-03-21T14:23:32.794376+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.144484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.144484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105143+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":875,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790924+0000","last_change":"2024-03-21T14:21:51.742321+0000","last_active":"2024-03-21T14:23:32.790924+0000","last_peered":"2024-03-21T14:23:32.790924+0000","last_clean":"2024-03-21T14:23:32.790924+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:23:32.790924+0000","last_undegraded":"2024-03-21T14:23:32.790924+0000","last_fullsized":"2024-03-21T14:23:32.790924+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.742159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.742159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":322,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791556+0000","last_change":"2024-03-21T14:23:04.813785+0000","last_active":"2024-03-21T14:23:32.791556+0000","last_peered":"2024-03-21T14:23:32.791556+0000","last_clean":"2024-03-21T14:23:32.791556+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:23:32.791556+0000","last_undegraded":"2024-03-21T14:23:32.791556+0000","last_fullsized":"2024-03-21T14:23:32.791556+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:23:04.813659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.813659+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.008687+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"441'32","reported_seq":923,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793260+0000","last_change":"2024-03-21T14:22:51.290072+0000","last_active":"2024-03-21T14:23:32.793260+0000","last_peered":"2024-03-21T14:23:32.793260+0000","last_clean":"2024-03-21T14:23:32.793260+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:23:32.793260+0000","last_undegraded":"2024-03-21T14:23:32.793260+0000","last_fullsized":"2024-03-21T14:23:32.793260+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:22:51.289757+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.289757+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:20.191203+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":329,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789660+0000","last_change":"2024-03-21T14:23:03.865434+0000","last_active":"2024-03-21T14:23:32.789660+0000","last_peered":"2024-03-21T14:23:32.789660+0000","last_clean":"2024-03-21T14:23:32.789660+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:23:32.789660+0000","last_undegraded":"2024-03-21T14:23:32.789660+0000","last_fullsized":"2024-03-21T14:23:32.789660+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:03.865222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:03.865222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.684378+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":844,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794280+0000","last_change":"2024-03-21T14:21:41.085311+0000","last_active":"2024-03-21T14:23:32.794280+0000","last_peered":"2024-03-21T14:23:32.794280+0000","last_clean":"2024-03-21T14:23:32.794280+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:23:32.794280+0000","last_undegraded":"2024-03-21T14:23:32.794280+0000","last_fullsized":"2024-03-21T14:23:32.794280+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:41.085192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.085192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":868,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790965+0000","last_change":"2024-03-21T14:22:53.685161+0000","last_active":"2024-03-21T14:23:32.790965+0000","last_peered":"2024-03-21T14:23:32.790965+0000","last_clean":"2024-03-21T14:23:32.790965+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:23:32.790965+0000","last_undegraded":"2024-03-21T14:23:32.790965+0000","last_fullsized":"2024-03-21T14:23:32.790965+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:53.685078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:53.685078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.209520+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'21","reported_seq":908,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791105+0000","last_change":"2024-03-21T14:23:00.867969+0000","last_active":"2024-03-21T14:23:32.791105+0000","last_peered":"2024-03-21T14:23:32.791105+0000","last_clean":"2024-03-21T14:23:32.791105+0000","last_became_active":"2024-03-21T14:19:14.682980+0000","last_became_peered":"2024-03-21T14:19:14.682980+0000","last_unstale":"2024-03-21T14:23:32.791105+0000","last_undegraded":"2024-03-21T14:23:32.791105+0000","last_fullsized":"2024-03-21T14:23:32.791105+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:23:00.867800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:00.867800+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.912184+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":136,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792733+0000","last_change":"2024-03-21T14:20:42.950594+0000","last_active":"2024-03-21T14:23:32.792733+0000","last_peered":"2024-03-21T14:23:32.792733+0000","last_clean":"2024-03-21T14:23:32.792733+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:23:32.792733+0000","last_undegraded":"2024-03-21T14:23:32.792733+0000","last_fullsized":"2024-03-21T14:23:32.792733+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.950500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.950500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.103036+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790911+0000","last_change":"2024-03-21T14:21:36.498785+0000","last_active":"2024-03-21T14:23:32.790911+0000","last_peered":"2024-03-21T14:23:32.790911+0000","last_clean":"2024-03-21T14:23:32.790911+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:23:32.790911+0000","last_undegraded":"2024-03-21T14:23:32.790911+0000","last_fullsized":"2024-03-21T14:23:32.790911+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:36.498684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:36.498684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.673448+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":859,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787687+0000","last_change":"2024-03-21T14:21:20.339156+0000","last_active":"2024-03-21T14:23:32.787687+0000","last_peered":"2024-03-21T14:23:32.787687+0000","last_clean":"2024-03-21T14:23:32.787687+0000","last_became_active":"2024-03-21T14:19:14.690097+0000","last_became_peered":"2024-03-21T14:19:14.690097+0000","last_unstale":"2024-03-21T14:23:32.787687+0000","last_undegraded":"2024-03-21T14:23:32.787687+0000","last_fullsized":"2024-03-21T14:23:32.787687+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:20.339046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.339046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":950,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787204+0000","last_change":"2024-03-21T14:21:50.314232+0000","last_active":"2024-03-21T14:23:32.787204+0000","last_peered":"2024-03-21T14:23:32.787204+0000","last_clean":"2024-03-21T14:23:32.787204+0000","last_became_active":"2024-03-21T14:21:26.767846+0000","last_became_peered":"2024-03-21T14:21:26.767846+0000","last_unstale":"2024-03-21T14:23:32.787204+0000","last_undegraded":"2024-03-21T14:23:32.787204+0000","last_fullsized":"2024-03-21T14:23:32.787204+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:21:50.314134+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.314134+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":725,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792015+0000","last_change":"2024-03-21T14:21:46.894626+0000","last_active":"2024-03-21T14:23:32.792015+0000","last_peered":"2024-03-21T14:23:32.792015+0000","last_clean":"2024-03-21T14:23:32.792015+0000","last_became_active":"2024-03-21T14:20:21.784416+0000","last_became_peered":"2024-03-21T14:20:21.784416+0000","last_unstale":"2024-03-21T14:23:32.792015+0000","last_undegraded":"2024-03-21T14:23:32.792015+0000","last_fullsized":"2024-03-21T14:23:32.792015+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.894547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.894547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":1079,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:33.746503+0000","last_change":"2024-03-21T14:21:50.631849+0000","last_active":"2024-03-21T14:23:33.746503+0000","last_peered":"2024-03-21T14:23:33.746503+0000","last_clean":"2024-03-21T14:23:33.746503+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:23:33.746503+0000","last_undegraded":"2024-03-21T14:23:33.746503+0000","last_fullsized":"2024-03-21T14:23:33.746503+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'10","last_scrub_stamp":"2024-03-21T14:21:50.631459+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.631459+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.673772+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":863,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790965+0000","last_change":"2024-03-21T14:21:34.062701+0000","last_active":"2024-03-21T14:23:32.790965+0000","last_peered":"2024-03-21T14:23:32.790965+0000","last_clean":"2024-03-21T14:23:32.790965+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:23:32.790965+0000","last_undegraded":"2024-03-21T14:23:32.790965+0000","last_fullsized":"2024-03-21T14:23:32.790965+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.062588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.062588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'35","reported_seq":890,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791638+0000","last_change":"2024-03-21T14:21:18.712180+0000","last_active":"2024-03-21T14:23:32.791638+0000","last_peered":"2024-03-21T14:23:32.791638+0000","last_clean":"2024-03-21T14:23:32.791638+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:23:32.791638+0000","last_undegraded":"2024-03-21T14:23:32.791638+0000","last_fullsized":"2024-03-21T14:23:32.791638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:18.712011+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:21:18.712011+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105100+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":824,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787527+0000","last_change":"2024-03-21T14:20:38.602527+0000","last_active":"2024-03-21T14:23:32.787527+0000","last_peered":"2024-03-21T14:23:32.787527+0000","last_clean":"2024-03-21T14:23:32.787527+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:23:32.787527+0000","last_undegraded":"2024-03-21T14:23:32.787527+0000","last_fullsized":"2024-03-21T14:23:32.787527+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:38.602405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:20:38.602405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":179,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791837+0000","last_change":"2024-03-21T14:20:44.917571+0000","last_active":"2024-03-21T14:23:32.791837+0000","last_peered":"2024-03-21T14:23:32.791837+0000","last_clean":"2024-03-21T14:23:32.791837+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:23:32.791837+0000","last_undegraded":"2024-03-21T14:23:32.791837+0000","last_fullsized":"2024-03-21T14:23:32.791837+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:20:44.917480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.917480+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":136,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793817+0000","last_change":"2024-03-21T14:22:43.375315+0000","last_active":"2024-03-21T14:23:32.793817+0000","last_peered":"2024-03-21T14:23:32.793817+0000","last_clean":"2024-03-21T14:23:32.793817+0000","last_became_active":"2024-03-21T14:20:21.784349+0000","last_became_peered":"2024-03-21T14:20:21.784349+0000","last_unstale":"2024-03-21T14:23:32.793817+0000","last_undegraded":"2024-03-21T14:23:32.793817+0000","last_fullsized":"2024-03-21T14:23:32.793817+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.375045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.375045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:00.172598+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"441'25","reported_seq":746,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791250+0000","last_change":"2024-03-21T14:21:37.034754+0000","last_active":"2024-03-21T14:23:32.791250+0000","last_peered":"2024-03-21T14:23:32.791250+0000","last_clean":"2024-03-21T14:23:32.791250+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:23:32.791250+0000","last_undegraded":"2024-03-21T14:23:32.791250+0000","last_fullsized":"2024-03-21T14:23:32.791250+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"338'19","last_scrub_stamp":"2024-03-21T14:21:37.034609+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:21:37.034609+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":681,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793850+0000","last_change":"2024-03-21T14:21:49.278092+0000","last_active":"2024-03-21T14:23:32.793850+0000","last_peered":"2024-03-21T14:23:32.793850+0000","last_clean":"2024-03-21T14:23:32.793850+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:23:32.793850+0000","last_undegraded":"2024-03-21T14:23:32.793850+0000","last_fullsized":"2024-03-21T14:23:32.793850+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:49.277585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.277585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139601+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":128,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791391+0000","last_change":"2024-03-21T14:20:21.787792+0000","last_active":"2024-03-21T14:23:32.791391+0000","last_peered":"2024-03-21T14:23:32.791391+0000","last_clean":"2024-03-21T14:23:32.791391+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:23:32.791391+0000","last_undegraded":"2024-03-21T14:23:32.791391+0000","last_fullsized":"2024-03-21T14:23:32.791391+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_clean_scrub_stamp":"2024-03-21T14:19:47.399758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.102923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":837,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786993+0000","last_change":"2024-03-21T14:22:39.540986+0000","last_active":"2024-03-21T14:23:32.786993+0000","last_peered":"2024-03-21T14:23:32.786993+0000","last_clean":"2024-03-21T14:23:32.786993+0000","last_became_active":"2024-03-21T14:21:26.767611+0000","last_became_peered":"2024-03-21T14:21:26.767611+0000","last_unstale":"2024-03-21T14:23:32.786993+0000","last_undegraded":"2024-03-21T14:23:32.786993+0000","last_fullsized":"2024-03-21T14:23:32.786993+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:39.540852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:39.540852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:44.126115+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"441'25","reported_seq":554,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792058+0000","last_change":"2024-03-21T14:21:49.128441+0000","last_active":"2024-03-21T14:23:32.792058+0000","last_peered":"2024-03-21T14:23:32.792058+0000","last_clean":"2024-03-21T14:23:32.792058+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:23:32.792058+0000","last_undegraded":"2024-03-21T14:23:32.792058+0000","last_fullsized":"2024-03-21T14:23:32.792058+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'19","last_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_deep_scrub":"432'19","last_deep_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.128273+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:35.151593+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":862,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790922+0000","last_change":"2024-03-21T14:22:43.886872+0000","last_active":"2024-03-21T14:23:32.790922+0000","last_peered":"2024-03-21T14:23:32.790922+0000","last_clean":"2024-03-21T14:23:32.790922+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:23:32.790922+0000","last_undegraded":"2024-03-21T14:23:32.790922+0000","last_fullsized":"2024-03-21T14:23:32.790922+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.886754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.886754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.618770+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":319,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792005+0000","last_change":"2024-03-21T14:21:26.752265+0000","last_active":"2024-03-21T14:23:32.792005+0000","last_peered":"2024-03-21T14:23:32.792005+0000","last_clean":"2024-03-21T14:23:32.792005+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:23:32.792005+0000","last_undegraded":"2024-03-21T14:23:32.792005+0000","last_fullsized":"2024-03-21T14:23:32.792005+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:26.752126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:21:26.752126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.196290+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":79,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.800757+0000","last_change":"2024-03-21T14:22:54.477723+0000","last_active":"2024-03-21T14:23:32.800757+0000","last_peered":"2024-03-21T14:23:32.800757+0000","last_clean":"2024-03-21T14:23:32.800757+0000","last_became_active":"2024-03-21T14:21:26.765914+0000","last_became_peered":"2024-03-21T14:21:26.765914+0000","last_unstale":"2024-03-21T14:23:32.800757+0000","last_undegraded":"2024-03-21T14:23:32.800757+0000","last_fullsized":"2024-03-21T14:23:32.800757+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.477581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.477581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:04.538947+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'18","reported_seq":145,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790872+0000","last_change":"2024-03-21T14:20:21.796765+0000","last_active":"2024-03-21T14:23:32.790872+0000","last_peered":"2024-03-21T14:23:32.790872+0000","last_clean":"2024-03-21T14:23:32.790872+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:23:32.790872+0000","last_undegraded":"2024-03-21T14:23:32.790872+0000","last_fullsized":"2024-03-21T14:23:32.790872+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'12","last_scrub_stamp":"2024-03-21T14:19:32.302574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:19:32.302574+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.102896+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"450'10953","reported_seq":11042,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.792160+0000","last_change":"2024-03-21T14:20:59.387014+0000","last_active":"2024-03-21T14:23:37.792160+0000","last_peered":"2024-03-21T14:23:37.792160+0000","last_clean":"2024-03-21T14:23:37.792160+0000","last_became_active":"2024-03-21T14:20:59.386663+0000","last_became_peered":"2024-03-21T14:20:59.386663+0000","last_unstale":"2024-03-21T14:23:37.792160+0000","last_undegraded":"2024-03-21T14:23:37.792160+0000","last_fullsized":"2024-03-21T14:23:37.792160+0000","mapping_epoch":411,"log_start":"441'6900","ondisk_log_start":"441'6900","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4053,"log_dups_size":0,"ondisk_log_size":4053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.673403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209518592,"num_objects":3197,"num_object_clones":0,"num_object_copies":9591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10953,"num_write_kb":452800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":726,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790057+0000","last_change":"2024-03-21T14:22:52.940837+0000","last_active":"2024-03-21T14:23:32.790057+0000","last_peered":"2024-03-21T14:23:32.790057+0000","last_clean":"2024-03-21T14:23:32.790057+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:23:32.790057+0000","last_undegraded":"2024-03-21T14:23:32.790057+0000","last_fullsized":"2024-03-21T14:23:32.790057+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_clean_scrub_stamp":"2024-03-21T14:22:52.940702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:19.596185+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":899,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791212+0000","last_change":"2024-03-21T14:21:41.979701+0000","last_active":"2024-03-21T14:23:32.791212+0000","last_peered":"2024-03-21T14:23:32.791212+0000","last_clean":"2024-03-21T14:23:32.791212+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:23:32.791212+0000","last_undegraded":"2024-03-21T14:23:32.791212+0000","last_fullsized":"2024-03-21T14:23:32.791212+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:21:41.979582+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.979582+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":799,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790602+0000","last_change":"2024-03-21T14:22:37.805371+0000","last_active":"2024-03-21T14:23:32.790602+0000","last_peered":"2024-03-21T14:23:32.790602+0000","last_clean":"2024-03-21T14:23:32.790602+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:23:32.790602+0000","last_undegraded":"2024-03-21T14:23:32.790602+0000","last_fullsized":"2024-03-21T14:23:32.790602+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:22:37.805239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:37.805239+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:56.435148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"372'10","reported_seq":1125,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790845+0000","last_change":"2024-03-21T14:21:35.014048+0000","last_active":"2024-03-21T14:23:32.790845+0000","last_peered":"2024-03-21T14:23:32.790845+0000","last_clean":"2024-03-21T14:23:32.790845+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:23:32.790845+0000","last_undegraded":"2024-03-21T14:23:32.790845+0000","last_fullsized":"2024-03-21T14:23:32.790845+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:21:35.013952+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_clean_scrub_stamp":"2024-03-21T14:21:35.013952+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"450'10935","reported_seq":11024,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:38.208669+0000","last_change":"2024-03-21T14:20:59.242772+0000","last_active":"2024-03-21T14:23:38.208669+0000","last_peered":"2024-03-21T14:23:38.208669+0000","last_clean":"2024-03-21T14:23:38.208669+0000","last_became_active":"2024-03-21T14:20:59.241791+0000","last_became_peered":"2024-03-21T14:20:59.241791+0000","last_unstale":"2024-03-21T14:23:38.208669+0000","last_undegraded":"2024-03-21T14:23:38.208669+0000","last_fullsized":"2024-03-21T14:23:38.208669+0000","mapping_epoch":411,"log_start":"438'5900","ondisk_log_start":"438'5900","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5035,"log_dups_size":0,"ondisk_log_size":5035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198377472,"num_objects":3027,"num_object_clones":0,"num_object_copies":9081,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3027,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":586,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794024+0000","last_change":"2024-03-21T14:22:41.593754+0000","last_active":"2024-03-21T14:23:32.794024+0000","last_peered":"2024-03-21T14:23:32.794024+0000","last_clean":"2024-03-21T14:23:32.794024+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:23:32.794024+0000","last_undegraded":"2024-03-21T14:23:32.794024+0000","last_fullsized":"2024-03-21T14:23:32.794024+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:41.593644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:41.593644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:41.855080+0000","scrub_duration":156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":877,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792095+0000","last_change":"2024-03-21T14:21:50.747507+0000","last_active":"2024-03-21T14:23:32.792095+0000","last_peered":"2024-03-21T14:23:32.792095+0000","last_clean":"2024-03-21T14:23:32.792095+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:23:32.792095+0000","last_undegraded":"2024-03-21T14:23:32.792095+0000","last_fullsized":"2024-03-21T14:23:32.792095+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.747341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":927,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787040+0000","last_change":"2024-03-21T14:20:21.797847+0000","last_active":"2024-03-21T14:23:32.787040+0000","last_peered":"2024-03-21T14:23:32.787040+0000","last_clean":"2024-03-21T14:23:32.787040+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:23:32.787040+0000","last_undegraded":"2024-03-21T14:23:32.787040+0000","last_fullsized":"2024-03-21T14:23:32.787040+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'18","last_scrub_stamp":"2024-03-21T14:18:29.432719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:29.432719+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":694,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789768+0000","last_change":"2024-03-21T14:21:54.295031+0000","last_active":"2024-03-21T14:23:32.789768+0000","last_peered":"2024-03-21T14:23:32.789768+0000","last_clean":"2024-03-21T14:23:32.789768+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:23:32.789768+0000","last_undegraded":"2024-03-21T14:23:32.789768+0000","last_fullsized":"2024-03-21T14:23:32.789768+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139492+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"450'10572","reported_seq":10661,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.675440+0000","last_change":"2024-03-21T14:20:59.387129+0000","last_active":"2024-03-21T14:23:37.675440+0000","last_peered":"2024-03-21T14:23:37.675440+0000","last_clean":"2024-03-21T14:23:37.675440+0000","last_became_active":"2024-03-21T14:20:59.386704+0000","last_became_peered":"2024-03-21T14:20:59.386704+0000","last_unstale":"2024-03-21T14:23:37.675440+0000","last_undegraded":"2024-03-21T14:23:37.675440+0000","last_fullsized":"2024-03-21T14:23:37.675440+0000","mapping_epoch":411,"log_start":"439'6400","ondisk_log_start":"439'6400","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.196059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199622656,"num_objects":3046,"num_object_clones":0,"num_object_copies":9138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10572,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":215,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789856+0000","last_change":"2024-03-21T14:22:54.306435+0000","last_active":"2024-03-21T14:23:32.789856+0000","last_peered":"2024-03-21T14:23:32.789856+0000","last_clean":"2024-03-21T14:23:32.789856+0000","last_became_active":"2024-03-21T14:19:14.700785+0000","last_became_peered":"2024-03-21T14:19:14.700785+0000","last_unstale":"2024-03-21T14:23:32.789856+0000","last_undegraded":"2024-03-21T14:23:32.789856+0000","last_fullsized":"2024-03-21T14:23:32.789856+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.306329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.306329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.846326+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":654,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789985+0000","last_change":"2024-03-21T14:23:04.356857+0000","last_active":"2024-03-21T14:23:32.789985+0000","last_peered":"2024-03-21T14:23:32.789985+0000","last_clean":"2024-03-21T14:23:32.789985+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:23:32.789985+0000","last_undegraded":"2024-03-21T14:23:32.789985+0000","last_fullsized":"2024-03-21T14:23:32.789985+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:04.356730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.356730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.865958+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":79,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.801943+0000","last_change":"2024-03-21T14:23:06.509103+0000","last_active":"2024-03-21T14:23:32.801943+0000","last_peered":"2024-03-21T14:23:32.801943+0000","last_clean":"2024-03-21T14:23:32.801943+0000","last_became_active":"2024-03-21T14:21:26.764088+0000","last_became_peered":"2024-03-21T14:21:26.764088+0000","last_unstale":"2024-03-21T14:23:32.801943+0000","last_undegraded":"2024-03-21T14:23:32.801943+0000","last_fullsized":"2024-03-21T14:23:32.801943+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:06.509028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:06.509028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.147435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"441'60","reported_seq":826,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790375+0000","last_change":"2024-03-21T14:20:49.145974+0000","last_active":"2024-03-21T14:23:32.790375+0000","last_peered":"2024-03-21T14:23:32.790375+0000","last_clean":"2024-03-21T14:23:32.790375+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:23:32.790375+0000","last_undegraded":"2024-03-21T14:23:32.790375+0000","last_fullsized":"2024-03-21T14:23:32.790375+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'44","last_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.145792+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"450'10631","reported_seq":13606,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.113344+0000","last_change":"2024-03-21T14:22:56.879969+0000","last_active":"2024-03-21T14:23:37.113344+0000","last_peered":"2024-03-21T14:23:37.113344+0000","last_clean":"2024-03-21T14:23:37.113344+0000","last_became_active":"2024-03-21T14:21:34.716304+0000","last_became_peered":"2024-03-21T14:21:34.716304+0000","last_unstale":"2024-03-21T14:23:37.113344+0000","last_undegraded":"2024-03-21T14:23:37.113344+0000","last_fullsized":"2024-03-21T14:23:37.113344+0000","mapping_epoch":427,"log_start":"439'6026","ondisk_log_start":"439'6026","created":411,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'7840","last_scrub_stamp":"2024-03-21T14:22:56.879850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:56.879850+0000","objects_scrubbed":6014,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:20.616700+0000","scrub_duration":1210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208601088,"num_objects":3183,"num_object_clones":0,"num_object_copies":9549,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3183,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10631,"num_write_kb":442048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1405,"num_bytes_recovered":92078080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":214,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789211+0000","last_change":"2024-03-21T14:22:29.603576+0000","last_active":"2024-03-21T14:23:32.789211+0000","last_peered":"2024-03-21T14:23:32.789211+0000","last_clean":"2024-03-21T14:23:32.789211+0000","last_became_active":"2024-03-21T14:19:34.690841+0000","last_became_peered":"2024-03-21T14:19:34.690841+0000","last_unstale":"2024-03-21T14:23:32.789211+0000","last_undegraded":"2024-03-21T14:23:32.789211+0000","last_fullsized":"2024-03-21T14:23:32.789211+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:29.603450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.603450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.754649+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":694,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789591+0000","last_change":"2024-03-21T14:21:46.343726+0000","last_active":"2024-03-21T14:23:32.789591+0000","last_peered":"2024-03-21T14:23:32.789591+0000","last_clean":"2024-03-21T14:23:32.789591+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:23:32.789591+0000","last_undegraded":"2024-03-21T14:23:32.789591+0000","last_fullsized":"2024-03-21T14:23:32.789591+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.343600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.343600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139544+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.800865+0000","last_change":"2024-03-21T14:22:50.084105+0000","last_active":"2024-03-21T14:23:32.800865+0000","last_peered":"2024-03-21T14:23:32.800865+0000","last_clean":"2024-03-21T14:23:32.800865+0000","last_became_active":"2024-03-21T14:22:50.083698+0000","last_became_peered":"2024-03-21T14:22:50.083698+0000","last_unstale":"2024-03-21T14:23:32.800865+0000","last_undegraded":"2024-03-21T14:23:32.800865+0000","last_fullsized":"2024-03-21T14:23:32.800865+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:47.337388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:47.337388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.949906+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"441'21","reported_seq":357,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790234+0000","last_change":"2024-03-21T14:21:20.291857+0000","last_active":"2024-03-21T14:23:32.790234+0000","last_peered":"2024-03-21T14:23:32.790234+0000","last_clean":"2024-03-21T14:23:32.790234+0000","last_became_active":"2024-03-21T14:19:14.692456+0000","last_became_peered":"2024-03-21T14:19:14.692456+0000","last_unstale":"2024-03-21T14:23:32.790234+0000","last_undegraded":"2024-03-21T14:23:32.790234+0000","last_fullsized":"2024-03-21T14:23:32.790234+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'15","last_scrub_stamp":"2024-03-21T14:21:20.291754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.291754+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"450'10323","reported_seq":10411,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.698665+0000","last_change":"2024-03-21T14:20:59.241783+0000","last_active":"2024-03-21T14:23:37.698665+0000","last_peered":"2024-03-21T14:23:37.698665+0000","last_clean":"2024-03-21T14:23:37.698665+0000","last_became_active":"2024-03-21T14:20:59.241477+0000","last_became_peered":"2024-03-21T14:20:59.241477+0000","last_unstale":"2024-03-21T14:23:37.698665+0000","last_undegraded":"2024-03-21T14:23:37.698665+0000","last_fullsized":"2024-03-21T14:23:37.698665+0000","mapping_epoch":411,"log_start":"439'6100","ondisk_log_start":"439'6100","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4223,"log_dups_size":0,"ondisk_log_size":4223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.195761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199950336,"num_objects":3051,"num_object_clones":0,"num_object_copies":9153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3051,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10323,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":830,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789382+0000","last_change":"2024-03-21T14:22:55.310949+0000","last_active":"2024-03-21T14:23:32.789382+0000","last_peered":"2024-03-21T14:23:32.789382+0000","last_clean":"2024-03-21T14:23:32.789382+0000","last_became_active":"2024-03-21T14:19:14.689666+0000","last_became_peered":"2024-03-21T14:19:14.689666+0000","last_unstale":"2024-03-21T14:23:32.789382+0000","last_undegraded":"2024-03-21T14:23:32.789382+0000","last_fullsized":"2024-03-21T14:23:32.789382+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:55.310815+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:55.310815+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:12.562170+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":833,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792154+0000","last_change":"2024-03-21T14:20:50.816435+0000","last_active":"2024-03-21T14:23:32.792154+0000","last_peered":"2024-03-21T14:23:32.792154+0000","last_clean":"2024-03-21T14:23:32.792154+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:23:32.792154+0000","last_undegraded":"2024-03-21T14:23:32.792154+0000","last_fullsized":"2024-03-21T14:23:32.792154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:50.816334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:50.816334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.104217+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":353,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790215+0000","last_change":"2024-03-21T14:22:53.863644+0000","last_active":"2024-03-21T14:23:32.790215+0000","last_peered":"2024-03-21T14:23:32.790215+0000","last_clean":"2024-03-21T14:23:32.790215+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:23:32.790215+0000","last_undegraded":"2024-03-21T14:23:32.790215+0000","last_fullsized":"2024-03-21T14:23:32.790215+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:22:53.863327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:22:53.863327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.416207+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":843,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786871+0000","last_change":"2024-03-21T14:22:33.543481+0000","last_active":"2024-03-21T14:23:32.786871+0000","last_peered":"2024-03-21T14:23:32.786871+0000","last_clean":"2024-03-21T14:23:32.786871+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:23:32.786871+0000","last_undegraded":"2024-03-21T14:23:32.786871+0000","last_fullsized":"2024-03-21T14:23:32.786871+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:33.543370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:33.543370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:57.805975+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"441'66","reported_seq":818,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793416+0000","last_change":"2024-03-21T14:22:45.361704+0000","last_active":"2024-03-21T14:23:32.793416+0000","last_peered":"2024-03-21T14:23:32.793416+0000","last_clean":"2024-03-21T14:23:32.793416+0000","last_became_active":"2024-03-21T14:19:14.693616+0000","last_became_peered":"2024-03-21T14:19:14.693616+0000","last_unstale":"2024-03-21T14:23:32.793416+0000","last_undegraded":"2024-03-21T14:23:32.793416+0000","last_fullsized":"2024-03-21T14:23:32.793416+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'66","last_scrub_stamp":"2024-03-21T14:22:45.361551+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:22:45.361551+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:59.222326+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":719,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789856+0000","last_change":"2024-03-21T14:21:32.597365+0000","last_active":"2024-03-21T14:23:32.789856+0000","last_peered":"2024-03-21T14:23:32.789856+0000","last_clean":"2024-03-21T14:23:32.789856+0000","last_became_active":"2024-03-21T14:19:34.690636+0000","last_became_peered":"2024-03-21T14:19:34.690636+0000","last_unstale":"2024-03-21T14:23:32.789856+0000","last_undegraded":"2024-03-21T14:23:32.789856+0000","last_fullsized":"2024-03-21T14:23:32.789856+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:32.597257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:32.597257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"450'10840","reported_seq":10928,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.676800+0000","last_change":"2024-03-21T14:20:59.387006+0000","last_active":"2024-03-21T14:23:37.676800+0000","last_peered":"2024-03-21T14:23:37.676800+0000","last_clean":"2024-03-21T14:23:37.676800+0000","last_became_active":"2024-03-21T14:20:59.386643+0000","last_became_peered":"2024-03-21T14:20:59.386643+0000","last_unstale":"2024-03-21T14:23:37.676800+0000","last_undegraded":"2024-03-21T14:23:37.676800+0000","last_fullsized":"2024-03-21T14:23:37.676800+0000","mapping_epoch":411,"log_start":"440'6600","ondisk_log_start":"440'6600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.196024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198967296,"num_objects":3036,"num_object_clones":0,"num_object_copies":9108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":444032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":1067,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786732+0000","last_change":"2024-03-21T14:21:52.357400+0000","last_active":"2024-03-21T14:23:32.786732+0000","last_peered":"2024-03-21T14:23:32.786732+0000","last_clean":"2024-03-21T14:23:32.786732+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:23:32.786732+0000","last_undegraded":"2024-03-21T14:23:32.786732+0000","last_fullsized":"2024-03-21T14:23:32.786732+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:52.357256+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:52.357256+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":857,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791232+0000","last_change":"2024-03-21T14:23:03.702113+0000","last_active":"2024-03-21T14:23:32.791232+0000","last_peered":"2024-03-21T14:23:32.791232+0000","last_clean":"2024-03-21T14:23:32.791232+0000","last_became_active":"2024-03-21T14:19:14.685026+0000","last_became_peered":"2024-03-21T14:19:14.685026+0000","last_unstale":"2024-03-21T14:23:32.791232+0000","last_undegraded":"2024-03-21T14:23:32.791232+0000","last_fullsized":"2024-03-21T14:23:32.791232+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:03.702034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:03.702034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.563384+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790958+0000","last_change":"2024-03-21T14:22:45.828480+0000","last_active":"2024-03-21T14:23:32.790958+0000","last_peered":"2024-03-21T14:23:32.790958+0000","last_clean":"2024-03-21T14:23:32.790958+0000","last_became_active":"2024-03-21T14:19:14.697007+0000","last_became_peered":"2024-03-21T14:19:14.697007+0000","last_unstale":"2024-03-21T14:23:32.790958+0000","last_undegraded":"2024-03-21T14:23:32.790958+0000","last_fullsized":"2024-03-21T14:23:32.790958+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:45.828270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:22:45.828270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:57.425654+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":689,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789071+0000","last_change":"2024-03-21T14:20:48.551092+0000","last_active":"2024-03-21T14:23:32.789071+0000","last_peered":"2024-03-21T14:23:32.789071+0000","last_clean":"2024-03-21T14:23:32.789071+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:23:32.789071+0000","last_undegraded":"2024-03-21T14:23:32.789071+0000","last_fullsized":"2024-03-21T14:23:32.789071+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.550975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.550975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.131821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":945,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790182+0000","last_change":"2024-03-21T14:21:47.628148+0000","last_active":"2024-03-21T14:23:32.790182+0000","last_peered":"2024-03-21T14:23:32.790182+0000","last_clean":"2024-03-21T14:23:32.790182+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:23:32.790182+0000","last_undegraded":"2024-03-21T14:23:32.790182+0000","last_fullsized":"2024-03-21T14:23:32.790182+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_deep_scrub":"236'22","last_deep_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.628002+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":672,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793017+0000","last_change":"2024-03-21T14:21:42.301835+0000","last_active":"2024-03-21T14:23:32.793017+0000","last_peered":"2024-03-21T14:23:32.793017+0000","last_clean":"2024-03-21T14:23:32.793017+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:23:32.793017+0000","last_undegraded":"2024-03-21T14:23:32.793017+0000","last_fullsized":"2024-03-21T14:23:32.793017+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:42.301114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:21:42.301114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.139518+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"450'10815","reported_seq":10904,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.646207+0000","last_change":"2024-03-21T14:20:59.387064+0000","last_active":"2024-03-21T14:23:37.646207+0000","last_peered":"2024-03-21T14:23:37.646207+0000","last_clean":"2024-03-21T14:23:37.646207+0000","last_became_active":"2024-03-21T14:20:59.386748+0000","last_became_peered":"2024-03-21T14:20:59.386748+0000","last_unstale":"2024-03-21T14:23:37.646207+0000","last_undegraded":"2024-03-21T14:23:37.646207+0000","last_fullsized":"2024-03-21T14:23:37.646207+0000","mapping_epoch":411,"log_start":"440'6600","ondisk_log_start":"440'6600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4215,"log_dups_size":0,"ondisk_log_size":4215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.195956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198246400,"num_objects":3025,"num_object_clones":0,"num_object_copies":9075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10815,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":318,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791472+0000","last_change":"2024-03-21T14:21:23.819765+0000","last_active":"2024-03-21T14:23:32.791472+0000","last_peered":"2024-03-21T14:23:32.791472+0000","last_clean":"2024-03-21T14:23:32.791472+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:23:32.791472+0000","last_undegraded":"2024-03-21T14:23:32.791472+0000","last_fullsized":"2024-03-21T14:23:32.791472+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:23.819644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:23.819644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105066+0000","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1167,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790463+0000","last_change":"2024-03-21T14:21:44.886680+0000","last_active":"2024-03-21T14:23:32.790463+0000","last_peered":"2024-03-21T14:23:32.790463+0000","last_clean":"2024-03-21T14:23:32.790463+0000","last_became_active":"2024-03-21T14:19:37.727165+0000","last_became_peered":"2024-03-21T14:19:37.727165+0000","last_unstale":"2024-03-21T14:23:32.790463+0000","last_undegraded":"2024-03-21T14:23:32.790463+0000","last_fullsized":"2024-03-21T14:23:32.790463+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:44.886598+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:44.886598+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":868,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794133+0000","last_change":"2024-03-21T14:22:30.336811+0000","last_active":"2024-03-21T14:23:32.794133+0000","last_peered":"2024-03-21T14:23:32.794133+0000","last_clean":"2024-03-21T14:23:32.794133+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:23:32.794133+0000","last_undegraded":"2024-03-21T14:23:32.794133+0000","last_fullsized":"2024-03-21T14:23:32.794133+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:30.335992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:22:30.335992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:48.200343+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"441'41","reported_seq":754,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787433+0000","last_change":"2024-03-21T14:22:58.255728+0000","last_active":"2024-03-21T14:23:32.787433+0000","last_peered":"2024-03-21T14:23:32.787433+0000","last_clean":"2024-03-21T14:23:32.787433+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:23:32.787433+0000","last_undegraded":"2024-03-21T14:23:32.787433+0000","last_fullsized":"2024-03-21T14:23:32.787433+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'41","last_scrub_stamp":"2024-03-21T14:22:58.255560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:58.255560+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:18.445411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":214,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793946+0000","last_change":"2024-03-21T14:23:07.337422+0000","last_active":"2024-03-21T14:23:32.793946+0000","last_peered":"2024-03-21T14:23:32.793946+0000","last_clean":"2024-03-21T14:23:32.793946+0000","last_became_active":"2024-03-21T14:19:14.690145+0000","last_became_peered":"2024-03-21T14:19:14.690145+0000","last_unstale":"2024-03-21T14:23:32.793946+0000","last_undegraded":"2024-03-21T14:23:32.793946+0000","last_fullsized":"2024-03-21T14:23:32.793946+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:07.333488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:07.333488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:12.814063+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"450'10905","reported_seq":10994,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.681615+0000","last_change":"2024-03-21T14:20:59.239970+0000","last_active":"2024-03-21T14:23:37.681615+0000","last_peered":"2024-03-21T14:23:37.681615+0000","last_clean":"2024-03-21T14:23:37.681615+0000","last_became_active":"2024-03-21T14:20:59.239659+0000","last_became_peered":"2024-03-21T14:20:59.239659+0000","last_unstale":"2024-03-21T14:23:37.681615+0000","last_undegraded":"2024-03-21T14:23:37.681615+0000","last_fullsized":"2024-03-21T14:23:37.681615+0000","mapping_epoch":411,"log_start":"440'6700","ondisk_log_start":"440'6700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4205,"log_dups_size":0,"ondisk_log_size":4205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.195989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":206110720,"num_objects":3145,"num_object_clones":0,"num_object_copies":9435,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3145,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10905,"num_write_kb":449600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":923,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789310+0000","last_change":"2024-03-21T14:22:32.735774+0000","last_active":"2024-03-21T14:23:32.789310+0000","last_peered":"2024-03-21T14:23:32.789310+0000","last_clean":"2024-03-21T14:23:32.789310+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:23:32.789310+0000","last_undegraded":"2024-03-21T14:23:32.789310+0000","last_fullsized":"2024-03-21T14:23:32.789310+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:32.735646+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:32.735646+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:53.418603+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":860,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793710+0000","last_change":"2024-03-21T14:20:48.131092+0000","last_active":"2024-03-21T14:23:32.793710+0000","last_peered":"2024-03-21T14:23:32.793710+0000","last_clean":"2024-03-21T14:23:32.793710+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:23:32.793710+0000","last_undegraded":"2024-03-21T14:23:32.793710+0000","last_fullsized":"2024-03-21T14:23:32.793710+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:48.130989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:20:48.130989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.670488+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":652,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787085+0000","last_change":"2024-03-21T14:22:51.407734+0000","last_active":"2024-03-21T14:23:32.787085+0000","last_peered":"2024-03-21T14:23:32.787085+0000","last_clean":"2024-03-21T14:23:32.787085+0000","last_became_active":"2024-03-21T14:20:21.778326+0000","last_became_peered":"2024-03-21T14:20:21.778326+0000","last_unstale":"2024-03-21T14:23:32.787085+0000","last_undegraded":"2024-03-21T14:23:32.787085+0000","last_fullsized":"2024-03-21T14:23:32.787085+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:51.407608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.407608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.985048+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":2028,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:36.825541+0000","last_change":"2024-03-21T14:21:41.487654+0000","last_active":"2024-03-21T14:23:36.825541+0000","last_peered":"2024-03-21T14:23:36.825541+0000","last_clean":"2024-03-21T14:23:36.825541+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:23:36.825541+0000","last_undegraded":"2024-03-21T14:23:36.825541+0000","last_fullsized":"2024-03-21T14:23:36.825541+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:21:41.487571+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.487571+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.673807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1198,"num_read_kb":1193,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":814,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791861+0000","last_change":"2024-03-21T14:22:51.894009+0000","last_active":"2024-03-21T14:23:32.791861+0000","last_peered":"2024-03-21T14:23:32.791861+0000","last_clean":"2024-03-21T14:23:32.791861+0000","last_became_active":"2024-03-21T14:19:34.692301+0000","last_became_peered":"2024-03-21T14:19:34.692301+0000","last_unstale":"2024-03-21T14:23:32.791861+0000","last_undegraded":"2024-03-21T14:23:32.791861+0000","last_fullsized":"2024-03-21T14:23:32.791861+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_deep_scrub":"441'36","last_deep_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.893561+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:55.310816+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":320,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790697+0000","last_change":"2024-03-21T14:21:40.525563+0000","last_active":"2024-03-21T14:23:32.790697+0000","last_peered":"2024-03-21T14:23:32.790697+0000","last_clean":"2024-03-21T14:23:32.790697+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:23:32.790697+0000","last_undegraded":"2024-03-21T14:23:32.790697+0000","last_fullsized":"2024-03-21T14:23:32.790697+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:40.525392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:21:40.525392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"450'10785","reported_seq":10874,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.796192+0000","last_change":"2024-03-21T14:20:59.110120+0000","last_active":"2024-03-21T14:23:37.796192+0000","last_peered":"2024-03-21T14:23:37.796192+0000","last_clean":"2024-03-21T14:23:37.796192+0000","last_became_active":"2024-03-21T14:20:59.109825+0000","last_became_peered":"2024-03-21T14:20:59.109825+0000","last_unstale":"2024-03-21T14:23:37.796192+0000","last_undegraded":"2024-03-21T14:23:37.796192+0000","last_fullsized":"2024-03-21T14:23:37.796192+0000","mapping_epoch":411,"log_start":"441'6700","ondisk_log_start":"441'6700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4085,"log_dups_size":0,"ondisk_log_size":4085,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:39.670436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200081408,"num_objects":3053,"num_object_clones":0,"num_object_copies":9159,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3053,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10785,"num_write_kb":442816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":651,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791712+0000","last_change":"2024-03-21T14:20:30.945601+0000","last_active":"2024-03-21T14:23:32.791712+0000","last_peered":"2024-03-21T14:23:32.791712+0000","last_clean":"2024-03-21T14:23:32.791712+0000","last_became_active":"2024-03-21T14:20:24.825637+0000","last_became_peered":"2024-03-21T14:20:24.825637+0000","last_unstale":"2024-03-21T14:23:32.791712+0000","last_undegraded":"2024-03-21T14:23:32.791712+0000","last_fullsized":"2024-03-21T14:23:32.791712+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:30.945471+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.945471+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.102985+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":857,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790309+0000","last_change":"2024-03-21T14:21:15.242152+0000","last_active":"2024-03-21T14:23:32.790309+0000","last_peered":"2024-03-21T14:23:32.790309+0000","last_clean":"2024-03-21T14:23:32.790309+0000","last_became_active":"2024-03-21T14:19:14.697998+0000","last_became_peered":"2024-03-21T14:19:14.697998+0000","last_unstale":"2024-03-21T14:23:32.790309+0000","last_undegraded":"2024-03-21T14:23:32.790309+0000","last_fullsized":"2024-03-21T14:23:32.790309+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:15.242057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:59.663279+0000","last_clean_scrub_stamp":"2024-03-21T14:21:15.242057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":863,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793998+0000","last_change":"2024-03-21T14:22:48.925469+0000","last_active":"2024-03-21T14:23:32.793998+0000","last_peered":"2024-03-21T14:23:32.793998+0000","last_clean":"2024-03-21T14:23:32.793998+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:23:32.793998+0000","last_undegraded":"2024-03-21T14:23:32.793998+0000","last_fullsized":"2024-03-21T14:23:32.793998+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:48.925350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.925350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:14.372873+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'39","reported_seq":599,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.783531+0000","last_change":"2024-03-21T14:22:29.489662+0000","last_active":"2024-03-21T14:23:32.783531+0000","last_peered":"2024-03-21T14:23:32.783531+0000","last_clean":"2024-03-21T14:23:32.783531+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:23:32.783531+0000","last_undegraded":"2024-03-21T14:23:32.783531+0000","last_fullsized":"2024-03-21T14:23:32.783531+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'29","last_scrub_stamp":"2024-03-21T14:22:29.489522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.489522+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:37.352819+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":218,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789160+0000","last_change":"2024-03-21T14:22:50.356522+0000","last_active":"2024-03-21T14:23:32.789160+0000","last_peered":"2024-03-21T14:23:32.789160+0000","last_clean":"2024-03-21T14:23:32.789160+0000","last_became_active":"2024-03-21T14:19:14.675538+0000","last_became_peered":"2024-03-21T14:19:14.675538+0000","last_unstale":"2024-03-21T14:23:32.789160+0000","last_undegraded":"2024-03-21T14:23:32.789160+0000","last_fullsized":"2024-03-21T14:23:32.789160+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:50.356413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:22:50.356413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:56.994527+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"450'10951","reported_seq":13964,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:38.216224+0000","last_change":"2024-03-21T14:21:41.926494+0000","last_active":"2024-03-21T14:23:38.216224+0000","last_peered":"2024-03-21T14:23:38.216224+0000","last_clean":"2024-03-21T14:23:38.216224+0000","last_became_active":"2024-03-21T14:21:41.923763+0000","last_became_peered":"2024-03-21T14:21:41.923763+0000","last_unstale":"2024-03-21T14:23:38.216224+0000","last_undegraded":"2024-03-21T14:23:38.216224+0000","last_fullsized":"2024-03-21T14:23:38.216224+0000","mapping_epoch":431,"log_start":"438'5900","ondisk_log_start":"438'5900","created":411,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5051,"log_dups_size":0,"ondisk_log_size":5051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202113048,"num_objects":3085,"num_object_clones":0,"num_object_copies":9255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3085,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10951,"num_write_kb":449089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":95420416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":825,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790987+0000","last_change":"2024-03-21T14:20:24.821031+0000","last_active":"2024-03-21T14:23:32.790987+0000","last_peered":"2024-03-21T14:23:32.790987+0000","last_clean":"2024-03-21T14:23:32.790987+0000","last_became_active":"2024-03-21T14:20:24.819844+0000","last_became_peered":"2024-03-21T14:20:24.819844+0000","last_unstale":"2024-03-21T14:23:32.790987+0000","last_undegraded":"2024-03-21T14:23:32.790987+0000","last_fullsized":"2024-03-21T14:23:32.790987+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:19:15.665015+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:19:15.665015+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.102863+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":356,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791690+0000","last_change":"2024-03-21T14:22:56.247550+0000","last_active":"2024-03-21T14:23:32.791690+0000","last_peered":"2024-03-21T14:23:32.791690+0000","last_clean":"2024-03-21T14:23:32.791690+0000","last_became_active":"2024-03-21T14:19:15.688238+0000","last_became_peered":"2024-03-21T14:19:15.688238+0000","last_unstale":"2024-03-21T14:23:32.791690+0000","last_undegraded":"2024-03-21T14:23:32.791690+0000","last_fullsized":"2024-03-21T14:23:32.791690+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:56.247428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:56.247428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:16.119271+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":785,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792155+0000","last_change":"2024-03-21T14:20:32.884624+0000","last_active":"2024-03-21T14:23:32.792155+0000","last_peered":"2024-03-21T14:23:32.792155+0000","last_clean":"2024-03-21T14:23:32.792155+0000","last_became_active":"2024-03-21T14:19:14.695943+0000","last_became_peered":"2024-03-21T14:19:14.695943+0000","last_unstale":"2024-03-21T14:23:32.792155+0000","last_undegraded":"2024-03-21T14:23:32.792155+0000","last_fullsized":"2024-03-21T14:23:32.792155+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:32.884514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:32.884514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.102955+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":908,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790968+0000","last_change":"2024-03-21T14:21:43.496288+0000","last_active":"2024-03-21T14:23:32.790968+0000","last_peered":"2024-03-21T14:23:32.790968+0000","last_clean":"2024-03-21T14:23:32.790968+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:23:32.790968+0000","last_undegraded":"2024-03-21T14:23:32.790968+0000","last_fullsized":"2024-03-21T14:23:32.790968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:43.496137+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:21:43.496137+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":863,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793955+0000","last_change":"2024-03-21T14:22:55.163828+0000","last_active":"2024-03-21T14:23:32.793955+0000","last_peered":"2024-03-21T14:23:32.793955+0000","last_clean":"2024-03-21T14:23:32.793955+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:23:32.793955+0000","last_undegraded":"2024-03-21T14:23:32.793955+0000","last_fullsized":"2024-03-21T14:23:32.793955+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'11","last_scrub_stamp":"2024-03-21T14:22:55.163691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:55.163691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:14.088158+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"450'10806","reported_seq":10895,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:36.893357+0000","last_change":"2024-03-21T14:20:59.110795+0000","last_active":"2024-03-21T14:23:36.893357+0000","last_peered":"2024-03-21T14:23:36.893357+0000","last_clean":"2024-03-21T14:23:36.893357+0000","last_became_active":"2024-03-21T14:20:59.110468+0000","last_became_peered":"2024-03-21T14:20:59.110468+0000","last_unstale":"2024-03-21T14:23:36.893357+0000","last_undegraded":"2024-03-21T14:23:36.893357+0000","last_fullsized":"2024-03-21T14:23:36.893357+0000","mapping_epoch":411,"log_start":"439'6400","ondisk_log_start":"439'6400","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4406,"log_dups_size":0,"ondisk_log_size":4406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208797696,"num_objects":3186,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10806,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":907,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791440+0000","last_change":"2024-03-21T14:20:52.764101+0000","last_active":"2024-03-21T14:23:32.791440+0000","last_peered":"2024-03-21T14:23:32.791440+0000","last_clean":"2024-03-21T14:23:32.791440+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:23:32.791440+0000","last_undegraded":"2024-03-21T14:23:32.791440+0000","last_fullsized":"2024-03-21T14:23:32.791440+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:52.763988+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.763988+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.104276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":857,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791104+0000","last_change":"2024-03-21T14:22:49.915554+0000","last_active":"2024-03-21T14:23:32.791104+0000","last_peered":"2024-03-21T14:23:32.791104+0000","last_clean":"2024-03-21T14:23:32.791104+0000","last_became_active":"2024-03-21T14:19:14.684876+0000","last_became_peered":"2024-03-21T14:19:14.684876+0000","last_unstale":"2024-03-21T14:23:32.791104+0000","last_undegraded":"2024-03-21T14:23:32.791104+0000","last_fullsized":"2024-03-21T14:23:32.791104+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:49.915445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:27.609702+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.915445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:04.268044+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":721,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791714+0000","last_change":"2024-03-21T14:20:51.116248+0000","last_active":"2024-03-21T14:23:32.791714+0000","last_peered":"2024-03-21T14:23:32.791714+0000","last_clean":"2024-03-21T14:23:32.791714+0000","last_became_active":"2024-03-21T14:19:34.692258+0000","last_became_peered":"2024-03-21T14:19:34.692258+0000","last_unstale":"2024-03-21T14:23:32.791714+0000","last_undegraded":"2024-03-21T14:23:32.791714+0000","last_fullsized":"2024-03-21T14:23:32.791714+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:20:51.116139+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:20:51.116139+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"441'42","reported_seq":948,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786805+0000","last_change":"2024-03-21T14:22:49.399821+0000","last_active":"2024-03-21T14:23:32.786805+0000","last_peered":"2024-03-21T14:23:32.786805+0000","last_clean":"2024-03-21T14:23:32.786805+0000","last_became_active":"2024-03-21T14:21:26.768622+0000","last_became_peered":"2024-03-21T14:21:26.768622+0000","last_unstale":"2024-03-21T14:23:32.786805+0000","last_undegraded":"2024-03-21T14:23:32.786805+0000","last_fullsized":"2024-03-21T14:23:32.786805+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-21T14:22:49.399707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.399707+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:00.879647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":1107,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.792265+0000","last_change":"2024-03-21T14:22:38.443967+0000","last_active":"2024-03-21T14:23:32.792265+0000","last_peered":"2024-03-21T14:23:32.792265+0000","last_clean":"2024-03-21T14:23:32.792265+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:23:32.792265+0000","last_undegraded":"2024-03-21T14:23:32.792265+0000","last_fullsized":"2024-03-21T14:23:32.792265+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:22:38.443825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:38.443825+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:43.527309+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"450'10728","reported_seq":11602,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:36.884640+0000","last_change":"2024-03-21T14:21:28.769176+0000","last_active":"2024-03-21T14:23:36.884640+0000","last_peered":"2024-03-21T14:23:36.884640+0000","last_clean":"2024-03-21T14:23:36.884640+0000","last_became_active":"2024-03-21T14:21:28.766274+0000","last_became_peered":"2024-03-21T14:21:28.766274+0000","last_unstale":"2024-03-21T14:23:36.884640+0000","last_undegraded":"2024-03-21T14:23:36.884640+0000","last_fullsized":"2024-03-21T14:23:36.884640+0000","mapping_epoch":425,"log_start":"439'6300","ondisk_log_start":"439'6300","created":411,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209977344,"num_objects":3204,"num_object_clones":0,"num_object_copies":9612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10728,"num_write_kb":445824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":59375616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":909,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787115+0000","last_change":"2024-03-21T14:21:19.302873+0000","last_active":"2024-03-21T14:23:32.787115+0000","last_peered":"2024-03-21T14:23:32.787115+0000","last_clean":"2024-03-21T14:23:32.787115+0000","last_became_active":"2024-03-21T14:19:14.682251+0000","last_became_peered":"2024-03-21T14:19:14.682251+0000","last_unstale":"2024-03-21T14:23:32.787115+0000","last_undegraded":"2024-03-21T14:23:32.787115+0000","last_fullsized":"2024-03-21T14:23:32.787115+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:19.302784+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.302784+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":857,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786891+0000","last_change":"2024-03-21T14:22:35.628602+0000","last_active":"2024-03-21T14:23:32.786891+0000","last_peered":"2024-03-21T14:23:32.786891+0000","last_clean":"2024-03-21T14:23:32.786891+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:23:32.786891+0000","last_undegraded":"2024-03-21T14:23:32.786891+0000","last_fullsized":"2024-03-21T14:23:32.786891+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:35.628491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.628491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:49.293615+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":853,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791310+0000","last_change":"2024-03-21T14:21:39.060809+0000","last_active":"2024-03-21T14:23:32.791310+0000","last_peered":"2024-03-21T14:23:32.791310+0000","last_clean":"2024-03-21T14:23:32.791310+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:23:32.791310+0000","last_undegraded":"2024-03-21T14:23:32.791310+0000","last_fullsized":"2024-03-21T14:23:32.791310+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:39.060697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:21:39.060697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105327+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'50","reported_seq":172,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790419+0000","last_change":"2024-03-21T14:23:04.313764+0000","last_active":"2024-03-21T14:23:32.790419+0000","last_peered":"2024-03-21T14:23:32.790419+0000","last_clean":"2024-03-21T14:23:32.790419+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:23:32.790419+0000","last_undegraded":"2024-03-21T14:23:32.790419+0000","last_fullsized":"2024-03-21T14:23:32.790419+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'50","last_scrub_stamp":"2024-03-21T14:23:04.313411+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.313411+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:11.545037+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":829,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786658+0000","last_change":"2024-03-21T14:22:43.461461+0000","last_active":"2024-03-21T14:23:32.786658+0000","last_peered":"2024-03-21T14:23:32.786658+0000","last_clean":"2024-03-21T14:23:32.786658+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:23:32.786658+0000","last_undegraded":"2024-03-21T14:23:32.786658+0000","last_fullsized":"2024-03-21T14:23:32.786658+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.461338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.461338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:02.086996+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"450'10702","reported_seq":10861,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.804765+0000","last_change":"2024-03-21T14:22:33.276757+0000","last_active":"2024-03-21T14:23:37.804765+0000","last_peered":"2024-03-21T14:23:37.804765+0000","last_clean":"2024-03-21T14:23:37.804765+0000","last_became_active":"2024-03-21T14:20:59.109191+0000","last_became_peered":"2024-03-21T14:20:59.109191+0000","last_unstale":"2024-03-21T14:23:37.804765+0000","last_undegraded":"2024-03-21T14:23:37.804765+0000","last_fullsized":"2024-03-21T14:23:37.804765+0000","mapping_epoch":411,"log_start":"441'6700","ondisk_log_start":"441'6700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'6212","last_scrub_stamp":"2024-03-21T14:22:33.276373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:33.276373+0000","objects_scrubbed":6148,"log_size":4002,"log_dups_size":0,"ondisk_log_size":4002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199491584,"num_objects":3044,"num_object_clones":0,"num_object_copies":9132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10702,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":822,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789502+0000","last_change":"2024-03-21T14:20:36.503605+0000","last_active":"2024-03-21T14:23:32.789502+0000","last_peered":"2024-03-21T14:23:32.789502+0000","last_clean":"2024-03-21T14:23:32.789502+0000","last_became_active":"2024-03-21T14:19:14.692919+0000","last_became_peered":"2024-03-21T14:19:14.692919+0000","last_unstale":"2024-03-21T14:23:32.789502+0000","last_undegraded":"2024-03-21T14:23:32.789502+0000","last_fullsized":"2024-03-21T14:23:32.789502+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:36.503498+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:36.503498+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":862,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789342+0000","last_change":"2024-03-21T14:20:43.142589+0000","last_active":"2024-03-21T14:23:32.789342+0000","last_peered":"2024-03-21T14:23:32.789342+0000","last_clean":"2024-03-21T14:23:32.789342+0000","last_became_active":"2024-03-21T14:19:34.691121+0000","last_became_peered":"2024-03-21T14:19:34.691121+0000","last_unstale":"2024-03-21T14:23:32.789342+0000","last_undegraded":"2024-03-21T14:23:32.789342+0000","last_fullsized":"2024-03-21T14:23:32.789342+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:43.142495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:20:43.142495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1097,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787474+0000","last_change":"2024-03-21T14:22:40.498837+0000","last_active":"2024-03-21T14:23:32.787474+0000","last_peered":"2024-03-21T14:23:32.787474+0000","last_clean":"2024-03-21T14:23:32.787474+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:23:32.787474+0000","last_undegraded":"2024-03-21T14:23:32.787474+0000","last_fullsized":"2024-03-21T14:23:32.787474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:22:40.498694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.498694+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:49.467479+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":136,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790659+0000","last_change":"2024-03-21T14:21:54.066071+0000","last_active":"2024-03-21T14:23:32.790659+0000","last_peered":"2024-03-21T14:23:32.790659+0000","last_clean":"2024-03-21T14:23:32.790659+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:23:32.790659+0000","last_undegraded":"2024-03-21T14:23:32.790659+0000","last_fullsized":"2024-03-21T14:23:32.790659+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.065922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.065922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105398+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'39","reported_seq":932,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790285+0000","last_change":"2024-03-21T14:21:19.723743+0000","last_active":"2024-03-21T14:23:32.790285+0000","last_peered":"2024-03-21T14:23:32.790285+0000","last_clean":"2024-03-21T14:23:32.790285+0000","last_became_active":"2024-03-21T14:21:19.713483+0000","last_became_peered":"2024-03-21T14:21:19.713483+0000","last_unstale":"2024-03-21T14:23:32.790285+0000","last_undegraded":"2024-03-21T14:23:32.790285+0000","last_fullsized":"2024-03-21T14:23:32.790285+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-03-21T14:20:42.035180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.035180+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"450'10840","reported_seq":10929,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.631957+0000","last_change":"2024-03-21T14:20:59.240921+0000","last_active":"2024-03-21T14:23:37.631957+0000","last_peered":"2024-03-21T14:23:37.631957+0000","last_clean":"2024-03-21T14:23:37.631957+0000","last_became_active":"2024-03-21T14:20:59.240641+0000","last_became_peered":"2024-03-21T14:20:59.240641+0000","last_unstale":"2024-03-21T14:23:37.631957+0000","last_undegraded":"2024-03-21T14:23:37.631957+0000","last_fullsized":"2024-03-21T14:23:37.631957+0000","mapping_epoch":411,"log_start":"440'6600","ondisk_log_start":"440'6600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:40.195912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204210176,"num_objects":3116,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10840,"num_write_kb":446592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":847,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.791549+0000","last_change":"2024-03-21T14:22:39.460174+0000","last_active":"2024-03-21T14:23:32.791549+0000","last_peered":"2024-03-21T14:23:32.791549+0000","last_clean":"2024-03-21T14:23:32.791549+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:23:32.791549+0000","last_undegraded":"2024-03-21T14:23:32.791549+0000","last_fullsized":"2024-03-21T14:23:32.791549+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:39.460024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:39.460024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:02.211969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":837,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787493+0000","last_change":"2024-03-21T14:21:51.322736+0000","last_active":"2024-03-21T14:23:32.787493+0000","last_peered":"2024-03-21T14:23:32.787493+0000","last_clean":"2024-03-21T14:23:32.787493+0000","last_became_active":"2024-03-21T14:21:26.767687+0000","last_became_peered":"2024-03-21T14:21:26.767687+0000","last_unstale":"2024-03-21T14:23:32.787493+0000","last_undegraded":"2024-03-21T14:23:32.787493+0000","last_fullsized":"2024-03-21T14:23:32.787493+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.322658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.322658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":853,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.786688+0000","last_change":"2024-03-21T14:22:31.537067+0000","last_active":"2024-03-21T14:23:32.786688+0000","last_peered":"2024-03-21T14:23:32.786688+0000","last_clean":"2024-03-21T14:23:32.786688+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:23:32.786688+0000","last_undegraded":"2024-03-21T14:23:32.786688+0000","last_fullsized":"2024-03-21T14:23:32.786688+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:31.536957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:31.536957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:41.005949+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'27","reported_seq":893,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787308+0000","last_change":"2024-03-21T14:18:32.414098+0000","last_active":"2024-03-21T14:23:32.787308+0000","last_peered":"2024-03-21T14:23:32.787308+0000","last_clean":"2024-03-21T14:23:32.787308+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:23:32.787308+0000","last_undegraded":"2024-03-21T14:23:32.787308+0000","last_fullsized":"2024-03-21T14:23:32.787308+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-03-21T14:18:32.413975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:18:32.413975+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"450'10670","reported_seq":10757,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:38.208483+0000","last_change":"2024-03-21T14:20:59.239976+0000","last_active":"2024-03-21T14:23:38.208483+0000","last_peered":"2024-03-21T14:23:38.208483+0000","last_clean":"2024-03-21T14:23:38.208483+0000","last_became_active":"2024-03-21T14:20:59.239396+0000","last_became_peered":"2024-03-21T14:20:59.239396+0000","last_unstale":"2024-03-21T14:23:38.208483+0000","last_undegraded":"2024-03-21T14:23:38.208483+0000","last_fullsized":"2024-03-21T14:23:38.208483+0000","mapping_epoch":411,"log_start":"438'5600","ondisk_log_start":"438'5600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198574080,"num_objects":3030,"num_object_clones":0,"num_object_copies":9090,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3030,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10670,"num_write_kb":438400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":830,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789341+0000","last_change":"2024-03-21T14:22:57.334699+0000","last_active":"2024-03-21T14:23:32.789341+0000","last_peered":"2024-03-21T14:23:32.789341+0000","last_clean":"2024-03-21T14:23:32.789341+0000","last_became_active":"2024-03-21T14:19:14.700353+0000","last_became_peered":"2024-03-21T14:19:14.700353+0000","last_unstale":"2024-03-21T14:23:32.789341+0000","last_undegraded":"2024-03-21T14:23:32.789341+0000","last_fullsized":"2024-03-21T14:23:32.789341+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:57.334563+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.334563+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.721763+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":640,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794397+0000","last_change":"2024-03-21T14:20:45.611947+0000","last_active":"2024-03-21T14:23:32.794397+0000","last_peered":"2024-03-21T14:23:32.794397+0000","last_clean":"2024-03-21T14:23:32.794397+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:23:32.794397+0000","last_undegraded":"2024-03-21T14:23:32.794397+0000","last_fullsized":"2024-03-21T14:23:32.794397+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.611844+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.131789+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":853,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794152+0000","last_change":"2024-03-21T14:21:54.619575+0000","last_active":"2024-03-21T14:23:32.794152+0000","last_peered":"2024-03-21T14:23:32.794152+0000","last_clean":"2024-03-21T14:23:32.794152+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:23:32.794152+0000","last_undegraded":"2024-03-21T14:23:32.794152+0000","last_fullsized":"2024-03-21T14:23:32.794152+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.617817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":140,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.789059+0000","last_change":"2024-03-21T14:22:41.364253+0000","last_active":"2024-03-21T14:23:32.789059+0000","last_peered":"2024-03-21T14:23:32.789059+0000","last_clean":"2024-03-21T14:23:32.789059+0000","last_became_active":"2024-03-21T14:20:19.703587+0000","last_became_peered":"2024-03-21T14:20:19.703587+0000","last_unstale":"2024-03-21T14:23:32.789059+0000","last_undegraded":"2024-03-21T14:23:32.789059+0000","last_fullsized":"2024-03-21T14:23:32.789059+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:41.363995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:41.363995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:51.499840+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":903,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787485+0000","last_change":"2024-03-21T14:20:46.480561+0000","last_active":"2024-03-21T14:23:32.787485+0000","last_peered":"2024-03-21T14:23:32.787485+0000","last_clean":"2024-03-21T14:23:32.787485+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:23:32.787485+0000","last_undegraded":"2024-03-21T14:23:32.787485+0000","last_fullsized":"2024-03-21T14:23:32.787485+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:20:46.480457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.480457+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"450'10735","reported_seq":10890,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:37.118806+0000","last_change":"2024-03-21T14:22:59.010537+0000","last_active":"2024-03-21T14:23:37.118806+0000","last_peered":"2024-03-21T14:23:37.118806+0000","last_clean":"2024-03-21T14:23:37.118806+0000","last_became_active":"2024-03-21T14:20:59.386715+0000","last_became_peered":"2024-03-21T14:20:59.386715+0000","last_unstale":"2024-03-21T14:23:37.118806+0000","last_undegraded":"2024-03-21T14:23:37.118806+0000","last_fullsized":"2024-03-21T14:23:37.118806+0000","mapping_epoch":411,"log_start":"439'6116","ondisk_log_start":"439'6116","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'8088","last_scrub_stamp":"2024-03-21T14:22:59.010282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:59.010282+0000","objects_scrubbed":5843,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:09.160780+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205848576,"num_objects":3141,"num_object_clones":0,"num_object_copies":9423,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3141,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10735,"num_write_kb":444032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1174,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.790464+0000","last_change":"2024-03-21T14:23:04.602240+0000","last_active":"2024-03-21T14:23:32.790464+0000","last_peered":"2024-03-21T14:23:32.790464+0000","last_clean":"2024-03-21T14:23:32.790464+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:23:32.790464+0000","last_undegraded":"2024-03-21T14:23:32.790464+0000","last_fullsized":"2024-03-21T14:23:32.790464+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:23:04.602105+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.602105+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:32.873082+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":849,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787515+0000","last_change":"2024-03-21T14:19:15.689377+0000","last_active":"2024-03-21T14:23:32.787515+0000","last_peered":"2024-03-21T14:23:32.787515+0000","last_clean":"2024-03-21T14:23:32.787515+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:23:32.787515+0000","last_undegraded":"2024-03-21T14:23:32.787515+0000","last_fullsized":"2024-03-21T14:23:32.787515+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:19:13.493205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:19:13.493205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":863,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.793848+0000","last_change":"2024-03-21T14:20:49.535756+0000","last_active":"2024-03-21T14:23:32.793848+0000","last_peered":"2024-03-21T14:23:32.793848+0000","last_clean":"2024-03-21T14:23:32.793848+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:23:32.793848+0000","last_undegraded":"2024-03-21T14:23:32.793848+0000","last_fullsized":"2024-03-21T14:23:32.793848+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_clean_scrub_stamp":"2024-03-21T14:20:49.535652+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:36.131856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":865,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.794191+0000","last_change":"2024-03-21T14:21:46.110127+0000","last_active":"2024-03-21T14:23:32.794191+0000","last_peered":"2024-03-21T14:23:32.794191+0000","last_clean":"2024-03-21T14:23:32.794191+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:23:32.794191+0000","last_undegraded":"2024-03-21T14:23:32.794191+0000","last_fullsized":"2024-03-21T14:23:32.794191+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.110008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.110008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:23:33.105359+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":908,"reported_epoch":450,"state":"active+clean","last_fresh":"2024-03-21T14:23:32.787733+0000","last_change":"2024-03-21T14:23:02.231025+0000","last_active":"2024-03-21T14:23:32.787733+0000","last_peered":"2024-03-21T14:23:32.787733+0000","last_clean":"2024-03-21T14:23:32.787733+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:23:32.787733+0000","last_undegraded":"2024-03-21T14:23:32.787733+0000","last_fullsized":"2024-03-21T14:23:32.787733+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:23:02.230890+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:23:02.230890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.516151+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3248488472,"num_objects":49569,"num_object_clones":0,"num_object_copies":148707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49569,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":172191,"num_write_kb":7096257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3767,"num_bytes_recovered":246874112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":9744166912,"data_stored":9744154696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70649,"ondisk_log_size":70649,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1198,"num_read_kb":1193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1542,"num_read_kb":1299,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":3812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1057,"ondisk_log_size":1057,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103425,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2111668,"kb_used_data":1223916,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":91633484,"statfs":{"total":95995035648,"available":93832687616,"internally_reserved":0,"allocated":1253289984,"data_stored":1252920282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":909048266},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":351,"seq":1507533520953,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473276,"kb_used_data":381812,"kb_used_omap":4,"kb_used_meta":91387,"kb_avail":93271876,"statfs":{"total":95995035648,"available":95510401024,"internally_reserved":0,"allocated":390975488,"data_stored":390660248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":93580964},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575137,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2683468,"kb_used_data":1785988,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":91061684,"statfs":{"total":95995035648,"available":93247164416,"internally_reserved":0,"allocated":1828851712,"data_stored":1828458025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":919009733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738662,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2054980,"kb_used_data":1203324,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":91690172,"statfs":{"total":95995035648,"available":93890736128,"internally_reserved":0,"allocated":1232203776,"data_stored":1231816914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":872085957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598292022,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892144,"kb_used_data":1537512,"kb_used_omap":7,"kb_used_meta":354616,"kb_avail":91853008,"statfs":{"total":95995035648,"available":94057480192,"internally_reserved":0,"allocated":1574412288,"data_stored":1574045745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":363126916},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182677,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2408272,"kb_used_data":1592392,"kb_used_omap":1,"kb_used_meta":815550,"kb_avail":91336880,"statfs":{"total":95995035648,"available":93528965120,"internally_reserved":0,"allocated":1630609408,"data_stored":1630211065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":835123655},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019153,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1944776,"kb_used_data":1029248,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":91800376,"statfs":{"total":95995035648,"available":94003585024,"internally_reserved":0,"allocated":1053949952,"data_stored":1053558995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692108,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1900096,"kb_used_data":1020664,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":91845056,"statfs":{"total":95995035648,"available":94049337344,"internally_reserved":0,"allocated":1045159936,"data_stored":1044778342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1217265664,"data_stored":1217265664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1817444352,"data_stored":1817444352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1404506112,"data_stored":1404502040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1608122368,"data_stored":1608122368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1045172224,"data_stored":1045168152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1026097152,"data_stored":1026097152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":387780608,"data_stored":387776536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1237778432,"data_stored":1237778432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:23:44.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-03-21T14:23:44.979 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:45.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:45.171 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:44 smithi118 ceph-mon[33862]: pgmap v996: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:45.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:44 smithi003 ceph-mon[29576]: pgmap v996: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:45.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:44 smithi003 ceph-mon[40510]: pgmap v996: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:45.274 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:45.402 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:45.444 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:45.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:45.546 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:45.575 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:45.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:45.720 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:45.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:45.850 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:45.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:45.980 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:45.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:46.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:46.120 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:45 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:23:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:45 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:23:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:45 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:23:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:45 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:23:46.222 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:46.250 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:46.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:23:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:23:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:23:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:23:46.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:23:46.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:23:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:23:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:45 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:23:46.352 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:46.380 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:46.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:46.482 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:46.523 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:46.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:46.654 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:23:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:23:46.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:46.802 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:46.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:46.904 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:46.930 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:46.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:47.032 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:47.076 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:47.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:47.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-mon[29576]: pgmap v997: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:47.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:23:47.174 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:23:47.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-mon[40510]: pgmap v997: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:47.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:23:47.175 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:46 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:23:47.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:46 smithi118 ceph-mon[33862]: pgmap v997: 158 pgs: 158 active+clean; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-21T14:23:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:46 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:23:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:46 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:23:47.206 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:47.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:47.308 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:47.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:47.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:47.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:47.492 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:47.594 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:47.633 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:47.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:47.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:47.762 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:47.864 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:47.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:47.900 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:23:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:23:47.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:48.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:47 smithi003 ceph-mon[29576]: 11.2 scrub starts 2024-03-21T14:23:48.054 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:48.054 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:47 smithi003 ceph-mon[40510]: 11.2 scrub starts 2024-03-21T14:23:48.054 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:48.137 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:48.164 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:48.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:47 smithi118 ceph-mon[33862]: 11.2 scrub starts 2024-03-21T14:23:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:23:48.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:48.291 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:48.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:48.411 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:23:48.411 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:23:48.412 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:23:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:23:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:23:48.413 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:23:48.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.430 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:48.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:48.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:48.566 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:48.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:48.668 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:48.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:48.809 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:48.839 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:48.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:48.969 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:48.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:48.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:23:49.071 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:49.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.123 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:49.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: pgmap v998: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 4.9 scrub starts 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 4.9 scrub ok 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:23:49.135 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2424896587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 1.a scrub starts 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 1.a scrub ok 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[29576]: 11.2 scrub ok 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: pgmap v998: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 4.9 scrub starts 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 4.9 scrub ok 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:23:49.136 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:23:49.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2424896587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:49.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 1.a scrub starts 2024-03-21T14:23:49.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 1.a scrub ok 2024-03-21T14:23:49.137 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:48 smithi003 ceph-mon[40510]: 11.2 scrub ok 2024-03-21T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: pgmap v998: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:23:49.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 4.9 scrub starts 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 4.9 scrub ok 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2424896587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 1.a scrub starts 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 1.a scrub ok 2024-03-21T14:23:49.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:48 smithi118 ceph-mon[33862]: 11.2 scrub ok 2024-03-21T14:23:49.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.278 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:49.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:49.379 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:49.410 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:49.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:49.515 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:49.545 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:49.647 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:49.676 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:49.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:49.777 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:49.806 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:49.908 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:49.937 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:49.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:49.949 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:23:49.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:23:49.951 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:49.951 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:49 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:50.039 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:50.087 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:49 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:49 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:49 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:49 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:49 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:49 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:50.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:50.223 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:50.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:50.372 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:50.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:50.423 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:50.423 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:50.474 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:50.501 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:50.520 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:23:50.520 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:23:50.520 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:23:50.520 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:23:50.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.520 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25232.0:220686 9.1 9:84ccbe40:::benchmark_data_smithi003_135814_object79952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e292)", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:15:47.467780+0000", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 483.04529577400001, 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13541199300000001, 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25232", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 220686 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467780+0000", 2024-03-21T14:23:50.521 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467780+0000", 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467781+0000", 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.452e-06 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.522 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467786+0000", 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2100000000000001e-06 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467798+0000", 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1807999999999999e-05 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467805+0000", 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9079999999999998e-06 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.523 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467855+0000", 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0420999999999997e-05 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.467948+0000", 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2130000000000001e-05 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,7", 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.468019+0000", 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1608999999999998e-05 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.524 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.470349+0000", 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023292899999999999 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.470695+0000", 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000346526 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.603076+0000", 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13238137 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.525 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.603137+0000", 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0977000000000002e-05 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:15:47.603192+0000", 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4292000000000001e-05 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.526 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:108678 10.5 10:aab15c17:::benchmark_data_smithi003_160672_object108677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:17:59.891846+0000", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 350.62122994399999, 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13512718600000001, 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 108678 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891846+0000", 2024-03-21T14:23:50.527 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891846+0000", 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891847+0000", 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.164e-06 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.528 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891867+0000", 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9967e-05 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891878+0000", 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0892e-05 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891884+0000", 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0850000000000002e-06 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.529 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.891924+0000", 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0411000000000003e-05 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.892032+0000", 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107944 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0", 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.892125+0000", 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3267e-05 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.530 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:17:59.894416+0000", 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0022907280000000001 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.026250+0000", 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.13183368200000001 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.026878+0000", 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00062871599999999998 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.531 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:00.026973+0000", 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.433e-05 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:140848 10.f 10:f38721bb:::benchmark_data_smithi003_160672_object140847:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:18:22.278080+0000", 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.234995556, 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12600556399999999, 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.532 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140848 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.278080+0000", 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.278080+0000", 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.533 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.278080+0000", 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0500000000000004e-07 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.278102+0000", 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1744999999999999e-05 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.278109+0000", 2024-03-21T14:23:50.534 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9589999999999998e-06 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.278115+0000", 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8000000000000004e-06 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.281445+0000", 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0033303230000000001 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.281569+0000", 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012420199999999999 2024-03-21T14:23:50.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4", 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.281729+0000", 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015953199999999999 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.287276+0000", 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0055472179999999996 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.403912+0000", 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.116636103 2024-03-21T14:23:50.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.403985+0000", 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2899999999999997e-05 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.404085+0000", 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000100277 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.537 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:140854 10.f 10:f3bc8973:::benchmark_data_smithi003_160672_object140853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:18:22.283334+0000", 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.22974133500003, 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.12105043, 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140854 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.538 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.283334+0000", 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.283334+0000", 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.283335+0000", 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2720000000000001e-06 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.539 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.283359+0000", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.4088e-05 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.283371+0000", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.146e-05 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.283379+0000", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6869999999999994e-06 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.540 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.284582+0000", 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0012034280000000001 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.284729+0000", 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014732999999999999 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4", 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.284875+0000", 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014536400000000001 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.541 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.288788+0000", 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0039134149999999999 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.404166+0000", 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.11537744699999999 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.404260+0000", 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4050999999999998e-05 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.542 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.404385+0000", 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000124888 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:140858 10.f 10:fadf26d2:::benchmark_data_smithi003_160672_object140857:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:18:22.286151+0000", 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.22692473000001, 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.121422986, 2024-03-21T14:23:50.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140858 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.286151+0000", 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.544 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.286151+0000", 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.286152+0000", 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.359e-06 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.286175+0000", 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2821999999999999e-05 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.545 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.286192+0000", 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6928000000000001e-05 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.286201+0000", 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2830000000000007e-06 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.546 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.287298+0000", 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001096781 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.287401+0000", 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010272000000000001 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4", 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.287496+0000", 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5156999999999997e-05 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.547 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.288905+0000", 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001409703 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.407414+0000", 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.118508355 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.407475+0000", 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1525000000000005e-05 2024-03-21T14:23:50.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.407574+0000", 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8352999999999998e-05 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:140861 10.f 10:fb9fd294:::benchmark_data_smithi003_160672_object140860:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:18:22.289149+0000", 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 328.223926909, 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.118800504, 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 140861 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.550 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289149+0000", 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289149+0000", 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289149+0000", 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0200000000000002e-07 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.551 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289167+0000", 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.7520000000000002e-05 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289174+0000", 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8979999999999999e-06 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289179+0000", 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9540000000000003e-06 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.552 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289225+0000", 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6801000000000001e-05 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289342+0000", 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116461 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4", 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.289488+0000", 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014660199999999999 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.553 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.291491+0000", 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020024779999999998 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.407616+0000", 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.116125467 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.407662+0000", 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6100999999999998e-05 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.554 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:18:22.407949+0000", 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028662000000000002 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250404 10.f 10:fd6011dd:::benchmark_data_smithi003_160672_object97028:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.296074+0000", 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.21700146699999, 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.16465118200000001, 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.555 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250404 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296074+0000", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296074+0000", 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296075+0000", 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.164e-06 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296080+0000", 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6750000000000001e-06 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296089+0000", 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4609999999999997e-06 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.557 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296106+0000", 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6524e-05 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296439+0000", 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033300499999999999 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296523+0000", 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4083000000000001e-05 2024-03-21T14:23:50.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.296605+0000", 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2199000000000004e-05 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.300200+0000", 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0035946200000000002 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.559 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.303083+0000", 2024-03-21T14:23:50.560 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002883072 2024-03-21T14:23:50.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.560 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.560 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.560 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.460557+0000", 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.157474217 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.460619+0000", 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1607999999999997e-05 2024-03-21T14:23:50.561 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.460725+0000", 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106554 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250409 10.f 10:f7a31467:::benchmark_data_smithi003_160672_object97033:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.305276+0000", 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 250.20779941999999, 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.18121155899999999, 2024-03-21T14:23:50.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250409 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305276+0000", 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.563 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305276+0000", 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305277+0000", 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6799999999999999e-07 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305281+0000", 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.0559999999999998e-06 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.564 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305290+0000", 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7849999999999997e-06 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305295+0000", 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7359999999999998e-06 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.565 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.305338+0000", 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2976999999999999e-05 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.457196+0000", 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.15185795699999999 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.457365+0000", 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016897700000000001 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.566 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.460810+0000", 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034448149999999999 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.475241+0000", 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014430899000000001 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.486305+0000", 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011063498 2024-03-21T14:23:50.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.486374+0000", 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9898000000000001e-05 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.486488+0000", 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000113093 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.568 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:252093 10.f 10:f724799d:::benchmark_data_smithi003_160672_object98717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:42.665309+0000", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 247.84776600699999, 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.23471895800000001, 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 252093 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665309+0000", 2024-03-21T14:23:50.569 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665309+0000", 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665311+0000", 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3710000000000001e-06 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.570 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665317+0000", 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7760000000000003e-06 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665327+0000", 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0208e-05 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665334+0000", 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5519999999999999e-06 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665429+0000", 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4449000000000003e-05 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665558+0000", 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012941100000000001 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 2,4", 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.665690+0000", 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013136499999999999 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899820+0000", 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.234130264 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899841+0000", 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.1129e-05 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899887+0000", 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6418999999999997e-05 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899934+0000", 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6411e-05 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.900028+0000", 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.4603000000000002e-05 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.574 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15666.0:252093 10.f e372/359 ondisk, result = 0)", 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:42.669882+0000", 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 247.84319302099999, 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22995980999999999, 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.669882+0000", 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.669882+0000", 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.669883+0000", 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5300000000000003e-07 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.669889+0000", 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2610000000000001e-06 2024-03-21T14:23:50.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898255+0000", 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22836664100000001 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898267+0000", 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1405e-05 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899802+0000", 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001535474 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899812+0000", 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8709999999999992e-06 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899842+0000", 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.0304999999999999e-05 2024-03-21T14:23:50.580 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop_reply(client.15666.0:252093 10.f e372/359 ondisk, result = 0)", 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:42.673953+0000", 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 247.83912284499999, 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22608161800000001, 2024-03-21T14:23:50.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.582 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "started", 2024-03-21T14:23:50.582 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.673953+0000", 2024-03-21T14:23:50.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.673953+0000", 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.673954+0000", 2024-03-21T14:23:50.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2270000000000001e-06 2024-03-21T14:23:50.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.673962+0000", 2024-03-21T14:23:50.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.994e-06 2024-03-21T14:23:50.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898253+0000", 2024-03-21T14:23:50.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22429069099999999 2024-03-21T14:23:50.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898267+0000", 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4116e-05 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899873+0000", 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0016067309999999999 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.588 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.899881+0000", 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.96e-06 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.900034+0000", 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015289899999999999 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_repop(client.15666.0:252123 10.5 e372/359 10:a08020fb:::benchmark_data_smithi003_160672_object98747:head v 372'15880, mlcod=372'15878)", 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:42.684115+0000", 2024-03-21T14:23:50.589 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 247.828960666, 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.22218159100000001, 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.684115+0000", 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.684115+0000", 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.684116+0000", 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1909999999999999e-06 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.684126+0000", 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0332999999999999e-05 2024-03-21T14:23:50.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898266+0000", 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.21413965500000001 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898277+0000", 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0983e-05 2024-03-21T14:23:50.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898683+0000", 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00040561200000000002 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.898710+0000", 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.6965e-05 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.906253+0000", 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0075435010000000002 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:42.906296+0000", 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3350999999999999e-05 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:25383 11.1 11:8e5f66b4:::benchmark_data_smithi003_188959_object25382:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e426)", 2024-03-21T14:23:50.594 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:29.700841+0000", 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 140.81223430700001, 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.114646415, 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 25383 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.700841+0000", 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.700841+0000", 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.700843+0000", 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.519e-06 2024-03-21T14:23:50.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.700865+0000", 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2799e-05 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.700876+0000", 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0919e-05 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.700884+0000", 2024-03-21T14:23:50.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2699999999999999e-06 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.763945+0000", 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.063061450000000005 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.764129+0000", 2024-03-21T14:23:50.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018381199999999999 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.764301+0000", 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017169799999999999 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.769547+0000", 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052464299999999998 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.770195+0000", 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00064761400000000002 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.815330+0000", 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.045135610999999999 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.815432+0000", 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000102149 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:29.815488+0000", 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5143999999999999e-05 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:144765 11.b 11:d55829b3:::benchmark_data_smithi003_188959_object33883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-03-21T14:23:50.601 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:13.460506+0000", 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 37.052569708999997, 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.078747284000000001, 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 144765 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460506+0000", 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460506+0000", 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460507+0000", 2024-03-21T14:23:50.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1100000000000004e-07 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460512+0000", 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9470000000000002e-06 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460522+0000", 2024-03-21T14:23:50.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0818e-05 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460530+0000", 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6909999999999997e-06 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460612+0000", 2024-03-21T14:23:50.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2132000000000006e-05 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460721+0000", 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010888 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:23:50.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.460828+0000", 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000107324 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.465528+0000", 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0047000289999999997 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.465541+0000", 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2473999999999999e-05 2024-03-21T14:23:50.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.539161+0000", 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.073619854999999998 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.539215+0000", 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3962999999999998e-05 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.539253+0000", 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8260000000000003e-05 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:145294 11.8 11:1445210b:::benchmark_data_smithi003_188959_object34412:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:13.968602+0000", 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 36.544473312999997, 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.076216217000000003, 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.609 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 145294 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968602+0000", 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968602+0000", 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968603+0000", 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2359999999999999e-06 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968609+0000", 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.9769999999999997e-06 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968620+0000", 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0969e-05 2024-03-21T14:23:50.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968629+0000", 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5310000000000001e-06 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968694+0000", 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5320999999999993e-05 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968810+0000", 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011624 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.968936+0000", 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012528600000000001 2024-03-21T14:23:50.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.971867+0000", 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029315629999999999 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.044546+0000", 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.072678861999999997 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.044710+0000", 2024-03-21T14:23:50.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016417499999999999 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.044773+0000", 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2367999999999996e-05 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.044818+0000", 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5689000000000003e-05 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:145299 11.1 11:8c0d896d:::benchmark_data_smithi003_188959_object34417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e446)", 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:13.972417+0000", 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 36.540658706000002, 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.095673913999999999, 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.617 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 145299 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.972417+0000", 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.972417+0000", 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.972417+0000", 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0299999999999998e-07 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:13.972421+0000", 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8369999999999999e-06 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.042829+0000", 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.070407564000000006 2024-03-21T14:23:50.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.042845+0000", 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5923e-05 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.043006+0000", 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016120800000000001 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.043497+0000", 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00049064099999999999 2024-03-21T14:23:50.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,2", 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.043607+0000", 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110031 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.050783+0000", 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0071760620000000004 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.058985+0000", 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0082022469999999993 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.067963+0000", 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089775610000000002 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.068042+0000", 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9401999999999996e-05 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:14.068091+0000", 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8735e-05 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.622 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:186961 11.a 11:5143f514:::benchmark_data_smithi003_188959_object76079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:50.421702+0000", 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.091373161999999994, 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.056239957, 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 186961 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.623 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421702+0000", 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421702+0000", 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421704+0000", 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.838e-06 2024-03-21T14:23:50.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421712+0000", 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8140000000000001e-06 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421727+0000", 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5418999999999999e-05 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421736+0000", 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4800000000000001e-06 2024-03-21T14:23:50.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421845+0000", 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108749 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.421993+0000", 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014844899999999999 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.422114+0000", 2024-03-21T14:23:50.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120998 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.431277+0000", 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091626309999999992 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.451137+0000", 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.019860577000000001 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.477788+0000", 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026650845999999999 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.477892+0000", 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010380899999999999 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.477942+0000", 2024-03-21T14:23:50.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0346999999999998e-05 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:186967 11.a 11:53d88ca4:::benchmark_data_smithi003_188959_object76085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:50.424672+0000", 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.088403647000000002, 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.053427005, 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.629 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 186967 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.424672+0000", 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.424672+0000", 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.424673+0000", 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2559999999999999e-06 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.424678+0000", 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0839999999999999e-06 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.424687+0000", 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8999999999999995e-06 2024-03-21T14:23:50.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.424695+0000", 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9559999999999997e-06 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.431313+0000", 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0066175160000000004 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.431428+0000", 2024-03-21T14:23:50.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115129 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.431538+0000", 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000110684 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.442166+0000", 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010627945999999999 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.451159+0000", 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089926620000000002 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.477981+0000", 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026822006999999998 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.478060+0000", 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9226000000000003e-05 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.478099+0000", 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8639000000000001e-05 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:186968 11.9 11:956378f7:::benchmark_data_smithi003_188959_object76086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:50.425471+0000", 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.087604556, 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.050177290999999999, 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.635 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 186968 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.425471+0000", 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.425471+0000", 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.425472+0000", 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.1500000000000003e-07 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.425477+0000", 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9740000000000001e-06 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.425487+0000", 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0219e-05 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.425495+0000", 2024-03-21T14:23:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.8290000000000008e-06 2024-03-21T14:23:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.435312+0000", 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0098171960000000003 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.435424+0000", 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011155299999999999 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,1", 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.435535+0000", 2024-03-21T14:23:50.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000111258 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.442521+0000", 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0069862420000000001 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.442559+0000", 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.7490000000000002e-05 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.474963+0000", 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.032404276000000003 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.475216+0000", 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00025318299999999998 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.475648+0000", 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000432156 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.641 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:186974 11.a 11:5534a63d:::benchmark_data_smithi003_188959_object76092:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:23:50.430899+0000", 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 0.082176492000000004, 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.047348888999999998, 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 186974 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:23:50.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.430899+0000", 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.430899+0000", 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.430901+0000", 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.99e-06 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.430909+0000", 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6550000000000004e-06 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.430931+0000", 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2362e-05 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.430941+0000", 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0227999999999999e-05 2024-03-21T14:23:50.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.433829+0000", 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.002887489 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.434314+0000", 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00048567700000000001 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.434447+0000", 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000132676 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.442206+0000", 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0077589 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.451329+0000", 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0091226280000000007 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.478136+0000", 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.026807348000000002 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.478212+0000", 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5986000000000001e-05 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:23:50.478248+0000", 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.595e-05 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.647 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:23:50.648 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:23:50.648 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:23:50.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.649 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:50.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:50.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:50.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:50.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:50.892 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:50.920 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:50.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:50.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:51.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:51.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:51.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: pgmap v999: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:23:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:50 smithi118 ceph-mon[33862]: osdmap e452: 8 total, 8 up, 8 in 2024-03-21T14:23:51.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:23:51.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: pgmap v999: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[29576]: osdmap e452: 8 total, 8 up, 8 in 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: pgmap v999: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:23:51.227 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:23:51.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:23:51.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:23:51.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:23:51.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:50 smithi003 ceph-mon[40510]: osdmap e452: 8 total, 8 up, 8 in 2024-03-21T14:23:51.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:51.308 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:51.410 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.449 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:51.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:51.550 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:51.577 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:51.679 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:51.715 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:51.816 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:51.845 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:51.947 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:51.986 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:51.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:52.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:52.134 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:52.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:51 smithi118 ceph-mon[33862]: 11.c scrub starts 2024-03-21T14:23:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:51 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:23:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:51 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:23:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:51 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:23:52.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:51 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:23:52.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[29576]: 11.c scrub starts 2024-03-21T14:23:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:23:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:23:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:23:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:23:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[40510]: 11.c scrub starts 2024-03-21T14:23:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:23:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:23:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:23:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:51 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:23:52.259 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:52.361 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:52.392 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:52.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:52.494 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:52.519 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:52.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:52.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:52.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:52.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:52.922 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:52.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:53.022 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:53.048 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:53.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:52 smithi118 ceph-mon[33862]: pgmap v1001: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 818 op/s 2024-03-21T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:52 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:52 smithi118 ceph-mon[33862]: 11.c scrub ok 2024-03-21T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:52 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:52 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:23:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:52 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:23:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:53.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[29576]: pgmap v1001: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 818 op/s 2024-03-21T14:23:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:23:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[29576]: 11.c scrub ok 2024-03-21T14:23:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:23:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:23:53.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:23:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[40510]: pgmap v1001: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 601 B/s rd, 0 B/s wr, 818 op/s 2024-03-21T14:23:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:23:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[40510]: 11.c scrub ok 2024-03-21T14:23:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:23:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:23:53.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:52 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:23:53.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:53.317 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:53.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:53.419 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:53.459 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:53.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:53.499 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-21T14:23:53.562 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:53.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.599 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:53.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:53.700 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:53.744 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:53.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:53.817 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:23:53.846 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:53.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:53.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:53.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:53.985 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:54.016 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:54.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:54.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:54.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:23:54.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:23:54.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:23:54.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:23:54.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:23:54.220 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:23:54.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:23:54.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:23:54.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:23:54.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:23:54.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:23:54.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:53 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:23:54.268 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:54.275 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:53 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:23:54.276 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:53 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:23:54.276 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:53 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:23:54.276 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:53 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:23:54.276 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:53 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:23:54.276 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:53 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:23:54.299 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:54.401 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:54.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:54.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:54.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:54.561 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:54.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:54.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:54.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:54.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:54.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:54.948 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:54.976 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:54.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:54.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:55.078 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:55.110 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:55.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.112 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:55.199 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: pgmap v1002: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: osdmap e453: 8 total, 8 up, 8 in 2024-03-21T14:23:55.200 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:54 smithi118 ceph-mon[33862]: 11.6 scrub starts 2024-03-21T14:23:55.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:55.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:55.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: pgmap v1002: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T14:23:55.202 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: osdmap e453: 8 total, 8 up, 8 in 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[29576]: 11.6 scrub starts 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: pgmap v1002: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:55.203 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:55.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:55.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: osdmap e453: 8 total, 8 up, 8 in 2024-03-21T14:23:55.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:54 smithi003 ceph-mon[40510]: 11.6 scrub starts 2024-03-21T14:23:55.212 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:55.239 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:55.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:55.374 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:55.476 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:55.502 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:55.604 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:55.634 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:55.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:55.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:55.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.792 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:55.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:55.893 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:55.917 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:55.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:55.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[29576]: 1.7 scrub starts 2024-03-21T14:23:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[29576]: 1.7 scrub ok 2024-03-21T14:23:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[29576]: 11.6 scrub ok 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[40510]: 1.7 scrub starts 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[40510]: 1.7 scrub ok 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:23:56.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:55 smithi003 ceph-mon[40510]: 11.6 scrub ok 2024-03-21T14:23:56.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:56.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:56.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:56.148 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:56.185 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:56.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:55 smithi118 ceph-mon[33862]: 1.7 scrub starts 2024-03-21T14:23:56.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:55 smithi118 ceph-mon[33862]: 1.7 scrub ok 2024-03-21T14:23:56.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:55 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:23:56.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:55 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:23:56.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:55 smithi118 ceph-mon[33862]: 11.6 scrub ok 2024-03-21T14:23:56.287 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.329 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:56.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:56.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:56.460 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:193477 11.f 11:f5e018cb:::benchmark_data_smithi003_188959_object82595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e454)", 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:23:56.487821+0000", 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.012322912, 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.012356371, 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:23:56.501 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 193477 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487821+0000", 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487821+0000", 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:23:56.502 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487822+0000", 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.569999999999999e-07 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487825+0000", 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.4869999999999998e-06 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487833+0000", 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8560000000000007e-06 2024-03-21T14:23:56.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487838+0000", 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0590000000000002e-06 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.487890+0000", 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1694999999999999e-05 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.504 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.488027+0000", 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000137329 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,7", 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.488143+0000", 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115937 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.490365+0000", 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022223619999999999 2024-03-21T14:23:56.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:23:56.490713+0000", 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000348024 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 1 2024-03-21T14:23:56.506 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:23:56.562 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:23:56.593 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:23:56.695 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:56.722 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:56.726 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:23:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:23:56.824 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:56.872 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:56.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:56.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[29576]: pgmap v1004: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[29576]: osdmap e454: 8 total, 8 up, 8 in 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[40510]: pgmap v1004: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T14:23:56.996 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:23:56.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:23:56.997 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:56 smithi003 ceph-mon[40510]: osdmap e454: 8 total, 8 up, 8 in 2024-03-21T14:23:57.024 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:57.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:57.143 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:23:57.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:57.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:57.220 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:56 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:23:57.220 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:56 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:23:57.220 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:56 smithi118 ceph-mon[33862]: pgmap v1004: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-21T14:23:57.220 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:56 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:23:57.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:56 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:23:57.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:56 smithi118 ceph-mon[33862]: osdmap e454: 8 total, 8 up, 8 in 2024-03-21T14:23:57.266 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:57.301 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:57.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:57.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:57.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:57.570 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:57.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:57.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:23:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:57.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:23:57.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:57.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:23:57.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-21T14:23:57.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:57.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:57.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:23:57.963 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:23:57.991 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:23:57.993 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:57.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:57.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:57.995 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 11 pgp_num to 16 2024-03-21T14:23:58.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:58.133 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:58.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3678996660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:58.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: 4.3 scrub starts 2024-03-21T14:23:58.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: 4.3 scrub ok 2024-03-21T14:23:58.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3176476378' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T14:23:58.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:57 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T14:23:58.187 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:23:58.227 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3678996660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: 4.3 scrub starts 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: 4.3 scrub ok 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3176476378' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:23:58.228 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:23:58.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:23:58.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3678996660' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:23:58.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: 4.3 scrub starts 2024-03-21T14:23:58.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: 4.3 scrub ok 2024-03-21T14:23:58.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3176476378' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T14:23:58.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:57 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-21T14:23:58.235 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:58.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.272 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:58.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:58.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:23:58.410 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:58.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:23:58.512 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:23:58.547 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:23:58.649 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:58.685 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:58.786 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:58.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:58.925 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:58.964 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:58.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: pgmap v1006: 158 pgs: 158 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 897 op/s 2024-03-21T14:23:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:23:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:23:59.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3678996660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[29576]: osdmap e455: 8 total, 8 up, 8 in 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: pgmap v1006: 158 pgs: 158 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 897 op/s 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3678996660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-21T14:23:59.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:58 smithi003 ceph-mon[40510]: osdmap e455: 8 total, 8 up, 8 in 2024-03-21T14:23:59.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:59.095 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:23:59.197 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:23:59.224 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:23:59.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:59.363 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:59.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:59.369 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: pgmap v1006: 158 pgs: 158 active+clean; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 686 B/s rd, 0 B/s wr, 897 op/s 2024-03-21T14:23:59.369 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:23:59.370 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:23:59.370 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:23:59.370 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:23:59.370 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3678996660' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:23:59.370 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-21T14:23:59.370 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:58 smithi118 ceph-mon[33862]: osdmap e455: 8 total, 8 up, 8 in 2024-03-21T14:23:59.464 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:23:59.505 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:59.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:23:59.606 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:59.654 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:59.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:59.756 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:23:59.817 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:23:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:23:59.919 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:23:59.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:23:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:23:59.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:00.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:00.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.105 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:00.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:00.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:00.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:59 smithi003 ceph-mon[29576]: 4.1f deep-scrub starts 2024-03-21T14:24:00.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:23:59 smithi003 ceph-mon[29576]: 4.1f deep-scrub ok 2024-03-21T14:24:00.235 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:59 smithi003 ceph-mon[40510]: 4.1f deep-scrub starts 2024-03-21T14:24:00.235 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:23:59 smithi003 ceph-mon[40510]: 4.1f deep-scrub ok 2024-03-21T14:24:00.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:00.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:00.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:59 smithi118 ceph-mon[33862]: 4.1f deep-scrub starts 2024-03-21T14:24:00.255 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:23:59 smithi118 ceph-mon[33862]: 4.1f deep-scrub ok 2024-03-21T14:24:00.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:00.378 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:00.424 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:24:00.424 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:24:00.424 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:24:00.424 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:24:00.425 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:24:00.480 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:00.488 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:24:00.506 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:00.531 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:24:00.574 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:24:00.608 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:00.618 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:24:00.646 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:00.656 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:24:00.697 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:24:00.738 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:24:00.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:00.781 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:24:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:24:00.791 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:00.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:00.936 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:00.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:01.038 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.078 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:01.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:01.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: pgmap v1008: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 888 op/s 2024-03-21T14:24:01.103 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: 11.7 deep-scrub starts 2024-03-21T14:24:01.104 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:24:01.104 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:24:01.104 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:24:01.104 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:24:01.104 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2994303415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:01.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-21T14:24:01.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: pgmap v1008: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 888 op/s 2024-03-21T14:24:01.154 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: 11.7 deep-scrub starts 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2994303415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: pgmap v1008: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 888 op/s 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: 11.7 deep-scrub starts 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:24:01.155 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2994303415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:01.180 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:01.222 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:01.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:01.324 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:01.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.361 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:01.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:01.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:01.480 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.504 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:01.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:01.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:01.633 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:01.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:01.735 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:01.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:01.873 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:01.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:02.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:02.047 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:02.149 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:01 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:24:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:01 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:24:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:01 smithi118 ceph-mon[33862]: 11.d scrub starts 2024-03-21T14:24:02.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:01 smithi118 ceph-mon[33862]: 11.d scrub ok 2024-03-21T14:24:02.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.183 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:02.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:24:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:24:02.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[29576]: 11.d scrub starts 2024-03-21T14:24:02.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[29576]: 11.d scrub ok 2024-03-21T14:24:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:24:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:24:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[40510]: 11.d scrub starts 2024-03-21T14:24:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:01 smithi003 ceph-mon[40510]: 11.d scrub ok 2024-03-21T14:24:02.285 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:02.316 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:02.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:02.418 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:02.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.452 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:02.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:02.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:02.592 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:02.694 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.729 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:02.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:02.830 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:02.862 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:02.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:02.964 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:03.005 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:03.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:03.107 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:03.137 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:03.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:03.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:03 smithi003 ceph-mon[29576]: pgmap v1009: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 717 op/s 2024-03-21T14:24:03.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:03 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:03.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:03.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:03 smithi003 ceph-mon[40510]: pgmap v1009: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 717 op/s 2024-03-21T14:24:03.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:03 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:03.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:03.282 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:03.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:03.328 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:03 smithi118 ceph-mon[33862]: pgmap v1009: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 717 op/s 2024-03-21T14:24:03.328 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:03 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:03.329 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:03.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:03.395 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:03.395 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:03.421 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:03.523 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:03.552 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:03.654 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:03.683 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:03.785 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:03.817 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:03.920 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:03.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:03.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:03.978 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-21T14:24:04.057 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:04.091 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-21T14:24:04.104 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:04.206 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:04.247 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:04.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:04.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:04 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:24:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:04 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:24:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:04 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:24:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:04 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:24:04.434 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:04.449 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:24:04.500 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:04.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:24:04.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:24:04.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:24:04.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:24:04.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:24:04.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:24:04.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:24:04.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:04 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:24:04.529 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:04.631 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:04.693 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:04.795 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.824 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:04.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:04.925 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:04.955 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:04.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:05.046 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[29576]: pgmap v1010: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T14:24:05.046 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:24:05.046 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[29576]: 5.9 scrub starts 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[29576]: 5.9 scrub ok 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[40510]: pgmap v1010: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[40510]: 5.9 scrub starts 2024-03-21T14:24:05.047 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:05 smithi003 ceph-mon[40510]: 5.9 scrub ok 2024-03-21T14:24:05.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:05.084 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:05.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:05 smithi118 ceph-mon[33862]: pgmap v1010: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-03-21T14:24:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:05 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:24:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:05 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:24:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:05 smithi118 ceph-mon[33862]: 5.9 scrub starts 2024-03-21T14:24:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:05 smithi118 ceph-mon[33862]: 5.9 scrub ok 2024-03-21T14:24:05.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:05.334 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:05.366 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:05.468 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:05.497 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:05.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:05.599 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:05.635 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:05.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.776 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:05.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:05.858 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 7, 6, 2] 2024-03-21T14:24:05.858 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-21T14:24:05.859 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-21T14:24:05.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-21T14:24:05.877 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:05.907 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:05.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:05.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:06.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:06.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.042 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:06.102 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:06.143 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:06.170 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:06.272 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:06.304 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:06.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:06 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:24:06.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:06 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:24:06.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:06 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:24:06.313 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:06 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:24:06.325 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:06 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:24:06.325 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:06 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:24:06.401 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:06.401 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:06.407 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:06.439 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:06.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:06.570 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:06.624 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:24:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:24:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:24:06.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.716 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:06.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:06.817 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:06.844 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:06.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:06.973 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:06.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:07.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:07.101 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:07.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:07.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-21T14:24:07.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:07.242 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:07.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:07.343 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:07.377 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:07.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:07.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: pgmap v1011: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-21T14:24:07.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: 6.1 deep-scrub starts 2024-03-21T14:24:07.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: 6.1 deep-scrub ok 2024-03-21T14:24:07.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: 3.1a scrub starts 2024-03-21T14:24:07.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: 3.1a scrub ok 2024-03-21T14:24:07.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: 6.1c scrub starts 2024-03-21T14:24:07.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-mon[33862]: 6.1c scrub ok 2024-03-21T14:24:07.478 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:07.486 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: pgmap v1011: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-21T14:24:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: 6.1 deep-scrub starts 2024-03-21T14:24:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: 6.1 deep-scrub ok 2024-03-21T14:24:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: 3.1a scrub starts 2024-03-21T14:24:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: 3.1a scrub ok 2024-03-21T14:24:07.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: 6.1c scrub starts 2024-03-21T14:24:07.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[29576]: 6.1c scrub ok 2024-03-21T14:24:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: pgmap v1011: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s 2024-03-21T14:24:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: 6.1 deep-scrub starts 2024-03-21T14:24:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: 6.1 deep-scrub ok 2024-03-21T14:24:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: 3.1a scrub starts 2024-03-21T14:24:07.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: 3.1a scrub ok 2024-03-21T14:24:07.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: 6.1c scrub starts 2024-03-21T14:24:07.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:07 smithi003 ceph-mon[40510]: 6.1c scrub ok 2024-03-21T14:24:07.518 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:07.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:07.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:07.661 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:07.763 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.812 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:07.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:07.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:24:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:24:07.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:07.956 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:07.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:08.038 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is set 2024-03-21T14:24:08.058 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:08.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:08.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:08.121 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: 6.1d deep-scrub starts 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: 6.1d deep-scrub ok 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2157735120' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:24:08.122 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:08 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:24:08.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:08.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.238 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:08.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:08.340 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:08.367 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:08.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:08.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: 6.1d deep-scrub starts 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: 6.1d deep-scrub ok 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2157735120' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:24:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: 6.1d deep-scrub starts 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: 6.1d deep-scrub ok 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2157735120' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:24:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:08 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:24:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:08.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:08.616 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:08.646 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:08.748 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:08.778 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:08.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:08.921 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:08.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:09.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.062 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:09.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:09.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:09.191 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:09.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:09.282 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: pgmap v1012: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-21T14:24:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: 5.17 scrub starts 2024-03-21T14:24:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: 5.17 scrub ok 2024-03-21T14:24:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: 11.7 deep-scrub ok 2024-03-21T14:24:09.283 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:09.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:24:09.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[29576]: osdmap e456: 8 total, 8 up, 8 in 2024-03-21T14:24:09.284 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: pgmap v1012: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-21T14:24:09.284 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: 5.17 scrub starts 2024-03-21T14:24:09.284 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: 5.17 scrub ok 2024-03-21T14:24:09.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:09.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: 11.7 deep-scrub ok 2024-03-21T14:24:09.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:09.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:24:09.285 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:09 smithi003 ceph-mon[40510]: osdmap e456: 8 total, 8 up, 8 in 2024-03-21T14:24:09.294 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:09.303 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: pgmap v1012: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 1.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-21T14:24:09.303 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: 5.17 scrub starts 2024-03-21T14:24:09.303 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: 5.17 scrub ok 2024-03-21T14:24:09.303 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:09.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: 11.7 deep-scrub ok 2024-03-21T14:24:09.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:09.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-21T14:24:09.304 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:09 smithi118 ceph-mon[33862]: osdmap e456: 8 total, 8 up, 8 in 2024-03-21T14:24:09.324 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:09.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:09.469 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:09.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:09.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:09.608 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:09.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:09.710 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:09.733 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:09.733 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:09.734 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:09.835 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:09.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:09.873 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:09.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:09.974 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:10.002 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:10.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:10.075 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:10.104 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:10.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:10.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:10.275 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:10.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:10.377 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:10.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:10.533 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-21T14:24:10.545 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:10.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:10.688 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:10.790 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.823 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:10.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:10.857 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:10.924 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:10.951 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:10.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:10.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:11.053 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:11.082 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:11.184 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:11.219 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:11.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:11 smithi003 ceph-mon[29576]: pgmap v1014: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T14:24:11.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:11 smithi003 ceph-mon[40510]: pgmap v1014: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T14:24:11.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:11.360 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:11.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:11 smithi118 ceph-mon[33862]: pgmap v1014: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T14:24:11.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.497 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:11.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:11.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:11.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:11.741 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:11.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.782 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:11.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:11.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:11.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:12.012 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:12.046 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:12.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:12.193 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:12.295 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:12.323 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:12.425 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.479 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:12.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:12.580 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:12.625 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:12.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:12.726 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:12.771 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:12.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:12.872 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:12.911 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:12.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:13.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:13.050 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:13.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:13.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:13.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:13 smithi003 ceph-mon[40510]: pgmap v1015: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T14:24:13.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:13 smithi003 ceph-mon[29576]: pgmap v1015: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T14:24:13.185 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:13.203 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:13 smithi118 ceph-mon[33862]: pgmap v1015: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-21T14:24:13.287 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:13.315 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:13.400 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:24:13.417 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.456 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:13.456 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:13.456 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:24:13.498 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:24:13.530 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:13.530 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:13.531 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:24:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=cleanup t=2024-03-21T14:24:13.449474157Z level=info msg="Completed cleanup jobs" duration=64.046399ms 2024-03-21T14:24:13.544 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:24:13.557 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:13.584 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:24:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.613 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:13.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:13.625 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:24:13.667 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:24:13.710 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:24:13.713 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:24:13.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:13.787 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:13.860 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:24:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafana.update.checker t=2024-03-21T14:24:13.544543017Z level=info msg="Update check succeeded" duration=74.752252ms 2024-03-21T14:24:13.860 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:24:13 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=plugins.update.checker t=2024-03-21T14:24:13.556855257Z level=info msg="Update check succeeded" duration=87.050988ms 2024-03-21T14:24:13.888 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:13.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:13.933 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:13.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:13.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-21T14:24:14.034 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:14.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:14.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:14 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2340374607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:14.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:14 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:24:14.182 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:14.220 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:14.232 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-21T14:24:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:14 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2340374607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:14 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:24:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:14 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2340374607' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:14.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:14 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:24:14.322 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:14.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.386 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:14.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:14.487 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:14.520 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:14.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:14.602 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:14.623 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:14.656 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:14.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:14.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:14.810 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:14.912 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:14.953 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:14.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:14.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:15.055 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:15.087 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:15.189 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:15.216 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:15.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:15.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:15.430 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:15 smithi003 ceph-mon[29576]: pgmap v1016: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:15.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:15 smithi003 ceph-mon[40510]: pgmap v1016: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:15.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:15 smithi118 ceph-mon[33862]: pgmap v1016: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 707 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:15.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.495 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:15.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:15.596 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:15.627 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:15.729 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:15.756 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:15.859 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:15.913 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:15.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:16.015 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:16.050 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:16.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:16.147 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:16.151 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.198 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:16.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:16.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:16.343 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:16.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:16.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:16.472 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:16.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:16.600 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:24:16.600 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:214670 11.3 11:c7d24be7:::benchmark_data_smithi003_188959_object103788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:24:16.587893+0000", 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.011418803, 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.011454351999999999, 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 214670 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.601 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587893+0000", 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587893+0000", 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587895+0000", 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.573e-06 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.602 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587901+0000", 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2310000000000001e-06 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587926+0000", 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4887000000000001e-05 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587935+0000", 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5049999999999996e-06 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.603 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.587985+0000", 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0121999999999999e-05 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.588078+0000", 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2872000000000001e-05 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.588414+0000", 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00033545899999999997 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.604 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.590641+0000", 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0022276010000000001 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.591336+0000", 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00069474800000000002 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:214676 11.4 11:2b183a35:::benchmark_data_smithi003_188959_object103794:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-21T14:24:16.605 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:24:16.597506+0000", 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.001805699, 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019610600000000001, 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 214676 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597506+0000", 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.606 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597506+0000", 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597508+0000", 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.641e-06 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597513+0000", 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7989999999999999e-06 2024-03-21T14:24:16.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597528+0000", 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4491000000000001e-05 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597538+0000", 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.7720000000000006e-06 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:16.608 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597607+0000", 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.9726000000000003e-05 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597740+0000", 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013268199999999999 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597850+0000", 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000110303 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.609 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:214677 11.3 11:c5dbd589:::benchmark_data_smithi003_188959_object103795:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:24:16.597563+0000", 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00174905, 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019914389999999998, 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "waiting for sub ops", 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 214677 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.610 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597563+0000", 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597563+0000", 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597563+0000", 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.51e-07 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.611 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597566+0000", 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8380000000000002e-06 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597572+0000", 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6949999999999997e-06 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597578+0000", 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4659999999999996e-06 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.612 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597632+0000", 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.3355e-05 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597749+0000", 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00011752300000000001 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,5", 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.597827+0000", 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7470000000000002e-05 2024-03-21T14:24:16.613 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:214678 11.4 11.72665914 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:24:16.598431+0000", 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 0.00088042000000000003, 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001205827, 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "queued for pg", 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 214678 2024-03-21T14:24:16.614 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.598431+0000", 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.598431+0000", 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.598432+0000", 2024-03-21T14:24:16.615 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2399999999999997e-07 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.598436+0000", 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7680000000000002e-06 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.598448+0000", 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2536e-05 2024-03-21T14:24:16.616 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:24:16.598456+0000", 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.571e-06 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout: "num_ops": 4 2024-03-21T14:24:16.617 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:24:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.618 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:16.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:16.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:16.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:24:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:24:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:24:16.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.761 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:16.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:16.862 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:16.889 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:16.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:16.929 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:24:16.929 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:24:16.929 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 456, 2024-03-21T14:24:16.929 INFO:teuthology.orchestra.run.smithi003.stdout: "fsid": "c60e082a-e78a-11ee-95cd-87774f69a715", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "created": "2024-03-21T13:56:54.530125+0000", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "modified": "2024-03-21T14:24:08.030288+0000", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "last_up_change": "2024-03-21T14:19:13.730963+0000", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "last_in_change": "2024-03-21T14:21:25.732577+0000", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_num": 5801984, 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_set": [ 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "noscrub", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "pglog_hardlimit", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "purged_snapdirs", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_deletes", 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: "sortbitwise" 2024-03-21T14:24:16.930 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_version": 20, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "full_ratio": 0.94999998807907104, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_snapshot": "", 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_max": 11, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd": 8, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "require_min_compat_client": "luminous", 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "min_compat_client": "luminous", 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "require_osd_release": "squid", 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_crimson": false, 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "pools": [ 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.931 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 1, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".mgr", 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:00:12.925535+0000", 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.932 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 11, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 11, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 11, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 11, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 11, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.933 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "270", 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "268", 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.934 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.935 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.936 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_max": 32, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_min": 1 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr": {} 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.9099999666213989, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.9099999666213989, 2024-03-21T14:24:16.937 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.9099999666213989, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.9099999666213989, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 2, 2024-03-21T14:24:16.938 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "datapool", 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.373659+0000", 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 8193, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.939 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 3, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 3, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 3, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 3, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 3, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 2, 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 68, 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.941 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.942 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:24:16.943 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd": {} 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 4.6500000953674316, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 4.6500000953674316, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 0.86000001430511475, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 4, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 4, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 3, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": ".rgw.root", 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:31.794831+0000", 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.945 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.946 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "66", 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.947 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.948 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.949 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.950 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.9700000286102295, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.9700000286102295, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.25, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.25, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 4, 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.log", 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:33.750394+0000", 2024-03-21T14:24:16.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:24:16.952 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "68", 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:24:16.953 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.954 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.955 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.956 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.190000057220459, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.190000057220459, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 2.5, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 2.5, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.957 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 5, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.control", 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:35.711498+0000", 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.958 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.959 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "70", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.960 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.961 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.962 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.5399999618530273, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.5399999618530273, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.75, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.75, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.963 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 6, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "default.rgw.meta", 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:02:37.763124+0000", 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.964 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 32, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 32, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 32, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 32, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 32, 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.965 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "72", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.966 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.967 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.968 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "options": { 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_bias": 4 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "rgw": {} 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.969 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 1.9700000286102295, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 1.9700000286102295, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "pool": 11, 2024-03-21T14:24:16.970 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": "unique_pool_4", 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "create_time": "2024-03-21T14:20:58.032622+0000", 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "flags_names": "hashpspool", 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "type": 1, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 3, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": 2, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_rule": 0, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_count": 0, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_target": 0, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_barrier": 0, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "object_hash": 2, 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscale_mode": "off", 2024-03-21T14:24:16.971 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": 16, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num": 16, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_placement_num_target": 16, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_target": 16, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num_pending": 16, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "last_pg_merge_meta": { 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "source_pgid": "0.0", 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "ready_epoch": 0, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_started": 0, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "last_epoch_clean": 0, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "source_version": "0'0", 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "target_version": "0'0" 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "last_change": "455", 2024-03-21T14:24:16.972 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend": "0", 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "last_force_op_resend_preluminous": "0", 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "auid": 0, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_mode": "selfmanaged", 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_seq": 0, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_epoch": 0, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_snaps": [], 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps": "[]", 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_bytes": 0, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "quota_max_objects": 0, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "tiers": [], 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "tier_of": -1, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "read_tier": -1, 2024-03-21T14:24:16.973 INFO:teuthology.orchestra.run.smithi003.stdout: "write_tier": -1, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_mode": "none", 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_bytes": 0, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "target_max_objects": 0, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_flush_age": 0, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "cache_min_evict_age": 0, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profile": "", 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_params": { 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "none" 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.974 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_period": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_count": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "use_gmt_hitset": true, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "min_read_recency_for_promote": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "min_write_recency_for_promote": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_grade_decay_rate": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "hit_set_search_last_n": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "grade_table": [], 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "stripe_width": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "expected_num_objects": 0, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "fast_read": false, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "options": {}, 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "application_metadata": { 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: "rados": {} 2024-03-21T14:24:16.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "read_balance": { 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "score_type": "Fair distribution", 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "score_acting": 2.630000114440918, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "score_stable": 2.630000114440918, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "optimal_score": 1.1399999856948853, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_acting": 3, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "raw_score_stable": 3, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity_weighted": 0.69770431518554688, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity": 0.80000001192092896, 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: "average_primary_affinity_weighted": 0.69999998807907104 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.976 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "daf9bc3c-4e73-4379-afb6-7ee181b686fb", 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 8, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 393, 2024-03-21T14:24:16.977 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:16.978 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6802", 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6804", 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.979 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6803", 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 930096433 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6801/930096433", 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6802/930096433", 2024-03-21T14:24:16.980 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6804/930096433", 2024-03-21T14:24:16.981 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6803/930096433", 2024-03-21T14:24:16.981 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:16.981 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:16.981 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:16.981 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "1652f24d-aa0e-471c-bf54-03814b1ee11d", 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 13, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 424, 2024-03-21T14:24:16.982 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6805", 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.983 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6806", 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6808", 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:24:16.984 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6807", 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1526485482 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6805/1526485482", 2024-03-21T14:24:16.985 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6806/1526485482", 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6808/1526485482", 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6807/1526485482", 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "5e67d907-f8d1-4ecc-85d3-df88da5c364c", 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:16.986 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 18, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 277, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 358, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 431, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 278, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6809", 2024-03-21T14:24:16.987 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6810", 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.988 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6812", 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6811", 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3011075847 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.989 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6809/3011075847", 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6810/3011075847", 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6812/3011075847", 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6811/3011075847", 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:24:16.990 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "ad8dabdf-fce5-4312-9e80-dbbb6ba2947c", 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 25, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 411, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.991 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6813", 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6814", 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.992 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6816", 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6815", 2024-03-21T14:24:16.993 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1918209225 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.3:6813/1918209225", 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.3:6814/1918209225", 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.3:6816/1918209225", 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.3:6815/1918209225", 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.994 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "b5d486b1-2cb6-46c7-8664-994af07ff7b5", 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 1, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 30, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 425, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.995 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6800", 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6801", 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:24:16.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6803", 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6802", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 296472230 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6800/296472230", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6801/296472230", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6803/296472230", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6802/296472230", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:16.998 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "bff24605-38ce-47ce-9142-4ced3766476e", 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0.0266571044921875, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 40, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 382, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:24:16.999 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6804", 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.000 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6805", 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6807", 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.001 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6806", 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 326385780 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6804/326385780", 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6805/326385780", 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6807/326385780", 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6806/326385780", 2024-03-21T14:24:17.002 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "73a781e7-8660-4885-b976-2aeb8515109a", 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 0.2286529541015625, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0.9384765625, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 48, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 332, 2024-03-21T14:24:17.003 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 351, 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 443, 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 333, 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6808", 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.004 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6809", 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:17.005 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6811", 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6810", 2024-03-21T14:24:17.006 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1863637416 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6808/1863637416", 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6809/1863637416", 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6811/1863637416", 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6810/1863637416", 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:17.007 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: "uuid": "3e95559b-79f6-40ba-9d98-d350bcdd60e0", 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: "up": 1, 2024-03-21T14:24:17.008 INFO:teuthology.orchestra.run.smithi003.stdout: "in": 1, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "weight": 1, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_affinity": 0.6165008544921875, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_begin": 0, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "last_clean_end": 0, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "up_from": 58, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "up_thru": 427, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "down_at": 0, 2024-03-21T14:24:17.009 INFO:teuthology.orchestra.run.smithi003.stdout: "lost_at": 0, 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addrs": { 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6812", 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.010 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addrs": { 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6813", 2024-03-21T14:24:17.011 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:24:17.012 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.012 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.012 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addrs": { 2024-03-21T14:24:17.012 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.012 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.013 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6815", 2024-03-21T14:24:17.013 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:24:17.013 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.013 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addrs": { 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.118:6814", 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 2128678671 2024-03-21T14:24:17.014 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: "public_addr": "172.21.15.118:6812/2128678671", 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_addr": "172.21.15.118:6813/2128678671", 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_back_addr": "172.21.15.118:6815/2128678671", 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: "heartbeat_front_addr": "172.21.15.118:6814/2128678671", 2024-03-21T14:24:17.015 INFO:teuthology.orchestra.run.smithi003.stdout: "state": [ 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: "exists", 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: "up" 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_xinfo": [ 2024-03-21T14:24:17.016 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.017 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 0, 2024-03-21T14:24:17.017 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.017 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.017 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.017 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.017 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.018 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:32.412264+0000", 2024-03-21T14:24:17.018 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:24:17.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.018 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.018 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 1, 2024-03-21T14:24:17.018 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:53.740214+0000", 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 2, 2024-03-21T14:24:17.019 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:49.284105+0000", 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 277 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 3, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.020 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:55.297076+0000", 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 4, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:19.912338+0000", 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 5, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:27.153048+0000", 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 6, 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.022 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:53.353004+0000", 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 332 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "osd": 7, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "down_stamp": "0.000000", 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_probability": 0, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "laggy_interval": 0, 2024-03-21T14:24:17.023 INFO:teuthology.orchestra.run.smithi003.stdout: "features": 4540701547738038271, 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: "old_weight": 0, 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: "last_purged_snaps_scrub": "2024-03-21T14:23:53.646998+0000", 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: "dead_epoch": 0 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap": [ 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.c", 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: "osds": [ 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: 0, 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: 6, 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: 7 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.024 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_items": [ 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.a", 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "3.13", 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.025 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 4, 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.4", 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 5 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.026 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.b", 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.f", 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.027 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 3, 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "4.13", 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:24:17.028 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 2 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.2", 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.029 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.4", 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 5, 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 0 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.a", 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.030 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "6.13", 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 2, 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 1 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "pgid": "11.f", 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "mappings": [ 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "from": 1, 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "to": 4 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_upmap_primaries": [], 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_temp": [], 2024-03-21T14:24:17.032 INFO:teuthology.orchestra.run.smithi003.stdout: "primary_temp": [], 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "blocklist": { 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/45234419": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/384540368": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3883474825": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/1598547836": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3687155565": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3884352348": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2319227492": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2154949954": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/4162970769": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2292819669": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:24:17.033 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/3411869253": "2024-03-22T14:03:15.227370+0000", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/520312891": "2024-03-22T13:57:56.576834+0000", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:0/2267426691": "2024-03-22T13:57:29.327821+0000", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "172.21.15.3:6800/3211426112": "2024-03-22T13:57:56.576834+0000" 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "range_blocklist": {}, 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "erasure_code_profiles": { 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "default": { 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-failure-domain": "osd", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "k": "2", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "m": "1", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "plugin": "jerasure", 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: "technique": "reed_sol_van" 2024-03-21T14:24:17.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "removed_snaps_queue": [], 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "new_removed_snaps": [], 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "new_purged_snaps": [], 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_node_flags": {}, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "device_class_flags": {}, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode": { 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_enabled": false, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_bucket_count": 0, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "degraded_stretch_mode": 0, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "recovering_stretch_mode": 0, 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: "stretch_mode_bucket": 0 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:17.035 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:24:17.036 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:17.066 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:17.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:17.169 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:17.195 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:17.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:17.336 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[29576]: pgmap v1017: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[29576]: pgmap v1018: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1087729272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:24:17.409 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[40510]: pgmap v1017: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:17.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:24:17.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:24:17.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[40510]: pgmap v1018: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T14:24:17.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:17.410 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:17 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1087729272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:24:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-mon[33862]: pgmap v1017: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 921 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:24:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:24:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-mon[33862]: pgmap v1018: 158 pgs: 158 active+clean; 700 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T14:24:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:17.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1087729272' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-21T14:24:17.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:17.472 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:17.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:17.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:17.612 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:17.714 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:17.746 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:17.841 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-21T14:24:17.848 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:17.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:24:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:24:17.957 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:17.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:17.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:18.059 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:18.087 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:18.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:18.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.243 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:18.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:18.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:18.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-21T14:24:18.384 INFO:tasks.thrashosds.thrasher:j is {'epoch': 456, 'fsid': 'c60e082a-e78a-11ee-95cd-87774f69a715', 'created': '2024-03-21T13:56:54.530125+0000', 'modified': '2024-03-21T14:24:08.030288+0000', 'last_up_change': '2024-03-21T14:19:13.730963+0000', 'last_in_change': '2024-03-21T14:21:25.732577+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-21T14:00:12.925535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '268', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-21T14:02:31.373659+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.650000095367432, 'score_stable': 4.650000095367432, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 4, 'raw_score_stable': 4, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-21T14:02:31.794831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-21T14:02:33.750394+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-21T14:02:35.711498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-21T14:02:37.763124+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-21T14:20:58.032622+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '455', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.630000114440918, 'score_stable': 2.630000114440918, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.6977043151855469, 'average_primary_affinity': 0.800000011920929, 'average_primary_affinity_weighted': 0.699999988079071}}], 'osds': [{'osd': 0, 'uuid': 'daf9bc3c-4e73-4379-afb6-7ee181b686fb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 393, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6801', 'nonce': 930096433}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6802', 'nonce': 930096433}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6804', 'nonce': 930096433}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6803', 'nonce': 930096433}]}, 'public_addr': '172.21.15.3:6801/930096433', 'cluster_addr': '172.21.15.3:6802/930096433', 'heartbeat_back_addr': '172.21.15.3:6804/930096433', 'heartbeat_front_addr': '172.21.15.3:6803/930096433', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1652f24d-aa0e-471c-bf54-03814b1ee11d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 424, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6805', 'nonce': 1526485482}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6806', 'nonce': 1526485482}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6808', 'nonce': 1526485482}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6807', 'nonce': 1526485482}]}, 'public_addr': '172.21.15.3:6805/1526485482', 'cluster_addr': '172.21.15.3:6806/1526485482', 'heartbeat_back_addr': '172.21.15.3:6808/1526485482', 'heartbeat_front_addr': '172.21.15.3:6807/1526485482', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5e67d907-f8d1-4ecc-85d3-df88da5c364c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 277, 'up_from': 358, 'up_thru': 431, 'down_at': 278, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6809', 'nonce': 3011075847}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6810', 'nonce': 3011075847}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6812', 'nonce': 3011075847}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6811', 'nonce': 3011075847}]}, 'public_addr': '172.21.15.3:6809/3011075847', 'cluster_addr': '172.21.15.3:6810/3011075847', 'heartbeat_back_addr': '172.21.15.3:6812/3011075847', 'heartbeat_front_addr': '172.21.15.3:6811/3011075847', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'ad8dabdf-fce5-4312-9e80-dbbb6ba2947c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 411, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6813', 'nonce': 1918209225}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6814', 'nonce': 1918209225}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6816', 'nonce': 1918209225}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6815', 'nonce': 1918209225}]}, 'public_addr': '172.21.15.3:6813/1918209225', 'cluster_addr': '172.21.15.3:6814/1918209225', 'heartbeat_back_addr': '172.21.15.3:6816/1918209225', 'heartbeat_front_addr': '172.21.15.3:6815/1918209225', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b5d486b1-2cb6-46c7-8664-994af07ff7b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 30, 'up_thru': 425, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6800', 'nonce': 296472230}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6801', 'nonce': 296472230}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6803', 'nonce': 296472230}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6802', 'nonce': 296472230}]}, 'public_addr': '172.21.15.118:6800/296472230', 'cluster_addr': '172.21.15.118:6801/296472230', 'heartbeat_back_addr': '172.21.15.118:6803/296472230', 'heartbeat_front_addr': '172.21.15.118:6802/296472230', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'bff24605-38ce-47ce-9142-4ced3766476e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.0266571044921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6804', 'nonce': 326385780}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6805', 'nonce': 326385780}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6807', 'nonce': 326385780}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6806', 'nonce': 326385780}]}, 'public_addr': '172.21.15.118:6804/326385780', 'cluster_addr': '172.21.15.118:6805/326385780', 'heartbeat_back_addr': '172.21.15.118:6807/326385780', 'heartbeat_front_addr': '172.21.15.118:6806/326385780', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '73a781e7-8660-4885-b976-2aeb8515109a', 'up': 1, 'in': 1, 'weight': 0.2286529541015625, 'primary_affinity': 0.9384765625, 'last_clean_begin': 48, 'last_clean_end': 332, 'up_from': 351, 'up_thru': 443, 'down_at': 333, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6808', 'nonce': 1863637416}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6809', 'nonce': 1863637416}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6811', 'nonce': 1863637416}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6810', 'nonce': 1863637416}]}, 'public_addr': '172.21.15.118:6808/1863637416', 'cluster_addr': '172.21.15.118:6809/1863637416', 'heartbeat_back_addr': '172.21.15.118:6811/1863637416', 'heartbeat_front_addr': '172.21.15.118:6810/1863637416', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3e95559b-79f6-40ba-9d98-d350bcdd60e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6165008544921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 58, 'up_thru': 427, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6812', 'nonce': 2128678671}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6813', 'nonce': 2128678671}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6815', 'nonce': 2128678671}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6814', 'nonce': 2128678671}]}, 'public_addr': '172.21.15.118:6812/2128678671', 'cluster_addr': '172.21.15.118:6813/2128678671', 'heartbeat_back_addr': '172.21.15.118:6815/2128678671', 'heartbeat_front_addr': '172.21.15.118:6814/2128678671', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:32.412264+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:53.740214+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:49.284105+0000', 'dead_epoch': 277}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:55.297076+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:19.912338+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:27.153048+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:53.353004+0000', 'dead_epoch': 332}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-21T14:23:53.646998+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.c', 'osds': [0, 6, 7]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.13', 'mappings': [{'from': 4, 'to': 0}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.2', 'mappings': [{'from': 2, 'to': 0}]}, {'pgid': '6.4', 'mappings': [{'from': 5, 'to': 0}]}, {'pgid': '6.a', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '6.13', 'mappings': [{'from': 2, 'to': 1}]}, {'pgid': '11.f', 'mappings': [{'from': 1, 'to': 4}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.3:0/45234419': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/384540368': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3883474825': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/1598547836': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:6800/3687155565': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/3884352348': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2319227492': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2154949954': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/4162970769': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/2292819669': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:0/3411869253': '2024-03-22T14:03:15.227370+0000', '172.21.15.3:0/520312891': '2024-03-22T13:57:56.576834+0000', '172.21.15.3:0/2267426691': '2024-03-22T13:57:29.327821+0000', '172.21.15.3:6800/3211426112': '2024-03-22T13:57:56.576834+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-21T14:24:18.384 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:24:18.400 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:18.502 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:18.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.552 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:18.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:18.653 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:18.672 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:18.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.692 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:18.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:18.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:18.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.851 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:18.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:18.912 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:18.952 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:18.980 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:18.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:19.003 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:19.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:19.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[29576]: pgmap v1019: 158 pgs: 158 active+clean; 432 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:19.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:18 smithi003 ceph-mon[40510]: pgmap v1019: 158 pgs: 158 active+clean; 432 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:19.082 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:19.109 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:19.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:18 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:18 smithi118 ceph-mon[33862]: pgmap v1019: 158 pgs: 158 active+clean; 432 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 1.07k op/s 2024-03-21T14:24:19.211 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:19.237 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:19.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:19.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.382 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:19.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:19.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:19.531 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:19.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:19.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:19.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:19.762 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:19.803 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:19.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:19.943 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:19.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:20.044 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:20.091 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:20.193 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:20.222 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:20.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:20.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:20.352 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:20.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:20.479 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:20.581 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:20.606 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:20.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.736 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:20.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:20.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:20.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:20.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:21.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:21.033 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:21.135 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:21.157 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is set 2024-03-21T14:24:21.163 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:21.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:21.296 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:21.357 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:21 smithi118 ceph-mon[33862]: pgmap v1020: 158 pgs: 158 active+clean; 366 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 860 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T14:24:21.358 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:21 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2873013675' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:21.358 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:21 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:21.397 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:21.429 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:21 smithi003 ceph-mon[29576]: pgmap v1020: 158 pgs: 158 active+clean; 366 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 860 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T14:24:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:21 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2873013675' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:21 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:21.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:21 smithi003 ceph-mon[40510]: pgmap v1020: 158 pgs: 158 active+clean; 366 MiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 860 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-21T14:24:21.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:21 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2873013675' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:21.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:21 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:21.531 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:21.563 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:21.665 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:21.696 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:21.797 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:21.826 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:21.928 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:21.959 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:21.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:22.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:22.101 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:22.195 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:24:22.195 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [], 2024-03-21T14:24:22.195 INFO:teuthology.orchestra.run.smithi118.stdout: "complaint_time": 30, 2024-03-21T14:24:22.195 INFO:teuthology.orchestra.run.smithi118.stdout: "num_blocked_ops": 0 2024-03-21T14:24:22.195 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:24:22.203 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.245 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:22.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:22.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:22.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:22 smithi003 ceph-mon[29576]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:22.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:22 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:24:22.406 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:22 smithi003 ceph-mon[29576]: osdmap e457: 8 total, 8 up, 8 in 2024-03-21T14:24:22.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:22 smithi003 ceph-mon[40510]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:22.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:22 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:24:22.406 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:22 smithi003 ceph-mon[40510]: osdmap e457: 8 total, 8 up, 8 in 2024-03-21T14:24:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:22 smithi118 ceph-mon[33862]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:22 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-21T14:24:22.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:22 smithi118 ceph-mon[33862]: osdmap e457: 8 total, 8 up, 8 in 2024-03-21T14:24:22.485 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:22.515 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:22.617 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:22.656 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:22.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:22.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:22.785 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:22.887 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:22.929 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:22.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:22.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-21T14:24:23.025 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Removed 110879 objects 2024-03-21T14:24:23.025 INFO:tasks.radosbench.radosbench.0.smithi003.stdout:Clean up completed and total clean up time :99.4167 2024-03-21T14:24:23.030 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:23.095 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:23.192 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:23.196 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:23.220 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:23.322 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:23.346 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:23.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:23 smithi118 ceph-mon[33862]: pgmap v1022: 158 pgs: 158 active+clean; 318 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 702 op/s 2024-03-21T14:24:23.448 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:23.471 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:23 smithi003 ceph-mon[29576]: pgmap v1022: 158 pgs: 158 active+clean; 318 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 702 op/s 2024-03-21T14:24:23.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:23 smithi003 ceph-mon[40510]: pgmap v1022: 158 pgs: 158 active+clean; 318 MiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 588 B/s rd, 0 B/s wr, 702 op/s 2024-03-21T14:24:23.573 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:23.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.598 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:23.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:23.699 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:23.723 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:23.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:23.825 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:23.848 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:23.950 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:23.975 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:23.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:24.075 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:24.100 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:24.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:24.201 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:24.225 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:24.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:24.327 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:24.355 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:24.457 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:24.484 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:24.586 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:24.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:24.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:24.754 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:24.779 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:24.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:24.881 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:24.905 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:24.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:24.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:25.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:25.025 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "size": 20, 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 600, 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "ops": [ 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268617 9.7 9:eb313569:::benchmark_data_smithi003_135814_object127883:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.862294+0000", 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.161030692, 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.036352554, 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.026 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268617 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862294+0000", 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.027 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862294+0000", 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862295+0000", 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4399999999999998e-07 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.862300+0000", 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8189999999999999e-06 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.028 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.866340+0000", 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00404061 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.866358+0000", 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7586000000000001e-05 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.029 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868673+0000", 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023157519999999999 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868683+0000", 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8130000000000007e-06 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.030 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898576+0000", 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0298928190000001 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898646+0000", 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0210999999999996e-05 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.031 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268627 9.8 9:125f1641:::benchmark_data_smithi003_135814_object127893:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.868574+0000", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.15475011199999, 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0303467589999999, 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268627 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868574+0000", 2024-03-21T14:24:25.032 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868574+0000", 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868575+0000", 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.58e-07 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.033 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868578+0000", 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.9009999999999998e-06 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868585+0000", 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8929999999999999e-06 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.868590+0000", 2024-03-21T14:24:25.034 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7540000000000003e-06 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870331+0000", 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0017408759999999999 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870342+0000", 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.075e-05 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897810+0000", 2024-03-21T14:24:25.035 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0274681649999999 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898921+0000", 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011109620000000001 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268628 9.7 9:ea86f15e:::benchmark_data_smithi003_135814_object127894:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.036 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870340+0000", 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.15298435900002, 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.031124903, 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268628 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870340+0000", 2024-03-21T14:24:25.037 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870340+0000", 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870341+0000", 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0270000000000001e-06 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.038 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870346+0000", 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7530000000000001e-06 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870357+0000", 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0767e-05 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870364+0000", 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7659999999999998e-06 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.039 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870662+0000", 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00029719099999999998 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870669+0000", 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8609999999999998e-06 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.040 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898862+0000", 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0281925110000001 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for missing object", 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898997+0000", 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013489700000000001 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.901394+0000", 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0023972780000000001 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.041 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.901465+0000", 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0852000000000005e-05 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268630 9.8 9:1462fa6d:::benchmark_data_smithi003_135814_object127896:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.15294360199999, 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.038780203, 2024-03-21T14:24:25.042 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268630 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.043 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870381+0000", 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0200000000000002e-07 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870385+0000", 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.292e-06 2024-03-21T14:24:25.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870391+0000", 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5060000000000001e-06 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870406+0000", 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.518e-05 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870452+0000", 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5859999999999998e-05 2024-03-21T14:24:25.045 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870461+0000", 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2709999999999998e-06 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.899383+0000", 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0289211490000001 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.046 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.899704+0000", 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00032184800000000001 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.899902+0000", 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00019798600000000001 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.908928+0000", 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0090258820000000007 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.047 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.908987+0000", 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8446999999999998e-05 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909061+0000", 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4682000000000001e-05 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909113+0000", 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1447999999999999e-05 2024-03-21T14:24:25.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909161+0000", 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8149999999999998e-05 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268633 9.8 9:17488b50:::benchmark_data_smithi003_135814_object127899:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870422+0000", 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.15290233100001, 2024-03-21T14:24:25.049 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0421929080000001, 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268633 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870422+0000", 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870422+0000", 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870423+0000", 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9100000000000004e-07 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.051 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870426+0000", 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.286e-06 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870433+0000", 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.2300000000000002e-06 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870439+0000", 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.9359999999999996e-06 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.052 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870488+0000", 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.9209999999999998e-05 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870496+0000", 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9629999999999998e-06 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.906627+0000", 2024-03-21T14:24:25.053 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.036130526 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.906759+0000", 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000131731 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.906887+0000", 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000128763 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.054 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909014+0000", 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021263699999999998 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.910484+0000", 2024-03-21T14:24:25.055 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0014707749999999999 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912494+0000", 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0020092780000000002 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912557+0000", 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3280000000000004e-05 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.056 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912615+0000", 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8069000000000003e-05 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268639 9.5 9:a8974fa7:::benchmark_data_smithi003_135814_object127905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.152839675, 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.027248454, 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.057 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268639 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.058 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870485+0000", 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.6199999999999998e-07 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870488+0000", 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7389999999999999e-06 2024-03-21T14:24:25.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870494+0000", 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1299999999999998e-06 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.870501+0000", 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.46e-06 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.871813+0000", 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001312548 2024-03-21T14:24:25.060 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.871823+0000", 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.8940000000000005e-06 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.889526+0000", 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0177033390000001 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.889846+0000", 2024-03-21T14:24:25.061 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00031953599999999998 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.890019+0000", 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017282699999999999 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897136+0000", 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071172700000000002 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.062 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897191+0000", 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4558999999999997e-05 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897670+0000", 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00047937000000000001 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897706+0000", 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5887999999999999e-05 2024-03-21T14:24:25.063 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897733+0000", 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7432000000000001e-05 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268640 9.8 9:19908561:::benchmark_data_smithi003_135814_object127906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.873233+0000", 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.15009159700003, 2024-03-21T14:24:25.064 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0394783320000001, 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268640 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873233+0000", 2024-03-21T14:24:25.065 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873233+0000", 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873234+0000", 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2699999999999999e-06 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.066 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873239+0000", 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8899999999999998e-06 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873249+0000", 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5200000000000003e-06 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873256+0000", 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.644e-06 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.067 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873307+0000", 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0921000000000003e-05 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.873316+0000", 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2439999999999999e-06 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909178+0000", 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0358614159999999 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909251+0000", 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3392999999999993e-05 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.909320+0000", 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8437999999999998e-05 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.069 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.070 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:25.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:25.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.910646+0000", 2024-03-21T14:24:25.072 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013268010000000001 2024-03-21T14:24:25.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.072 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.072 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.072 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.911254+0000", 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00060741600000000005 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912654+0000", 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.001399874 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912686+0000", 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.273e-05 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.073 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.912711+0000", 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4774999999999999e-05 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25232.0:268643 9.5 9:aae2a19c:::benchmark_data_smithi003_135814_object127909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e305)", 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:16:20.880225+0000", 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 484.14309956599999, 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0218630360000001, 2024-03-21T14:24:25.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25232", 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/280890026", 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 268643 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880225+0000", 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.075 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880225+0000", 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880226+0000", 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0800000000000004e-07 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880230+0000", 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.9530000000000001e-06 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.076 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880236+0000", 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4189999999999999e-06 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880241+0000", 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.933e-06 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.077 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880293+0000", 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2160000000000002e-05 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:20.880307+0000", 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.343e-05 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.894036+0000", 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.013729887 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.078 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.894152+0000", 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000115622 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.894270+0000", 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000118305 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.079 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.897147+0000", 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.002876768 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.898277+0000", 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0011301 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.080 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.901985+0000", 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.003708065 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.902042+0000", 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6913999999999998e-05 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:16:21.902088+0000", 2024-03-21T14:24:25.081 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5772000000000002e-05 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23379 11.4 11:2e3ac6f5:::benchmark_data_smithi003_188959_object23378:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.746947+0000", 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.276377051, 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4304255870000002, 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.082 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23379 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.746947+0000", 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.083 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.746947+0000", 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.746949+0000", 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5939999999999999e-06 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747000+0000", 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1158e-05 2024-03-21T14:24:25.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747014+0000", 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4038e-05 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747023+0000", 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4840000000000004e-06 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.085 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747068+0000", 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5249e-05 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747227+0000", 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000159371 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:24:25.086 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747388+0000", 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016095200000000001 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.749530+0000", 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0021418359999999998 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.755257+0000", 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.005727256 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.087 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762367+0000", 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071094590000000003 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762380+0000", 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2816999999999999e-05 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.088 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761071+0000", 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99869118999999995 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761086+0000", 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5654999999999998e-05 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.177263+0000", 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4161765470000001 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.089 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.177373+0000", 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000109981 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23381 11.4 11:29bb2c9e:::benchmark_data_smithi003_188959_object23380:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.747321+0000", 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.276003675, 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4307675550000001, 2024-03-21T14:24:25.090 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23381 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747321+0000", 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.091 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747321+0000", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747321+0000", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7800000000000001e-07 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747347+0000", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.5658999999999999e-05 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.092 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747355+0000", 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.4839999999999998e-06 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.747362+0000", 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.0870000000000001e-06 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748267+0000", 2024-03-21T14:24:25.093 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00090570899999999998 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748400+0000", 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000132841 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748523+0000", 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012234400000000001 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.094 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.750455+0000", 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019322499999999999 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.755299+0000", 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0048443649999999998 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762431+0000", 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0071315909999999996 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.095 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762441+0000", 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0448999999999999e-05 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761200+0000", 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99875920600000001 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761211+0000", 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0759e-05 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.096 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178028+0000", 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4168169349999999 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178088+0000", 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.0198000000000002e-05 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.097 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23386 11.4 11:2cfd1a55:::benchmark_data_smithi003_188959_object23385:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.748866+0000", 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.27445804499999, 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4293230349999999, 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23386 2024-03-21T14:24:25.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748866+0000", 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748866+0000", 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.099 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748868+0000", 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1030000000000001e-06 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748900+0000", 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2904000000000002e-05 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.100 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748920+0000", 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.9806999999999999e-05 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.748928+0000", 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.7829999999999999e-06 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.749563+0000", 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000634667 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.101 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.749703+0000", 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000140118 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.749842+0000", 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00013913700000000001 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.750709+0000", 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000867383 2024-03-21T14:24:25.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.756061+0000", 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0053512239999999999 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762490+0000", 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0064294629999999998 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762500+0000", 2024-03-21T14:24:25.103 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.0468999999999999e-05 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761254+0000", 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.99875383100000004 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761264+0000", 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.4350000000000003e-06 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.104 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178127+0000", 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4168628009999999 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178189+0000", 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2910000000000006e-05 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.105 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23391 11.4 11:2e54069d:::benchmark_data_smithi003_188959_object23390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.758021+0000", 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.26530394700001, 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.42866969, 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "reached pg", 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23391 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.106 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758021+0000", 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758021+0000", 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758022+0000", 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.561e-06 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.107 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758057+0000", 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5329e-05 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758070+0000", 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2751e-05 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758079+0000", 2024-03-21T14:24:25.108 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.7499999999999992e-06 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758141+0000", 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2224999999999995e-05 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758315+0000", 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000173661 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.109 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 1,2", 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.758461+0000", 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00014627299999999999 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762541+0000", 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0040800189999999998 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.762551+0000", 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.5119999999999997e-06 2024-03-21T14:24:25.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761302+0000", 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.998751842 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761312+0000", 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.6350000000000001e-06 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178206+0000", 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.41689435 2024-03-21T14:24:25.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for degraded object", 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178555+0000", 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00034840199999999999 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.186623+0000", 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0080682529999999992 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.112 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.186690+0000", 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7126999999999996e-05 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23397 11.4 11:2795a6d3:::benchmark_data_smithi003_188959_object23396:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.775509+0000", 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.24781520100001, 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.411701291, 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.113 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23397 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775509+0000", 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.114 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775509+0000", 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775511+0000", 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.5149999999999999e-06 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775544+0000", 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.2894e-05 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.115 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775558+0000", 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4596000000000001e-05 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775566+0000", 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9389999999999997e-06 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775628+0000", 2024-03-21T14:24:25.116 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1612000000000006e-05 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.775637+0000", 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.789e-06 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761342+0000", 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.98570514300000001 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.117 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761350+0000", 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.3359999999999995e-06 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178653+0000", 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4173028379999999 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178774+0000", 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000121426 2024-03-21T14:24:25.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.178900+0000", 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00012576800000000001 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.185091+0000", 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0061913309999999996 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.119 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.185362+0000", 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000270974 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.186733+0000", 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0013702219999999999 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.186797+0000", 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4263000000000005e-05 2024-03-21T14:24:25.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187211+0000", 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00041364499999999997 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23438 11.4 11:294db742:::benchmark_data_smithi003_188959_object23437:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.808824+0000", 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.21450045899999, 2024-03-21T14:24:25.121 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3789254909999999, 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23438 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.808824+0000", 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.808824+0000", 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.808826+0000", 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.2730000000000001e-06 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.123 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.808878+0000", 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.2176000000000003e-05 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.808903+0000", 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.4810000000000001e-05 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.808931+0000", 2024-03-21T14:24:25.124 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7665e-05 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.809051+0000", 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000120512 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.809068+0000", 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.6679000000000001e-05 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.125 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761382+0000", 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.952313627 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761390+0000", 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.2700000000000004e-06 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.179745+0000", 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.418355193 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.126 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.179846+0000", 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000100788 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.179964+0000", 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000117817 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.127 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.185146+0000", 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0051818949999999997 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.185369+0000", 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00022307499999999999 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187332+0000", 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0019632059999999999 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.128 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187369+0000", 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7354000000000002e-05 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187749+0000", 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038015100000000001 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.129 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23457 11.4 11:23463607:::benchmark_data_smithi003_188959_object23456:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.820159+0000", 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.20316504100001, 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3682524169999999, 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23457 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.130 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820159+0000", 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820159+0000", 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.131 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820160+0000", 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2399999999999996e-07 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820211+0000", 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.0617e-05 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820221+0000", 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.9529999999999994e-06 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.132 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820229+0000", 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8760000000000005e-06 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820280+0000", 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.1706999999999998e-05 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.820288+0000", 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.6739999999999997e-06 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.133 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761418+0000", 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.94112968299999999 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761426+0000", 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.8590000000000005e-06 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.134 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.180656+0000", 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4192298239999999 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.180815+0000", 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015964000000000001 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.181016+0000", 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00020087300000000001 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.135 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.186583+0000", 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0055668250000000001 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187286+0000", 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070321700000000001 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187810+0000", 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00052407199999999999 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.136 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.187867+0000", 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6573000000000002e-05 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.188412+0000", 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00054509999999999997 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23491 11.4 11:2145a127:::benchmark_data_smithi003_188959_object23490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.137 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.841460+0000", 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.181864617, 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.347698887, 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23491 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.138 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841460+0000", 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841460+0000", 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841461+0000", 2024-03-21T14:24:25.139 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.2299999999999999e-07 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841529+0000", 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7988e-05 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841544+0000", 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.505e-05 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.140 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841557+0000", 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.3118e-05 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841614+0000", 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.6731000000000003e-05 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.841620+0000", 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.7700000000000004e-06 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.141 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761452+0000", 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91983193600000002 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761460+0000", 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.0460000000000006e-06 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.142 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.181909+0000", 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4204488129999999 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.182069+0000", 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000159352 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.182770+0000", 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070174500000000004 2024-03-21T14:24:25.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.186592+0000", 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0038212379999999998 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.188474+0000", 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0018827049999999999 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.188589+0000", 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000114437 2024-03-21T14:24:25.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.188650+0000", 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.122e-05 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.189159+0000", 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00050881500000000003 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23502 11.4 11:2ce3d966:::benchmark_data_smithi003_188959_object23501:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.849244+0000", 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.174080503, 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3528700520000001, 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23502 2024-03-21T14:24:25.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849244+0000", 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849244+0000", 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.147 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849245+0000", 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.1099999999999995e-07 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849303+0000", 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.8414e-05 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849310+0000", 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.3719999999999999e-06 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.148 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849315+0000", 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.7589999999999997e-06 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849367+0000", 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.164e-05 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.149 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.849375+0000", 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 7.9580000000000007e-06 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761525+0000", 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.91215014999999999 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761534+0000", 2024-03-21T14:24:25.150 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.5669999999999995e-06 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.189839+0000", 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4283056219999999 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.190028+0000", 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018851499999999999 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.190733+0000", 2024-03-21T14:24:25.151 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00070522499999999997 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.195504+0000", 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0047708890000000004 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.195523+0000", 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.8845e-05 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.201271+0000", 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0057484529999999997 2024-03-21T14:24:25.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.201328+0000", 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.7017e-05 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.202114+0000", 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00078591499999999999 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23522 11.4 11:299589eb:::benchmark_data_smithi003_188959_object23521:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.863710+0000", 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.15961495100001, 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.338932995, 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23522 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.154 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863710+0000", 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863710+0000", 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863711+0000", 2024-03-21T14:24:25.155 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.212e-06 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863748+0000", 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7307e-05 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863760+0000", 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2119e-05 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863767+0000", 2024-03-21T14:24:25.156 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.5339999999999996e-06 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863834+0000", 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.6840000000000004e-05 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.863846+0000", 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2114e-05 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.157 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761563+0000", 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.89771695299999998 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761571+0000", 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.4109999999999996e-06 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.191868+0000", 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.430297318 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.158 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.192048+0000", 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00017946699999999999 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.192235+0000", 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000187107 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.159 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.195531+0000", 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0032960390000000002 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.195566+0000", 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5552000000000001e-05 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.202161+0000", 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.006594651 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.160 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.202199+0000", 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.7914000000000003e-05 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.202643+0000", 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00044345700000000002 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23543 11.4 11:29f3dd6f:::benchmark_data_smithi003_188959_object23542:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.879391+0000", 2024-03-21T14:24:25.161 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.14393374100001, 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3237603789999999, 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23543 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.162 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879391+0000", 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879391+0000", 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879391+0000", 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.8100000000000002e-07 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.163 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879437+0000", 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.5945000000000001e-05 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879447+0000", 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 9.346e-06 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879453+0000", 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.3659999999999997e-06 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.164 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879502+0000", 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 4.8937999999999998e-05 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.879508+0000", 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.2009999999999996e-06 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761601+0000", 2024-03-21T14:24:25.165 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.88209316199999999 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.761610+0000", 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 8.8640000000000002e-06 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.193162+0000", 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4315513049999999 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.166 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.193344+0000", 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00018217299999999999 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.193505+0000", 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00016091 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.196608+0000", 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0031029600000000001 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.167 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.199046+0000", 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00243831 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.202709+0000", 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0036631419999999999 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.202763+0000", 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 5.4079999999999997e-05 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.168 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.203151+0000", 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00038799599999999999 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "description": "osd_op(client.25811.0:23558 11.4 11:2b9c5c65:::benchmark_data_smithi003_188959_object23557:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "initiated_at": "2024-03-21T14:21:25.895485+0000", 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "age": 179.127839572, 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.3081201619999998, 2024-03-21T14:24:25.169 INFO:teuthology.orchestra.run.smithi118.stdout: "type_data": { 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_info": { 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client": "client.25811", 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "tid": 23558 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "events": [ 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "initiated", 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895485+0000", 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.170 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "header_read", 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895485+0000", 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "throttled", 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895486+0000", 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2389999999999999e-06 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "all_read", 2024-03-21T14:24:25.171 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895515+0000", 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.8929e-05 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "dispatched", 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895527+0000", 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.1494999999999999e-05 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "queued_for_pg", 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895533+0000", 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.4370000000000004e-06 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.172 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895594+0000", 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 6.1252000000000002e-05 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:25.895606+0000", 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.2047e-05 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.764715+0000", 2024-03-21T14:24:25.173 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.86910908499999995 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for peered", 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:26.764733+0000", 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.7346e-05 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "reached_pg", 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.194365+0000", 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 1.4296321160000001 2024-03-21T14:24:25.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "started", 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.194522+0000", 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00015709200000000001 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "waiting for subops from 2,6", 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.194695+0000", 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.000172984 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "op_commit", 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.199058+0000", 2024-03-21T14:24:25.175 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.004363179 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.199086+0000", 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 2.7691e-05 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.203195+0000", 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.0041095990000000002 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.176 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "commit_sent", 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.203231+0000", 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 3.5488999999999999e-05 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "event": "done", 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "time": "2024-03-21T14:21:28.203605+0000", 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: "duration": 0.00037418199999999998 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-21T14:24:25.177 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-21T14:24:25.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:25.201 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:25.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:25.332 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:25.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:25 smithi118 ceph-mon[33862]: pgmap v1023: 158 pgs: 158 active+clean; 22 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T14:24:25.433 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.458 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:25.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:25.466 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:25 smithi003 ceph-mon[29576]: pgmap v1023: 158 pgs: 158 active+clean; 22 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T14:24:25.466 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:25 smithi003 ceph-mon[40510]: pgmap v1023: 158 pgs: 158 active+clean; 22 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-21T14:24:25.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:25.582 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:25.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:25.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:25.839 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:25.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:25.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:25.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:25.965 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:25.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:26.032 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-21T14:24:26.066 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:26.098 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:26.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:26.188 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:24:26.188 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:24:26.201 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:26.204 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:24:26.223 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:24:26.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.226 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:26.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:26.244 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:24:26.264 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:24:26.284 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:24:26.284 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:26.304 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:24:26.306 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:24:26.328 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:26.351 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:26.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:26.406 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/680020583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:26.453 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:26.477 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:26.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/680020583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:26.490 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/680020583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:26.579 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:26.588 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:24:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.603 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:26.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:26.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:26.728 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:24:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:24:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:24:26.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:26.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.856 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:26.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:26.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:26.982 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:26.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:27.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:27.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:27.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:27.233 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:27.335 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:27.359 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:27.429 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:27 smithi118 ceph-mon[33862]: pgmap v1024: 158 pgs: 158 active+clean; 22 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T14:24:27.461 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:27.486 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:27.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:27 smithi003 ceph-mon[29576]: pgmap v1024: 158 pgs: 158 active+clean; 22 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T14:24:27.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:27 smithi003 ceph-mon[40510]: pgmap v1024: 158 pgs: 158 active+clean; 22 MiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-21T14:24:27.588 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:27.611 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:27.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:27.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:24:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:24:27.713 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:27.714 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:27.737 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:27.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:27.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.864 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:27.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:27.966 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:27.991 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:27.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:28.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.119 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:28.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:28.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:28.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:28.318 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:28.318 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:28 smithi003 ceph-mon[29576]: pgmap v1025: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 689 op/s 2024-03-21T14:24:28.319 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:28.319 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:28 smithi003 ceph-mon[40510]: pgmap v1025: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 689 op/s 2024-03-21T14:24:28.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:28.370 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:28.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:28.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:28 smithi118 ceph-mon[33862]: pgmap v1025: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 689 op/s 2024-03-21T14:24:28.471 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.499 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:28.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:28.518 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:24:28.521 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:24:28.565 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-21T14:24:28.566 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-21T14:24:28.600 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:28.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.634 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:28.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:28.641 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:28.641 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:28.736 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:28.762 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:28.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:28.863 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:28.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:28.888 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:28.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:28.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:29.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.014 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:29.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:29.115 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:29.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.140 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:29.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:29.241 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:29.266 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:29.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:29.274 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1025,"stamp":"2024-03-21T14:24:28.091817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2807,"num_read_kb":2559,"num_write":222768,"num_write_kb":7096302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3952,"num_bytes_recovered":246878200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74470,"ondisk_log_size":74470,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5815448,"kb_used_data":20192,"kb_used_omap":21,"kb_used_meta":5794730,"kb_avail":744145768,"statfs":{"total":767960285184,"available":762005266432,"internally_reserved":0,"allocated":20676608,"data_stored":17812376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22067,"internal_metadata":5933803981},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-452132888,"num_objects":-6900,"num_object_clones":0,"num_object_copies":-20700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6900,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6900,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.016121"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":845,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153092+0000","last_change":"2024-03-21T14:23:04.359116+0000","last_active":"2024-03-21T14:24:21.153092+0000","last_peered":"2024-03-21T14:24:21.153092+0000","last_clean":"2024-03-21T14:24:21.153092+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:24:21.153092+0000","last_undegraded":"2024-03-21T14:24:21.153092+0000","last_fullsized":"2024-03-21T14:24:21.153092+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:04.358989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.358989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":875,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146323+0000","last_change":"2024-03-21T14:23:45.511561+0000","last_active":"2024-03-21T14:24:21.146323+0000","last_peered":"2024-03-21T14:24:21.146323+0000","last_clean":"2024-03-21T14:24:21.146323+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:24:21.146323+0000","last_undegraded":"2024-03-21T14:24:21.146323+0000","last_fullsized":"2024-03-21T14:24:21.146323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:45.511445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:23:45.511445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:14.138019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":422,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162176+0000","last_change":"2024-03-21T14:23:06.316701+0000","last_active":"2024-03-21T14:24:21.162176+0000","last_peered":"2024-03-21T14:24:21.162176+0000","last_clean":"2024-03-21T14:24:21.162176+0000","last_became_active":"2024-03-21T14:19:34.695760+0000","last_became_peered":"2024-03-21T14:19:34.695760+0000","last_unstale":"2024-03-21T14:24:21.162176+0000","last_undegraded":"2024-03-21T14:24:21.162176+0000","last_fullsized":"2024-03-21T14:24:21.162176+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:06.316132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:06.316132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:29.219900+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":921,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152106+0000","last_change":"2024-03-21T14:24:06.763081+0000","last_active":"2024-03-21T14:24:21.152106+0000","last_peered":"2024-03-21T14:24:21.152106+0000","last_clean":"2024-03-21T14:24:21.152106+0000","last_became_active":"2024-03-21T14:20:21.782187+0000","last_became_peered":"2024-03-21T14:20:21.782187+0000","last_unstale":"2024-03-21T14:24:21.152106+0000","last_undegraded":"2024-03-21T14:24:21.152106+0000","last_fullsized":"2024-03-21T14:24:21.152106+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:24:06.762955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:24:06.762955+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:25.336804+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":150,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154649+0000","last_change":"2024-03-21T14:23:07.529805+0000","last_active":"2024-03-21T14:24:21.154649+0000","last_peered":"2024-03-21T14:24:21.154649+0000","last_clean":"2024-03-21T14:24:21.154649+0000","last_became_active":"2024-03-21T14:20:21.782401+0000","last_became_peered":"2024-03-21T14:20:21.782401+0000","last_unstale":"2024-03-21T14:24:21.154649+0000","last_undegraded":"2024-03-21T14:24:21.154649+0000","last_fullsized":"2024-03-21T14:24:21.154649+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:07.529663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:07.529663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:22.958204+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":857,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.151994+0000","last_change":"2024-03-21T14:20:52.477097+0000","last_active":"2024-03-21T14:24:21.151994+0000","last_peered":"2024-03-21T14:24:21.151994+0000","last_clean":"2024-03-21T14:24:21.151994+0000","last_became_active":"2024-03-21T14:19:14.689459+0000","last_became_peered":"2024-03-21T14:19:14.689459+0000","last_unstale":"2024-03-21T14:24:21.151994+0000","last_undegraded":"2024-03-21T14:24:21.151994+0000","last_fullsized":"2024-03-21T14:24:21.151994+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:52.477009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.477009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:29.998919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":867,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152702+0000","last_change":"2024-03-21T14:20:37.939217+0000","last_active":"2024-03-21T14:24:21.152702+0000","last_peered":"2024-03-21T14:24:21.152702+0000","last_clean":"2024-03-21T14:24:21.152702+0000","last_became_active":"2024-03-21T14:19:14.695703+0000","last_became_peered":"2024-03-21T14:19:14.695703+0000","last_unstale":"2024-03-21T14:24:21.152702+0000","last_undegraded":"2024-03-21T14:24:21.152702+0000","last_fullsized":"2024-03-21T14:24:21.152702+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:37.939080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:37.939080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'46","reported_seq":924,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161848+0000","last_change":"2024-03-21T14:20:40.919011+0000","last_active":"2024-03-21T14:24:21.161848+0000","last_peered":"2024-03-21T14:24:21.161848+0000","last_clean":"2024-03-21T14:24:21.161848+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:24:21.161848+0000","last_undegraded":"2024-03-21T14:24:21.161848+0000","last_fullsized":"2024-03-21T14:24:21.161848+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-03-21T14:20:40.918895+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:20:40.918895+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":727,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146307+0000","last_change":"2024-03-21T14:22:57.984639+0000","last_active":"2024-03-21T14:24:21.146307+0000","last_peered":"2024-03-21T14:24:21.146307+0000","last_clean":"2024-03-21T14:24:21.146307+0000","last_became_active":"2024-03-21T14:19:34.690300+0000","last_became_peered":"2024-03-21T14:19:34.690300+0000","last_unstale":"2024-03-21T14:24:21.146307+0000","last_undegraded":"2024-03-21T14:24:21.146307+0000","last_fullsized":"2024-03-21T14:24:21.146307+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:57.984394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.984394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":873,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.140486+0000","last_change":"2024-03-21T14:22:48.510243+0000","last_active":"2024-03-21T14:24:21.140486+0000","last_peered":"2024-03-21T14:24:21.140486+0000","last_clean":"2024-03-21T14:24:21.140486+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:24:21.140486+0000","last_undegraded":"2024-03-21T14:24:21.140486+0000","last_fullsized":"2024-03-21T14:24:21.140486+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:48.510139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.510139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":666,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154140+0000","last_change":"2024-03-21T14:22:54.329383+0000","last_active":"2024-03-21T14:24:21.154140+0000","last_peered":"2024-03-21T14:24:21.154140+0000","last_clean":"2024-03-21T14:24:21.154140+0000","last_became_active":"2024-03-21T14:16:21.870892+0000","last_became_peered":"2024-03-21T14:16:21.870892+0000","last_unstale":"2024-03-21T14:24:21.154140+0000","last_undegraded":"2024-03-21T14:24:21.154140+0000","last_fullsized":"2024-03-21T14:24:21.154140+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.329270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.329270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"441'21","reported_seq":879,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153267+0000","last_change":"2024-03-21T14:22:48.380073+0000","last_active":"2024-03-21T14:24:21.153267+0000","last_peered":"2024-03-21T14:24:21.153267+0000","last_clean":"2024-03-21T14:24:21.153267+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:24:21.153267+0000","last_undegraded":"2024-03-21T14:24:21.153267+0000","last_fullsized":"2024-03-21T14:24:21.153267+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:22:48.379990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.379990+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":816,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146469+0000","last_change":"2024-03-21T14:22:57.054038+0000","last_active":"2024-03-21T14:24:21.146469+0000","last_peered":"2024-03-21T14:24:21.146469+0000","last_clean":"2024-03-21T14:24:21.146469+0000","last_became_active":"2024-03-21T14:20:21.776360+0000","last_became_peered":"2024-03-21T14:20:21.776360+0000","last_unstale":"2024-03-21T14:24:21.146469+0000","last_undegraded":"2024-03-21T14:24:21.146469+0000","last_fullsized":"2024-03-21T14:24:21.146469+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:57.053921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.053921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.452840+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":866,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155480+0000","last_change":"2024-03-21T14:21:47.782509+0000","last_active":"2024-03-21T14:24:21.155480+0000","last_peered":"2024-03-21T14:24:21.155480+0000","last_clean":"2024-03-21T14:24:21.155480+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:24:21.155480+0000","last_undegraded":"2024-03-21T14:24:21.155480+0000","last_fullsized":"2024-03-21T14:24:21.155480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:47.782401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.782401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.898881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":332,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156827+0000","last_change":"2024-03-21T14:21:53.335510+0000","last_active":"2024-03-21T14:24:21.156827+0000","last_peered":"2024-03-21T14:24:21.156827+0000","last_clean":"2024-03-21T14:24:21.156827+0000","last_became_active":"2024-03-21T14:16:21.862103+0000","last_became_peered":"2024-03-21T14:16:21.862103+0000","last_unstale":"2024-03-21T14:24:21.156827+0000","last_undegraded":"2024-03-21T14:24:21.156827+0000","last_fullsized":"2024-03-21T14:24:21.156827+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_clean_scrub_stamp":"2024-03-21T14:21:53.335395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"445'10","reported_seq":870,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153682+0000","last_change":"2024-03-21T14:22:57.290034+0000","last_active":"2024-03-21T14:24:21.153682+0000","last_peered":"2024-03-21T14:24:21.153682+0000","last_clean":"2024-03-21T14:24:21.153682+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:24:21.153682+0000","last_undegraded":"2024-03-21T14:24:21.153682+0000","last_fullsized":"2024-03-21T14:24:21.153682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'10","last_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_deep_scrub":"445'10","last_deep_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.289894+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":845,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154143+0000","last_change":"2024-03-21T14:22:32.629084+0000","last_active":"2024-03-21T14:24:21.154143+0000","last_peered":"2024-03-21T14:24:21.154143+0000","last_clean":"2024-03-21T14:24:21.154143+0000","last_became_active":"2024-03-21T14:12:00.532161+0000","last_became_peered":"2024-03-21T14:12:00.532161+0000","last_unstale":"2024-03-21T14:24:21.154143+0000","last_undegraded":"2024-03-21T14:24:21.154143+0000","last_fullsized":"2024-03-21T14:24:21.154143+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:32.628950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:32.628950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":858,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148123+0000","last_change":"2024-03-21T14:22:35.833709+0000","last_active":"2024-03-21T14:24:21.148123+0000","last_peered":"2024-03-21T14:24:21.148123+0000","last_clean":"2024-03-21T14:24:21.148123+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:24:21.148123+0000","last_undegraded":"2024-03-21T14:24:21.148123+0000","last_fullsized":"2024-03-21T14:24:21.148123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:35.833457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.833457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885482+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":716,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153595+0000","last_change":"2024-03-21T14:23:48.599708+0000","last_active":"2024-03-21T14:24:21.153595+0000","last_peered":"2024-03-21T14:24:21.153595+0000","last_clean":"2024-03-21T14:24:21.153595+0000","last_became_active":"2024-03-21T14:19:34.686106+0000","last_became_peered":"2024-03-21T14:19:34.686106+0000","last_unstale":"2024-03-21T14:24:21.153595+0000","last_undegraded":"2024-03-21T14:24:21.153595+0000","last_fullsized":"2024-03-21T14:24:21.153595+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:48.599583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:23:48.599583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:16.933578+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":790,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.159147+0000","last_change":"2024-03-21T14:21:50.323084+0000","last_active":"2024-03-21T14:24:21.159147+0000","last_peered":"2024-03-21T14:24:21.159147+0000","last_clean":"2024-03-21T14:24:21.159147+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:24:21.159147+0000","last_undegraded":"2024-03-21T14:24:21.159147+0000","last_fullsized":"2024-03-21T14:24:21.159147+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'34","last_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_deep_scrub":"432'34","last_deep_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.322580+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528115+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":862,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153263+0000","last_change":"2024-03-21T14:21:33.288900+0000","last_active":"2024-03-21T14:24:21.153263+0000","last_peered":"2024-03-21T14:24:21.153263+0000","last_clean":"2024-03-21T14:24:21.153263+0000","last_became_active":"2024-03-21T14:12:00.533799+0000","last_became_peered":"2024-03-21T14:12:00.533799+0000","last_unstale":"2024-03-21T14:24:21.153263+0000","last_undegraded":"2024-03-21T14:24:21.153263+0000","last_fullsized":"2024-03-21T14:24:21.153263+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:33.288792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:33.288792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":850,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157027+0000","last_change":"2024-03-21T14:22:29.551806+0000","last_active":"2024-03-21T14:24:21.157027+0000","last_peered":"2024-03-21T14:24:21.157027+0000","last_clean":"2024-03-21T14:24:21.157027+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:24:21.157027+0000","last_undegraded":"2024-03-21T14:24:21.157027+0000","last_fullsized":"2024-03-21T14:24:21.157027+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:29.551666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.551666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":858,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153313+0000","last_change":"2024-03-21T14:24:05.745866+0000","last_active":"2024-03-21T14:24:21.153313+0000","last_peered":"2024-03-21T14:24:21.153313+0000","last_clean":"2024-03-21T14:24:21.153313+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:24:21.153313+0000","last_undegraded":"2024-03-21T14:24:21.153313+0000","last_fullsized":"2024-03-21T14:24:21.153313+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:05.745734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:24:05.745734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:23.836874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":903,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.160185+0000","last_change":"2024-03-21T14:22:35.357291+0000","last_active":"2024-03-21T14:24:21.160185+0000","last_peered":"2024-03-21T14:24:21.160185+0000","last_clean":"2024-03-21T14:24:21.160185+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:24:21.160185+0000","last_undegraded":"2024-03-21T14:24:21.160185+0000","last_fullsized":"2024-03-21T14:24:21.160185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-03-21T14:22:35.357134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.357134+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":374,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.150807+0000","last_change":"2024-03-21T14:24:06.511883+0000","last_active":"2024-03-21T14:24:21.150807+0000","last_peered":"2024-03-21T14:24:21.150807+0000","last_clean":"2024-03-21T14:24:21.150807+0000","last_became_active":"2024-03-21T14:19:14.686523+0000","last_became_peered":"2024-03-21T14:19:14.686523+0000","last_unstale":"2024-03-21T14:24:21.150807+0000","last_undegraded":"2024-03-21T14:24:21.150807+0000","last_fullsized":"2024-03-21T14:24:21.150807+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:24:06.511735+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:24:06.511735+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:16.512627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"425'15","reported_seq":142,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.151737+0000","last_change":"2024-03-21T14:23:55.104612+0000","last_active":"2024-03-21T14:24:21.151737+0000","last_peered":"2024-03-21T14:24:21.151737+0000","last_clean":"2024-03-21T14:24:21.151737+0000","last_became_active":"2024-03-21T14:21:26.767116+0000","last_became_peered":"2024-03-21T14:21:26.767116+0000","last_unstale":"2024-03-21T14:24:21.151737+0000","last_undegraded":"2024-03-21T14:24:21.151737+0000","last_fullsized":"2024-03-21T14:24:21.151737+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'15","last_scrub_stamp":"2024-03-21T14:23:55.104477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:55.104477+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:01.118776+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":884,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146234+0000","last_change":"2024-03-21T14:23:47.508772+0000","last_active":"2024-03-21T14:24:21.146234+0000","last_peered":"2024-03-21T14:24:21.146234+0000","last_clean":"2024-03-21T14:24:21.146234+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:24:21.146234+0000","last_undegraded":"2024-03-21T14:24:21.146234+0000","last_fullsized":"2024-03-21T14:24:21.146234+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:23:47.508655+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:23:47.508655+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:51.250232+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":604,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157202+0000","last_change":"2024-03-21T14:23:47.842051+0000","last_active":"2024-03-21T14:24:21.157202+0000","last_peered":"2024-03-21T14:24:21.157202+0000","last_clean":"2024-03-21T14:24:21.157202+0000","last_became_active":"2024-03-21T14:20:21.790051+0000","last_became_peered":"2024-03-21T14:20:21.790051+0000","last_unstale":"2024-03-21T14:24:21.157202+0000","last_undegraded":"2024-03-21T14:24:21.157202+0000","last_fullsized":"2024-03-21T14:24:21.157202+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-21T14:23:47.841947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:47.841947+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:01.554818+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":862,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.147660+0000","last_change":"2024-03-21T14:24:06.928634+0000","last_active":"2024-03-21T14:24:21.147660+0000","last_peered":"2024-03-21T14:24:21.147660+0000","last_clean":"2024-03-21T14:24:21.147660+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:24:21.147660+0000","last_undegraded":"2024-03-21T14:24:21.147660+0000","last_fullsized":"2024-03-21T14:24:21.147660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:06.928522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:06.928522+0000","last_clean_scrub_stamp":"2024-03-21T14:24:06.928522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:07.837386+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":523,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157167+0000","last_change":"2024-03-21T14:22:58.625867+0000","last_active":"2024-03-21T14:24:21.157167+0000","last_peered":"2024-03-21T14:24:21.157167+0000","last_clean":"2024-03-21T14:24:21.157167+0000","last_became_active":"2024-03-21T14:19:14.675463+0000","last_became_peered":"2024-03-21T14:19:14.675463+0000","last_unstale":"2024-03-21T14:24:21.157167+0000","last_undegraded":"2024-03-21T14:24:21.157167+0000","last_fullsized":"2024-03-21T14:24:21.157167+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:58.625708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:22:58.625708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.285980+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":880,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161844+0000","last_change":"2024-03-21T14:20:45.936416+0000","last_active":"2024-03-21T14:24:21.161844+0000","last_peered":"2024-03-21T14:24:21.161844+0000","last_clean":"2024-03-21T14:24:21.161844+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:24:21.161844+0000","last_undegraded":"2024-03-21T14:24:21.161844+0000","last_fullsized":"2024-03-21T14:24:21.161844+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:45.936318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:20:45.936318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.406297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'32","reported_seq":610,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.149070+0000","last_change":"2024-03-21T14:23:58.466202+0000","last_active":"2024-03-21T14:24:21.149070+0000","last_peered":"2024-03-21T14:24:21.149070+0000","last_clean":"2024-03-21T14:24:21.149070+0000","last_became_active":"2024-03-21T14:16:21.869446+0000","last_became_peered":"2024-03-21T14:16:21.869446+0000","last_unstale":"2024-03-21T14:24:21.149070+0000","last_undegraded":"2024-03-21T14:24:21.149070+0000","last_fullsized":"2024-03-21T14:24:21.149070+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:23:58.465987+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-21T14:23:58.465987+0000","last_clean_scrub_stamp":"2024-03-21T14:23:58.465987+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:09.941332+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":221,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.163127+0000","last_change":"2024-03-21T14:22:40.390363+0000","last_active":"2024-03-21T14:24:21.163127+0000","last_peered":"2024-03-21T14:24:21.163127+0000","last_clean":"2024-03-21T14:24:21.163127+0000","last_became_active":"2024-03-21T14:19:14.679790+0000","last_became_peered":"2024-03-21T14:19:14.679790+0000","last_unstale":"2024-03-21T14:24:21.163127+0000","last_undegraded":"2024-03-21T14:24:21.163127+0000","last_fullsized":"2024-03-21T14:24:21.163127+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:40.390274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.390274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528293+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":540,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152111+0000","last_change":"2024-03-21T14:23:59.766842+0000","last_active":"2024-03-21T14:24:21.152111+0000","last_peered":"2024-03-21T14:24:21.152111+0000","last_clean":"2024-03-21T14:24:21.152111+0000","last_became_active":"2024-03-21T14:12:00.539861+0000","last_became_peered":"2024-03-21T14:12:00.539861+0000","last_unstale":"2024-03-21T14:24:21.152111+0000","last_undegraded":"2024-03-21T14:24:21.152111+0000","last_fullsized":"2024-03-21T14:24:21.152111+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:59.766720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:23:59.766720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:16.273534+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":334,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153407+0000","last_change":"2024-03-21T14:21:21.656390+0000","last_active":"2024-03-21T14:24:21.153407+0000","last_peered":"2024-03-21T14:24:21.153407+0000","last_clean":"2024-03-21T14:24:21.153407+0000","last_became_active":"2024-03-21T14:16:21.862793+0000","last_became_peered":"2024-03-21T14:16:21.862793+0000","last_unstale":"2024-03-21T14:24:21.153407+0000","last_undegraded":"2024-03-21T14:24:21.153407+0000","last_fullsized":"2024-03-21T14:24:21.153407+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:21.656278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:21.656278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"441'14","reported_seq":903,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162014+0000","last_change":"2024-03-21T14:22:49.297359+0000","last_active":"2024-03-21T14:24:21.162014+0000","last_peered":"2024-03-21T14:24:21.162014+0000","last_clean":"2024-03-21T14:24:21.162014+0000","last_became_active":"2024-03-21T14:12:00.537881+0000","last_became_peered":"2024-03-21T14:12:00.537881+0000","last_unstale":"2024-03-21T14:24:21.162014+0000","last_undegraded":"2024-03-21T14:24:21.162014+0000","last_fullsized":"2024-03-21T14:24:21.162014+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'14","last_scrub_stamp":"2024-03-21T14:22:49.297239+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.297239+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":879,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155196+0000","last_change":"2024-03-21T14:21:34.144597+0000","last_active":"2024-03-21T14:24:21.155196+0000","last_peered":"2024-03-21T14:24:21.155196+0000","last_clean":"2024-03-21T14:24:21.155196+0000","last_became_active":"2024-03-21T14:19:34.693279+0000","last_became_peered":"2024-03-21T14:19:34.693279+0000","last_unstale":"2024-03-21T14:24:21.155196+0000","last_undegraded":"2024-03-21T14:24:21.155196+0000","last_fullsized":"2024-03-21T14:24:21.155196+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:34.144484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:13.840591+0000","last_clean_scrub_stamp":"2024-03-21T14:21:34.144484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":897,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156690+0000","last_change":"2024-03-21T14:24:07.911902+0000","last_active":"2024-03-21T14:24:21.156690+0000","last_peered":"2024-03-21T14:24:21.156690+0000","last_clean":"2024-03-21T14:24:21.156690+0000","last_became_active":"2024-03-21T14:12:00.544916+0000","last_became_peered":"2024-03-21T14:12:00.544916+0000","last_unstale":"2024-03-21T14:24:21.156690+0000","last_undegraded":"2024-03-21T14:24:21.156690+0000","last_fullsized":"2024-03-21T14:24:21.156690+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:07.911674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:24:07.911674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:11.377069+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":337,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155287+0000","last_change":"2024-03-21T14:23:04.813785+0000","last_active":"2024-03-21T14:24:21.155287+0000","last_peered":"2024-03-21T14:24:21.155287+0000","last_clean":"2024-03-21T14:24:21.155287+0000","last_became_active":"2024-03-21T14:16:34.620846+0000","last_became_peered":"2024-03-21T14:16:34.620846+0000","last_unstale":"2024-03-21T14:24:21.155287+0000","last_undegraded":"2024-03-21T14:24:21.155287+0000","last_fullsized":"2024-03-21T14:24:21.155287+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:23:04.813659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.813659+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885511+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"441'32","reported_seq":937,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161705+0000","last_change":"2024-03-21T14:22:51.290072+0000","last_active":"2024-03-21T14:24:21.161705+0000","last_peered":"2024-03-21T14:24:21.161705+0000","last_clean":"2024-03-21T14:24:21.161705+0000","last_became_active":"2024-03-21T14:12:00.531749+0000","last_became_peered":"2024-03-21T14:12:00.531749+0000","last_unstale":"2024-03-21T14:24:21.161705+0000","last_undegraded":"2024-03-21T14:24:21.161705+0000","last_fullsized":"2024-03-21T14:24:21.161705+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:22:51.289757+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.289757+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":343,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146094+0000","last_change":"2024-03-21T14:23:03.865434+0000","last_active":"2024-03-21T14:24:21.146094+0000","last_peered":"2024-03-21T14:24:21.146094+0000","last_clean":"2024-03-21T14:24:21.146094+0000","last_became_active":"2024-03-21T14:16:21.864286+0000","last_became_peered":"2024-03-21T14:16:21.864286+0000","last_unstale":"2024-03-21T14:24:21.146094+0000","last_undegraded":"2024-03-21T14:24:21.146094+0000","last_fullsized":"2024-03-21T14:24:21.146094+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:03.865222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:03.865222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.684378+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":859,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155109+0000","last_change":"2024-03-21T14:21:41.085311+0000","last_active":"2024-03-21T14:24:21.155109+0000","last_peered":"2024-03-21T14:24:21.155109+0000","last_clean":"2024-03-21T14:24:21.155109+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:24:21.155109+0000","last_undegraded":"2024-03-21T14:24:21.155109+0000","last_fullsized":"2024-03-21T14:24:21.155109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:41.085192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.085192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":883,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156293+0000","last_change":"2024-03-21T14:22:53.685161+0000","last_active":"2024-03-21T14:24:21.156293+0000","last_peered":"2024-03-21T14:24:21.156293+0000","last_clean":"2024-03-21T14:24:21.156293+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:24:21.156293+0000","last_undegraded":"2024-03-21T14:24:21.156293+0000","last_fullsized":"2024-03-21T14:24:21.156293+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:53.685078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:53.685078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.901709+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"441'21","reported_seq":922,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148521+0000","last_change":"2024-03-21T14:23:00.867969+0000","last_active":"2024-03-21T14:24:21.148521+0000","last_peered":"2024-03-21T14:24:21.148521+0000","last_clean":"2024-03-21T14:24:21.148521+0000","last_became_active":"2024-03-21T14:19:14.682980+0000","last_became_peered":"2024-03-21T14:19:14.682980+0000","last_unstale":"2024-03-21T14:24:21.148521+0000","last_undegraded":"2024-03-21T14:24:21.148521+0000","last_fullsized":"2024-03-21T14:24:21.148521+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:23:00.867800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:00.867800+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":150,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161392+0000","last_change":"2024-03-21T14:20:42.950594+0000","last_active":"2024-03-21T14:24:21.161392+0000","last_peered":"2024-03-21T14:24:21.161392+0000","last_clean":"2024-03-21T14:24:21.161392+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:24:21.161392+0000","last_undegraded":"2024-03-21T14:24:21.161392+0000","last_fullsized":"2024-03-21T14:24:21.161392+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:42.950500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:42.950500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":887,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148427+0000","last_change":"2024-03-21T14:24:02.640134+0000","last_active":"2024-03-21T14:24:21.148427+0000","last_peered":"2024-03-21T14:24:21.148427+0000","last_clean":"2024-03-21T14:24:21.148427+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:24:21.148427+0000","last_undegraded":"2024-03-21T14:24:21.148427+0000","last_fullsized":"2024-03-21T14:24:21.148427+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:02.640026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:24:02.640026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:15.321422+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":873,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156924+0000","last_change":"2024-03-21T14:21:20.339156+0000","last_active":"2024-03-21T14:24:21.156924+0000","last_peered":"2024-03-21T14:24:21.156924+0000","last_clean":"2024-03-21T14:24:21.156924+0000","last_became_active":"2024-03-21T14:19:14.690097+0000","last_became_peered":"2024-03-21T14:19:14.690097+0000","last_unstale":"2024-03-21T14:24:21.156924+0000","last_undegraded":"2024-03-21T14:24:21.156924+0000","last_fullsized":"2024-03-21T14:24:21.156924+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:20.339046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:20.339046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":964,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153784+0000","last_change":"2024-03-21T14:21:50.314232+0000","last_active":"2024-03-21T14:24:21.153784+0000","last_peered":"2024-03-21T14:24:21.153784+0000","last_clean":"2024-03-21T14:24:21.153784+0000","last_became_active":"2024-03-21T14:21:26.767846+0000","last_became_peered":"2024-03-21T14:21:26.767846+0000","last_unstale":"2024-03-21T14:24:21.153784+0000","last_undegraded":"2024-03-21T14:24:21.153784+0000","last_fullsized":"2024-03-21T14:24:21.153784+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'24","last_scrub_stamp":"2024-03-21T14:21:50.314134+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.314134+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":748,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157083+0000","last_change":"2024-03-21T14:23:57.913008+0000","last_active":"2024-03-21T14:24:21.157083+0000","last_peered":"2024-03-21T14:24:21.157083+0000","last_clean":"2024-03-21T14:24:21.157083+0000","last_became_active":"2024-03-21T14:20:21.784416+0000","last_became_peered":"2024-03-21T14:20:21.784416+0000","last_unstale":"2024-03-21T14:24:21.157083+0000","last_undegraded":"2024-03-21T14:24:21.157083+0000","last_fullsized":"2024-03-21T14:24:21.157083+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:57.912590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:57.912590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:20.819713+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"307'10","reported_seq":1103,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.147874+0000","last_change":"2024-03-21T14:21:50.631849+0000","last_active":"2024-03-21T14:24:21.147874+0000","last_peered":"2024-03-21T14:24:21.147874+0000","last_clean":"2024-03-21T14:24:21.147874+0000","last_became_active":"2024-03-21T14:16:21.869412+0000","last_became_peered":"2024-03-21T14:16:21.869412+0000","last_unstale":"2024-03-21T14:24:21.147874+0000","last_undegraded":"2024-03-21T14:24:21.147874+0000","last_fullsized":"2024-03-21T14:24:21.147874+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'10","last_scrub_stamp":"2024-03-21T14:21:50.631459+0000","last_deep_scrub":"290'9","last_deep_scrub_stamp":"2024-03-21T14:15:37.540505+0000","last_clean_scrub_stamp":"2024-03-21T14:21:50.631459+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885215+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":885,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157282+0000","last_change":"2024-03-21T14:23:52.837199+0000","last_active":"2024-03-21T14:24:21.157282+0000","last_peered":"2024-03-21T14:24:21.157282+0000","last_clean":"2024-03-21T14:24:21.157282+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:24:21.157282+0000","last_undegraded":"2024-03-21T14:24:21.157282+0000","last_fullsized":"2024-03-21T14:24:21.157282+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:52.837122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:17.788113+0000","last_clean_scrub_stamp":"2024-03-21T14:23:52.837122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:10.825448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'35","reported_seq":905,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152329+0000","last_change":"2024-03-21T14:21:18.712180+0000","last_active":"2024-03-21T14:24:21.152329+0000","last_peered":"2024-03-21T14:24:21.152329+0000","last_clean":"2024-03-21T14:24:21.152329+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:24:21.152329+0000","last_undegraded":"2024-03-21T14:24:21.152329+0000","last_fullsized":"2024-03-21T14:24:21.152329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:18.712011+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:21:18.712011+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":839,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154513+0000","last_change":"2024-03-21T14:20:38.602527+0000","last_active":"2024-03-21T14:24:21.154513+0000","last_peered":"2024-03-21T14:24:21.154513+0000","last_clean":"2024-03-21T14:24:21.154513+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:24:21.154513+0000","last_undegraded":"2024-03-21T14:24:21.154513+0000","last_fullsized":"2024-03-21T14:24:21.154513+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:38.602405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:20:38.602405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:29.998959+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":203,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152535+0000","last_change":"2024-03-21T14:20:44.917571+0000","last_active":"2024-03-21T14:24:21.152535+0000","last_peered":"2024-03-21T14:24:21.152535+0000","last_clean":"2024-03-21T14:24:21.152535+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:24:21.152535+0000","last_undegraded":"2024-03-21T14:24:21.152535+0000","last_fullsized":"2024-03-21T14:24:21.152535+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:20:44.917480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:20:44.917480+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.402683+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":150,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162558+0000","last_change":"2024-03-21T14:22:43.375315+0000","last_active":"2024-03-21T14:24:21.162558+0000","last_peered":"2024-03-21T14:24:21.162558+0000","last_clean":"2024-03-21T14:24:21.162558+0000","last_became_active":"2024-03-21T14:20:21.784349+0000","last_became_peered":"2024-03-21T14:20:21.784349+0000","last_unstale":"2024-03-21T14:24:21.162558+0000","last_undegraded":"2024-03-21T14:24:21.162558+0000","last_fullsized":"2024-03-21T14:24:21.162558+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.375045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.375045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"441'25","reported_seq":768,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156627+0000","last_change":"2024-03-21T14:23:59.864410+0000","last_active":"2024-03-21T14:24:21.156627+0000","last_peered":"2024-03-21T14:24:21.156627+0000","last_clean":"2024-03-21T14:24:21.156627+0000","last_became_active":"2024-03-21T14:16:21.866680+0000","last_became_peered":"2024-03-21T14:16:21.866680+0000","last_unstale":"2024-03-21T14:24:21.156627+0000","last_undegraded":"2024-03-21T14:24:21.156627+0000","last_fullsized":"2024-03-21T14:24:21.156627+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-21T14:23:59.864286+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:23:59.864286+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:03.471787+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":696,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.159713+0000","last_change":"2024-03-21T14:21:49.278092+0000","last_active":"2024-03-21T14:24:21.159713+0000","last_peered":"2024-03-21T14:24:21.159713+0000","last_clean":"2024-03-21T14:24:21.159713+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:24:21.159713+0000","last_undegraded":"2024-03-21T14:24:21.159713+0000","last_fullsized":"2024-03-21T14:24:21.159713+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:49.277585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.277585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528144+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":150,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154480+0000","last_change":"2024-03-21T14:24:07.445043+0000","last_active":"2024-03-21T14:24:21.154480+0000","last_peered":"2024-03-21T14:24:21.154480+0000","last_clean":"2024-03-21T14:24:21.154480+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:24:21.154480+0000","last_undegraded":"2024-03-21T14:24:21.154480+0000","last_fullsized":"2024-03-21T14:24:21.154480+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:07.444921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:47.399758+0000","last_clean_scrub_stamp":"2024-03-21T14:24:07.444921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:11.851414+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":859,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152737+0000","last_change":"2024-03-21T14:23:44.712504+0000","last_active":"2024-03-21T14:24:21.152737+0000","last_peered":"2024-03-21T14:24:21.152737+0000","last_clean":"2024-03-21T14:24:21.152737+0000","last_became_active":"2024-03-21T14:21:26.767611+0000","last_became_peered":"2024-03-21T14:21:26.767611+0000","last_unstale":"2024-03-21T14:24:21.152737+0000","last_undegraded":"2024-03-21T14:24:21.152737+0000","last_fullsized":"2024-03-21T14:24:21.152737+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:44.712391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:44.712391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:08.665286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"441'25","reported_seq":569,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152922+0000","last_change":"2024-03-21T14:21:49.128441+0000","last_active":"2024-03-21T14:24:21.152922+0000","last_peered":"2024-03-21T14:24:21.152922+0000","last_clean":"2024-03-21T14:24:21.152922+0000","last_became_active":"2024-03-21T14:12:20.562757+0000","last_became_peered":"2024-03-21T14:12:20.562757+0000","last_unstale":"2024-03-21T14:24:21.152922+0000","last_undegraded":"2024-03-21T14:24:21.152922+0000","last_fullsized":"2024-03-21T14:24:21.152922+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'19","last_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_deep_scrub":"432'19","last_deep_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_clean_scrub_stamp":"2024-03-21T14:21:49.128273+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":877,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155741+0000","last_change":"2024-03-21T14:22:43.886872+0000","last_active":"2024-03-21T14:24:21.155741+0000","last_peered":"2024-03-21T14:24:21.155741+0000","last_clean":"2024-03-21T14:24:21.155741+0000","last_became_active":"2024-03-21T14:12:00.544416+0000","last_became_peered":"2024-03-21T14:12:00.544416+0000","last_unstale":"2024-03-21T14:24:21.155741+0000","last_undegraded":"2024-03-21T14:24:21.155741+0000","last_fullsized":"2024-03-21T14:24:21.155741+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.886754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.886754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":333,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152788+0000","last_change":"2024-03-21T14:21:26.752265+0000","last_active":"2024-03-21T14:24:21.152788+0000","last_peered":"2024-03-21T14:24:21.152788+0000","last_clean":"2024-03-21T14:24:21.152788+0000","last_became_active":"2024-03-21T14:16:21.870538+0000","last_became_peered":"2024-03-21T14:16:21.870538+0000","last_unstale":"2024-03-21T14:24:21.152788+0000","last_undegraded":"2024-03-21T14:24:21.152788+0000","last_fullsized":"2024-03-21T14:24:21.152788+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:26.752126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:09.506940+0000","last_clean_scrub_stamp":"2024-03-21T14:21:26.752126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":102,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.151520+0000","last_change":"2024-03-21T14:24:05.078492+0000","last_active":"2024-03-21T14:24:21.151520+0000","last_peered":"2024-03-21T14:24:21.151520+0000","last_clean":"2024-03-21T14:24:21.151520+0000","last_became_active":"2024-03-21T14:21:26.765914+0000","last_became_peered":"2024-03-21T14:21:26.765914+0000","last_unstale":"2024-03-21T14:24:21.151520+0000","last_undegraded":"2024-03-21T14:24:21.151520+0000","last_fullsized":"2024-03-21T14:24:21.151520+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:05.078358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:24:05.078358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:21.603071+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'18","reported_seq":167,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153077+0000","last_change":"2024-03-21T14:24:00.379779+0000","last_active":"2024-03-21T14:24:21.153077+0000","last_peered":"2024-03-21T14:24:21.153077+0000","last_clean":"2024-03-21T14:24:21.153077+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:24:21.153077+0000","last_undegraded":"2024-03-21T14:24:21.153077+0000","last_fullsized":"2024-03-21T14:24:21.153077+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'18","last_scrub_stamp":"2024-03-21T14:24:00.379651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:24:00.379651+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:19.229186+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"457'15508","reported_seq":15678,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.979245+0000","last_change":"2024-03-21T14:24:07.815634+0000","last_active":"2024-03-21T14:24:22.979245+0000","last_peered":"2024-03-21T14:24:22.979245+0000","last_clean":"2024-03-21T14:24:22.979245+0000","last_became_active":"2024-03-21T14:20:59.386663+0000","last_became_peered":"2024-03-21T14:20:59.386663+0000","last_unstale":"2024-03-21T14:24:22.979245+0000","last_undegraded":"2024-03-21T14:24:22.979245+0000","last_fullsized":"2024-03-21T14:24:22.979245+0000","mapping_epoch":411,"log_start":"451'11502","ondisk_log_start":"451'11502","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'14399","last_scrub_stamp":"2024-03-21T14:24:07.815416+0000","last_deep_scrub":"455'14399","last_deep_scrub_stamp":"2024-03-21T14:24:07.815416+0000","last_clean_scrub_stamp":"2024-03-21T14:24:07.815416+0000","objects_scrubbed":1358,"log_size":4006,"log_dups_size":0,"ondisk_log_size":4006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:28.936042+0000","scrub_duration":8349,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14150,"num_write_kb":452800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":740,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.147759+0000","last_change":"2024-03-21T14:22:52.940837+0000","last_active":"2024-03-21T14:24:21.147759+0000","last_peered":"2024-03-21T14:24:21.147759+0000","last_clean":"2024-03-21T14:24:21.147759+0000","last_became_active":"2024-03-21T14:16:21.867212+0000","last_became_peered":"2024-03-21T14:16:21.867212+0000","last_unstale":"2024-03-21T14:24:21.147759+0000","last_undegraded":"2024-03-21T14:24:21.147759+0000","last_fullsized":"2024-03-21T14:24:21.147759+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_clean_scrub_stamp":"2024-03-21T14:22:52.940702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885699+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":921,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157494+0000","last_change":"2024-03-21T14:23:49.812603+0000","last_active":"2024-03-21T14:24:21.157494+0000","last_peered":"2024-03-21T14:24:21.157494+0000","last_clean":"2024-03-21T14:24:21.157494+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:24:21.157494+0000","last_undegraded":"2024-03-21T14:24:21.157494+0000","last_fullsized":"2024-03-21T14:24:21.157494+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:23:49.812444+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:19:42.440997+0000","last_clean_scrub_stamp":"2024-03-21T14:23:49.812444+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:10.140498+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"441'39","reported_seq":813,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156549+0000","last_change":"2024-03-21T14:22:37.805371+0000","last_active":"2024-03-21T14:24:21.156549+0000","last_peered":"2024-03-21T14:24:21.156549+0000","last_clean":"2024-03-21T14:24:21.156549+0000","last_became_active":"2024-03-21T14:16:21.863065+0000","last_became_peered":"2024-03-21T14:16:21.863065+0000","last_unstale":"2024-03-21T14:24:21.156549+0000","last_undegraded":"2024-03-21T14:24:21.156549+0000","last_fullsized":"2024-03-21T14:24:21.156549+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'27","last_scrub_stamp":"2024-03-21T14:22:37.805239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:37.805239+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.898960+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"372'10","reported_seq":1158,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.143168+0000","last_change":"2024-03-21T14:24:04.829291+0000","last_active":"2024-03-21T14:24:22.143168+0000","last_peered":"2024-03-21T14:24:22.143168+0000","last_clean":"2024-03-21T14:24:22.143168+0000","last_became_active":"2024-03-21T14:12:00.534008+0000","last_became_peered":"2024-03-21T14:12:00.534008+0000","last_unstale":"2024-03-21T14:24:22.143168+0000","last_undegraded":"2024-03-21T14:24:22.143168+0000","last_fullsized":"2024-03-21T14:24:22.143168+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:24:04.829201+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:19:47.479770+0000","last_clean_scrub_stamp":"2024-03-21T14:24:04.829201+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:28.509035+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"457'13962","reported_seq":14092,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.991711+0000","last_change":"2024-03-21T14:23:55.191722+0000","last_active":"2024-03-21T14:24:22.991711+0000","last_peered":"2024-03-21T14:24:22.991711+0000","last_clean":"2024-03-21T14:24:22.991711+0000","last_became_active":"2024-03-21T14:20:59.241791+0000","last_became_peered":"2024-03-21T14:20:59.241791+0000","last_unstale":"2024-03-21T14:24:22.991711+0000","last_undegraded":"2024-03-21T14:24:22.991711+0000","last_fullsized":"2024-03-21T14:24:22.991711+0000","mapping_epoch":411,"log_start":"445'8500","ondisk_log_start":"445'8500","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"453'12136","last_scrub_stamp":"2024-03-21T14:23:55.191435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:23:55.191435+0000","objects_scrubbed":1855,"log_size":5462,"log_dups_size":0,"ondisk_log_size":5462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:09.310729+0000","scrub_duration":878,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13962,"num_write_kb":446784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":609,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154890+0000","last_change":"2024-03-21T14:23:44.449144+0000","last_active":"2024-03-21T14:24:21.154890+0000","last_peered":"2024-03-21T14:24:21.154890+0000","last_clean":"2024-03-21T14:24:21.154890+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:24:21.154890+0000","last_undegraded":"2024-03-21T14:24:21.154890+0000","last_fullsized":"2024-03-21T14:24:21.154890+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:44.449016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:44.449016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:07.449061+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":899,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157061+0000","last_change":"2024-03-21T14:23:56.886568+0000","last_active":"2024-03-21T14:24:21.157061+0000","last_peered":"2024-03-21T14:24:21.157061+0000","last_clean":"2024-03-21T14:24:21.157061+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:24:21.157061+0000","last_undegraded":"2024-03-21T14:24:21.157061+0000","last_fullsized":"2024-03-21T14:24:21.157061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:56.886448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:50.747341+0000","last_clean_scrub_stamp":"2024-03-21T14:23:56.886448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:57.178943+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"441'24","reported_seq":950,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152826+0000","last_change":"2024-03-21T14:23:47.639293+0000","last_active":"2024-03-21T14:24:21.152826+0000","last_peered":"2024-03-21T14:24:21.152826+0000","last_clean":"2024-03-21T14:24:21.152826+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:24:21.152826+0000","last_undegraded":"2024-03-21T14:24:21.152826+0000","last_fullsized":"2024-03-21T14:24:21.152826+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-21T14:23:47.639144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:47.639144+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:17.150777+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":708,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154172+0000","last_change":"2024-03-21T14:21:54.295031+0000","last_active":"2024-03-21T14:24:21.154172+0000","last_peered":"2024-03-21T14:24:21.154172+0000","last_clean":"2024-03-21T14:24:21.154172+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:24:21.154172+0000","last_undegraded":"2024-03-21T14:24:21.154172+0000","last_fullsized":"2024-03-21T14:24:21.154172+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.294885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.294885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528021+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"457'13618","reported_seq":13722,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.944159+0000","last_change":"2024-03-21T14:20:59.387129+0000","last_active":"2024-03-21T14:24:22.944159+0000","last_peered":"2024-03-21T14:24:22.944159+0000","last_clean":"2024-03-21T14:24:22.944159+0000","last_became_active":"2024-03-21T14:20:59.386704+0000","last_became_peered":"2024-03-21T14:20:59.386704+0000","last_unstale":"2024-03-21T14:24:22.944159+0000","last_undegraded":"2024-03-21T14:24:22.944159+0000","last_fullsized":"2024-03-21T14:24:22.944159+0000","mapping_epoch":411,"log_start":"447'9400","ondisk_log_start":"447'9400","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4218,"log_dups_size":0,"ondisk_log_size":4218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.406188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13618,"num_write_kb":435776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":229,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154263+0000","last_change":"2024-03-21T14:22:54.306435+0000","last_active":"2024-03-21T14:24:21.154263+0000","last_peered":"2024-03-21T14:24:21.154263+0000","last_clean":"2024-03-21T14:24:21.154263+0000","last_became_active":"2024-03-21T14:19:14.700785+0000","last_became_peered":"2024-03-21T14:19:14.700785+0000","last_unstale":"2024-03-21T14:24:21.154263+0000","last_undegraded":"2024-03-21T14:24:21.154263+0000","last_fullsized":"2024-03-21T14:24:21.154263+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:54.306329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:54.306329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.528689+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":669,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154467+0000","last_change":"2024-03-21T14:23:04.356857+0000","last_active":"2024-03-21T14:24:21.154467+0000","last_peered":"2024-03-21T14:24:21.154467+0000","last_clean":"2024-03-21T14:24:21.154467+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:24:21.154467+0000","last_undegraded":"2024-03-21T14:24:21.154467+0000","last_fullsized":"2024-03-21T14:24:21.154467+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:04.356730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.356730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.865958+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":94,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153309+0000","last_change":"2024-03-21T14:23:06.509103+0000","last_active":"2024-03-21T14:24:21.153309+0000","last_peered":"2024-03-21T14:24:21.153309+0000","last_clean":"2024-03-21T14:24:21.153309+0000","last_became_active":"2024-03-21T14:21:26.764088+0000","last_became_peered":"2024-03-21T14:21:26.764088+0000","last_unstale":"2024-03-21T14:24:21.153309+0000","last_undegraded":"2024-03-21T14:24:21.153309+0000","last_fullsized":"2024-03-21T14:24:21.153309+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:06.509028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:06.509028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.147435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.a","version":"441'60","reported_seq":848,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148141+0000","last_change":"2024-03-21T14:23:49.533942+0000","last_active":"2024-03-21T14:24:21.148141+0000","last_peered":"2024-03-21T14:24:21.148141+0000","last_clean":"2024-03-21T14:24:21.148141+0000","last_became_active":"2024-03-21T14:12:00.528704+0000","last_became_peered":"2024-03-21T14:12:00.528704+0000","last_unstale":"2024-03-21T14:24:21.148141+0000","last_undegraded":"2024-03-21T14:24:21.148141+0000","last_fullsized":"2024-03-21T14:24:21.148141+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'60","last_scrub_stamp":"2024-03-21T14:23:49.531209+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:23:49.531209+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:56.845474+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"457'13814","reported_seq":16804,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:23.006043+0000","last_change":"2024-03-21T14:22:56.879969+0000","last_active":"2024-03-21T14:24:23.006043+0000","last_peered":"2024-03-21T14:24:23.006043+0000","last_clean":"2024-03-21T14:24:23.006043+0000","last_became_active":"2024-03-21T14:21:34.716304+0000","last_became_peered":"2024-03-21T14:21:34.716304+0000","last_unstale":"2024-03-21T14:24:23.006043+0000","last_undegraded":"2024-03-21T14:24:23.006043+0000","last_fullsized":"2024-03-21T14:24:23.006043+0000","mapping_epoch":427,"log_start":"446'9226","ondisk_log_start":"446'9226","created":411,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'7840","last_scrub_stamp":"2024-03-21T14:22:56.879850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:56.879850+0000","objects_scrubbed":6014,"log_size":4588,"log_dups_size":0,"ondisk_log_size":4588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.993137+0000","scrub_duration":1210,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13814,"num_write_kb":442048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1405,"num_bytes_recovered":92078080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":228,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.150095+0000","last_change":"2024-03-21T14:22:29.603576+0000","last_active":"2024-03-21T14:24:21.150095+0000","last_peered":"2024-03-21T14:24:21.150095+0000","last_clean":"2024-03-21T14:24:21.150095+0000","last_became_active":"2024-03-21T14:19:34.690841+0000","last_became_peered":"2024-03-21T14:19:34.690841+0000","last_unstale":"2024-03-21T14:24:21.150095+0000","last_undegraded":"2024-03-21T14:24:21.150095+0000","last_fullsized":"2024-03-21T14:24:21.150095+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:29.603450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:05:34.000689+0000","last_clean_scrub_stamp":"2024-03-21T14:22:29.603450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528173+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":708,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.163350+0000","last_change":"2024-03-21T14:21:46.343726+0000","last_active":"2024-03-21T14:24:21.163350+0000","last_peered":"2024-03-21T14:24:21.163350+0000","last_clean":"2024-03-21T14:24:21.163350+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:24:21.163350+0000","last_undegraded":"2024-03-21T14:24:21.163350+0000","last_fullsized":"2024-03-21T14:24:21.163350+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.343600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.343600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528081+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":45,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.151485+0000","last_change":"2024-03-21T14:23:57.068323+0000","last_active":"2024-03-21T14:24:21.151485+0000","last_peered":"2024-03-21T14:24:21.151485+0000","last_clean":"2024-03-21T14:24:21.151485+0000","last_became_active":"2024-03-21T14:22:50.083698+0000","last_became_peered":"2024-03-21T14:22:50.083698+0000","last_unstale":"2024-03-21T14:24:21.151485+0000","last_undegraded":"2024-03-21T14:24:21.151485+0000","last_fullsized":"2024-03-21T14:24:21.151485+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:57.068246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:23:57.068246+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:23.516484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.b","version":"441'21","reported_seq":380,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.158282+0000","last_change":"2024-03-21T14:24:03.444498+0000","last_active":"2024-03-21T14:24:21.158282+0000","last_peered":"2024-03-21T14:24:21.158282+0000","last_clean":"2024-03-21T14:24:21.158282+0000","last_became_active":"2024-03-21T14:19:14.692456+0000","last_became_peered":"2024-03-21T14:19:14.692456+0000","last_unstale":"2024-03-21T14:24:21.158282+0000","last_undegraded":"2024-03-21T14:24:21.158282+0000","last_fullsized":"2024-03-21T14:24:21.158282+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:24:03.444356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:24:03.444356+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:19.660764+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.b","version":"457'13374","reported_seq":13478,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:23.023319+0000","last_change":"2024-03-21T14:20:59.241783+0000","last_active":"2024-03-21T14:24:23.023319+0000","last_peered":"2024-03-21T14:24:23.023319+0000","last_clean":"2024-03-21T14:24:23.023319+0000","last_became_active":"2024-03-21T14:20:59.241477+0000","last_became_peered":"2024-03-21T14:20:59.241477+0000","last_unstale":"2024-03-21T14:24:23.023319+0000","last_undegraded":"2024-03-21T14:24:23.023319+0000","last_fullsized":"2024-03-21T14:24:23.023319+0000","mapping_epoch":411,"log_start":"447'9200","ondisk_log_start":"447'9200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.408640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13374,"num_write_kb":427968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":844,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.150204+0000","last_change":"2024-03-21T14:22:55.310949+0000","last_active":"2024-03-21T14:24:21.150204+0000","last_peered":"2024-03-21T14:24:21.150204+0000","last_clean":"2024-03-21T14:24:21.150204+0000","last_became_active":"2024-03-21T14:19:14.689666+0000","last_became_peered":"2024-03-21T14:19:14.689666+0000","last_unstale":"2024-03-21T14:24:21.150204+0000","last_undegraded":"2024-03-21T14:24:21.150204+0000","last_fullsized":"2024-03-21T14:24:21.150204+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:55.310815+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:55.310815+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528326+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":848,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153018+0000","last_change":"2024-03-21T14:20:50.816435+0000","last_active":"2024-03-21T14:24:21.153018+0000","last_peered":"2024-03-21T14:24:21.153018+0000","last_clean":"2024-03-21T14:24:21.153018+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:24:21.153018+0000","last_undegraded":"2024-03-21T14:24:21.153018+0000","last_fullsized":"2024-03-21T14:24:21.153018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:50.816334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:20:50.816334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.405609+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":367,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.147981+0000","last_change":"2024-03-21T14:22:53.863644+0000","last_active":"2024-03-21T14:24:21.147981+0000","last_peered":"2024-03-21T14:24:21.147981+0000","last_clean":"2024-03-21T14:24:21.147981+0000","last_became_active":"2024-03-21T14:16:21.864111+0000","last_became_peered":"2024-03-21T14:16:21.864111+0000","last_unstale":"2024-03-21T14:24:21.147981+0000","last_undegraded":"2024-03-21T14:24:21.147981+0000","last_fullsized":"2024-03-21T14:24:21.147981+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:22:53.863327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:22:53.863327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885643+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":857,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153977+0000","last_change":"2024-03-21T14:22:33.543481+0000","last_active":"2024-03-21T14:24:21.153977+0000","last_peered":"2024-03-21T14:24:21.153977+0000","last_clean":"2024-03-21T14:24:21.153977+0000","last_became_active":"2024-03-21T14:12:00.541449+0000","last_became_peered":"2024-03-21T14:12:00.541449+0000","last_unstale":"2024-03-21T14:24:21.153977+0000","last_undegraded":"2024-03-21T14:24:21.153977+0000","last_fullsized":"2024-03-21T14:24:21.153977+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:33.543370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:33.543370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"441'66","reported_seq":841,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.150920+0000","last_change":"2024-03-21T14:22:45.361704+0000","last_active":"2024-03-21T14:24:21.150920+0000","last_peered":"2024-03-21T14:24:21.150920+0000","last_clean":"2024-03-21T14:24:21.150920+0000","last_became_active":"2024-03-21T14:19:14.693616+0000","last_became_peered":"2024-03-21T14:19:14.693616+0000","last_unstale":"2024-03-21T14:24:21.150920+0000","last_undegraded":"2024-03-21T14:24:21.150920+0000","last_fullsized":"2024-03-21T14:24:21.150920+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'66","last_scrub_stamp":"2024-03-21T14:22:45.361551+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:22:45.361551+0000","objects_scrubbed":10,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528264+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":734,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148217+0000","last_change":"2024-03-21T14:21:32.597365+0000","last_active":"2024-03-21T14:24:21.148217+0000","last_peered":"2024-03-21T14:24:21.148217+0000","last_clean":"2024-03-21T14:24:21.148217+0000","last_became_active":"2024-03-21T14:19:34.690636+0000","last_became_peered":"2024-03-21T14:19:34.690636+0000","last_unstale":"2024-03-21T14:24:21.148217+0000","last_undegraded":"2024-03-21T14:24:21.148217+0000","last_fullsized":"2024-03-21T14:24:21.148217+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:32.597257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:32.597257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885159+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.8","version":"457'13876","reported_seq":13980,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:23.002034+0000","last_change":"2024-03-21T14:20:59.387006+0000","last_active":"2024-03-21T14:24:23.002034+0000","last_peered":"2024-03-21T14:24:23.002034+0000","last_clean":"2024-03-21T14:24:23.002034+0000","last_became_active":"2024-03-21T14:20:59.386643+0000","last_became_peered":"2024-03-21T14:20:59.386643+0000","last_unstale":"2024-03-21T14:24:23.002034+0000","last_undegraded":"2024-03-21T14:24:23.002034+0000","last_fullsized":"2024-03-21T14:24:23.002034+0000","mapping_epoch":411,"log_start":"446'9700","ondisk_log_start":"446'9700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.406270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13876,"num_write_kb":444032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":1081,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152858+0000","last_change":"2024-03-21T14:21:52.357400+0000","last_active":"2024-03-21T14:24:21.152858+0000","last_peered":"2024-03-21T14:24:21.152858+0000","last_clean":"2024-03-21T14:24:21.152858+0000","last_became_active":"2024-03-21T14:16:34.615428+0000","last_became_peered":"2024-03-21T14:16:34.615428+0000","last_unstale":"2024-03-21T14:24:21.152858+0000","last_undegraded":"2024-03-21T14:24:21.152858+0000","last_fullsized":"2024-03-21T14:24:21.152858+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:52.357256+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:52.357256+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":871,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156726+0000","last_change":"2024-03-21T14:23:03.702113+0000","last_active":"2024-03-21T14:24:21.156726+0000","last_peered":"2024-03-21T14:24:21.156726+0000","last_clean":"2024-03-21T14:24:21.156726+0000","last_became_active":"2024-03-21T14:19:14.685026+0000","last_became_peered":"2024-03-21T14:19:14.685026+0000","last_unstale":"2024-03-21T14:24:21.156726+0000","last_undegraded":"2024-03-21T14:24:21.156726+0000","last_fullsized":"2024-03-21T14:24:21.156726+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:03.702034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:23:03.702034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.563384+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":875,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146642+0000","last_change":"2024-03-21T14:22:45.828480+0000","last_active":"2024-03-21T14:24:21.146642+0000","last_peered":"2024-03-21T14:24:21.146642+0000","last_clean":"2024-03-21T14:24:21.146642+0000","last_became_active":"2024-03-21T14:19:14.697007+0000","last_became_peered":"2024-03-21T14:19:14.697007+0000","last_unstale":"2024-03-21T14:24:21.146642+0000","last_undegraded":"2024-03-21T14:24:21.146642+0000","last_fullsized":"2024-03-21T14:24:21.146642+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:45.828270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:04.811636+0000","last_clean_scrub_stamp":"2024-03-21T14:22:45.828270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885456+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":711,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.149943+0000","last_change":"2024-03-21T14:23:52.341897+0000","last_active":"2024-03-21T14:24:21.149943+0000","last_peered":"2024-03-21T14:24:21.149943+0000","last_clean":"2024-03-21T14:24:21.149943+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:24:21.149943+0000","last_undegraded":"2024-03-21T14:24:21.149943+0000","last_fullsized":"2024-03-21T14:24:21.149943+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:52.341783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:23:52.341783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:01.937881+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":960,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148900+0000","last_change":"2024-03-21T14:21:47.628148+0000","last_active":"2024-03-21T14:24:21.148900+0000","last_peered":"2024-03-21T14:24:21.148900+0000","last_clean":"2024-03-21T14:24:21.148900+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:24:21.148900+0000","last_undegraded":"2024-03-21T14:24:21.148900+0000","last_fullsized":"2024-03-21T14:24:21.148900+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'22","last_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_deep_scrub":"236'22","last_deep_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_clean_scrub_stamp":"2024-03-21T14:21:47.628002+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885133+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":686,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.165626+0000","last_change":"2024-03-21T14:21:42.301835+0000","last_active":"2024-03-21T14:24:21.165626+0000","last_peered":"2024-03-21T14:24:21.165626+0000","last_clean":"2024-03-21T14:24:21.165626+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:24:21.165626+0000","last_undegraded":"2024-03-21T14:24:21.165626+0000","last_fullsized":"2024-03-21T14:24:21.165626+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:42.301114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:02.070177+0000","last_clean_scrub_stamp":"2024-03-21T14:21:42.301114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528052+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"457'13840","reported_seq":13944,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.981101+0000","last_change":"2024-03-21T14:20:59.387064+0000","last_active":"2024-03-21T14:24:22.981101+0000","last_peered":"2024-03-21T14:24:22.981101+0000","last_clean":"2024-03-21T14:24:22.981101+0000","last_became_active":"2024-03-21T14:20:59.386748+0000","last_became_peered":"2024-03-21T14:20:59.386748+0000","last_unstale":"2024-03-21T14:24:22.981101+0000","last_undegraded":"2024-03-21T14:24:22.981101+0000","last_fullsized":"2024-03-21T14:24:22.981101+0000","mapping_epoch":411,"log_start":"447'9600","ondisk_log_start":"447'9600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.405580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13840,"num_write_kb":442880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":332,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154570+0000","last_change":"2024-03-21T14:21:23.819765+0000","last_active":"2024-03-21T14:24:21.154570+0000","last_peered":"2024-03-21T14:24:21.154570+0000","last_clean":"2024-03-21T14:24:21.154570+0000","last_became_active":"2024-03-21T14:16:21.865996+0000","last_became_peered":"2024-03-21T14:16:21.865996+0000","last_unstale":"2024-03-21T14:24:21.154570+0000","last_undegraded":"2024-03-21T14:24:21.154570+0000","last_fullsized":"2024-03-21T14:24:21.154570+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:23.819644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:21:23.819644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":138,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1190,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.156348+0000","last_change":"2024-03-21T14:23:50.807489+0000","last_active":"2024-03-21T14:24:21.156348+0000","last_peered":"2024-03-21T14:24:21.156348+0000","last_clean":"2024-03-21T14:24:21.156348+0000","last_became_active":"2024-03-21T14:19:37.727165+0000","last_became_peered":"2024-03-21T14:19:37.727165+0000","last_unstale":"2024-03-21T14:24:21.156348+0000","last_undegraded":"2024-03-21T14:24:21.156348+0000","last_fullsized":"2024-03-21T14:24:21.156348+0000","mapping_epoch":368,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":369,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:23:50.807371+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:23:50.807371+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:10.725966+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":882,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162916+0000","last_change":"2024-03-21T14:22:30.336811+0000","last_active":"2024-03-21T14:24:21.162916+0000","last_peered":"2024-03-21T14:24:21.162916+0000","last_clean":"2024-03-21T14:24:21.162916+0000","last_became_active":"2024-03-21T14:12:00.529151+0000","last_became_peered":"2024-03-21T14:12:00.529151+0000","last_unstale":"2024-03-21T14:24:21.162916+0000","last_undegraded":"2024-03-21T14:24:21.162916+0000","last_fullsized":"2024-03-21T14:24:21.162916+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:30.335992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:22:30.335992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"441'41","reported_seq":778,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154057+0000","last_change":"2024-03-21T14:22:58.255728+0000","last_active":"2024-03-21T14:24:21.154057+0000","last_peered":"2024-03-21T14:24:21.154057+0000","last_clean":"2024-03-21T14:24:21.154057+0000","last_became_active":"2024-03-21T14:12:00.545302+0000","last_became_peered":"2024-03-21T14:12:00.545302+0000","last_unstale":"2024-03-21T14:24:21.154057+0000","last_undegraded":"2024-03-21T14:24:21.154057+0000","last_fullsized":"2024-03-21T14:24:21.154057+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'41","last_scrub_stamp":"2024-03-21T14:22:58.255560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:58.255560+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":228,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.159805+0000","last_change":"2024-03-21T14:23:07.337422+0000","last_active":"2024-03-21T14:24:21.159805+0000","last_peered":"2024-03-21T14:24:21.159805+0000","last_clean":"2024-03-21T14:24:21.159805+0000","last_became_active":"2024-03-21T14:19:14.690145+0000","last_became_peered":"2024-03-21T14:19:14.690145+0000","last_unstale":"2024-03-21T14:24:21.159805+0000","last_undegraded":"2024-03-21T14:24:21.159805+0000","last_fullsized":"2024-03-21T14:24:21.159805+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:07.333488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:23:07.333488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528355+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"457'14050","reported_seq":14154,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:23.001862+0000","last_change":"2024-03-21T14:20:59.239970+0000","last_active":"2024-03-21T14:24:23.001862+0000","last_peered":"2024-03-21T14:24:23.001862+0000","last_clean":"2024-03-21T14:24:23.001862+0000","last_became_active":"2024-03-21T14:20:59.239659+0000","last_became_peered":"2024-03-21T14:20:59.239659+0000","last_unstale":"2024-03-21T14:24:23.001862+0000","last_undegraded":"2024-03-21T14:24:23.001862+0000","last_fullsized":"2024-03-21T14:24:23.001862+0000","mapping_epoch":411,"log_start":"447'9800","ondisk_log_start":"447'9800","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.405539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14050,"num_write_kb":449600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":937,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148036+0000","last_change":"2024-03-21T14:22:32.735774+0000","last_active":"2024-03-21T14:24:21.148036+0000","last_peered":"2024-03-21T14:24:21.148036+0000","last_clean":"2024-03-21T14:24:21.148036+0000","last_became_active":"2024-03-21T14:14:35.593708+0000","last_became_peered":"2024-03-21T14:14:35.593708+0000","last_unstale":"2024-03-21T14:24:21.148036+0000","last_undegraded":"2024-03-21T14:24:21.148036+0000","last_fullsized":"2024-03-21T14:24:21.148036+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:32.735646+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:32.735646+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885400+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":882,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.149991+0000","last_change":"2024-03-21T14:23:52.499737+0000","last_active":"2024-03-21T14:24:21.149991+0000","last_peered":"2024-03-21T14:24:21.149991+0000","last_clean":"2024-03-21T14:24:21.149991+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:24:21.149991+0000","last_undegraded":"2024-03-21T14:24:21.149991+0000","last_fullsized":"2024-03-21T14:24:21.149991+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:52.499618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:23:52.499618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:21.280420+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":666,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152651+0000","last_change":"2024-03-21T14:22:51.407734+0000","last_active":"2024-03-21T14:24:21.152651+0000","last_peered":"2024-03-21T14:24:21.152651+0000","last_clean":"2024-03-21T14:24:21.152651+0000","last_became_active":"2024-03-21T14:20:21.778326+0000","last_became_peered":"2024-03-21T14:20:21.778326+0000","last_unstale":"2024-03-21T14:24:21.152651+0000","last_undegraded":"2024-03-21T14:24:21.152651+0000","last_fullsized":"2024-03-21T14:24:21.152651+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:51.407608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.407608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":2091,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:25.964460+0000","last_change":"2024-03-21T14:21:41.487654+0000","last_active":"2024-03-21T14:24:25.964460+0000","last_peered":"2024-03-21T14:24:25.964460+0000","last_clean":"2024-03-21T14:24:25.964460+0000","last_became_active":"2024-03-21T14:16:34.618717+0000","last_became_peered":"2024-03-21T14:16:34.618717+0000","last_unstale":"2024-03-21T14:24:25.964460+0000","last_undegraded":"2024-03-21T14:24:25.964460+0000","last_fullsized":"2024-03-21T14:24:25.964460+0000","mapping_epoch":311,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":312,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:21:41.487571+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:41.487571+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:29.984321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1247,"num_read_kb":1242,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":828,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155582+0000","last_change":"2024-03-21T14:22:51.894009+0000","last_active":"2024-03-21T14:24:21.155582+0000","last_peered":"2024-03-21T14:24:21.155582+0000","last_clean":"2024-03-21T14:24:21.155582+0000","last_became_active":"2024-03-21T14:19:34.692301+0000","last_became_peered":"2024-03-21T14:19:34.692301+0000","last_unstale":"2024-03-21T14:24:21.155582+0000","last_undegraded":"2024-03-21T14:24:21.155582+0000","last_fullsized":"2024-03-21T14:24:21.155582+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_deep_scrub":"441'36","last_deep_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_clean_scrub_stamp":"2024-03-21T14:22:51.893561+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885430+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":334,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.149232+0000","last_change":"2024-03-21T14:21:40.525563+0000","last_active":"2024-03-21T14:24:21.149232+0000","last_peered":"2024-03-21T14:24:21.149232+0000","last_clean":"2024-03-21T14:24:21.149232+0000","last_became_active":"2024-03-21T14:16:21.868575+0000","last_became_peered":"2024-03-21T14:16:21.868575+0000","last_unstale":"2024-03-21T14:24:21.149232+0000","last_undegraded":"2024-03-21T14:24:21.149232+0000","last_fullsized":"2024-03-21T14:24:21.149232+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:40.525392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:21:40.525392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885069+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"457'13838","reported_seq":13973,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.990360+0000","last_change":"2024-03-21T14:23:51.543181+0000","last_active":"2024-03-21T14:24:22.990360+0000","last_peered":"2024-03-21T14:24:22.990360+0000","last_clean":"2024-03-21T14:24:22.990360+0000","last_became_active":"2024-03-21T14:20:59.109825+0000","last_became_peered":"2024-03-21T14:20:59.109825+0000","last_unstale":"2024-03-21T14:24:22.990360+0000","last_undegraded":"2024-03-21T14:24:22.990360+0000","last_fullsized":"2024-03-21T14:24:22.990360+0000","mapping_epoch":411,"log_start":"447'9800","ondisk_log_start":"447'9800","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"452'11774","last_scrub_stamp":"2024-03-21T14:23:51.542896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:23:51.542896+0000","objects_scrubbed":2103,"log_size":4038,"log_dups_size":0,"ondisk_log_size":4038,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:58.823214+0000","scrub_duration":1001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13838,"num_write_kb":442816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":665,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152542+0000","last_change":"2024-03-21T14:20:30.945601+0000","last_active":"2024-03-21T14:24:21.152542+0000","last_peered":"2024-03-21T14:24:21.152542+0000","last_clean":"2024-03-21T14:24:21.152542+0000","last_became_active":"2024-03-21T14:20:24.825637+0000","last_became_peered":"2024-03-21T14:20:24.825637+0000","last_unstale":"2024-03-21T14:24:21.152542+0000","last_undegraded":"2024-03-21T14:24:21.152542+0000","last_fullsized":"2024-03-21T14:24:21.152542+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:30.945471+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:30.945471+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":879,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.146373+0000","last_change":"2024-03-21T14:24:05.738637+0000","last_active":"2024-03-21T14:24:21.146373+0000","last_peered":"2024-03-21T14:24:21.146373+0000","last_clean":"2024-03-21T14:24:21.146373+0000","last_became_active":"2024-03-21T14:19:14.697998+0000","last_became_peered":"2024-03-21T14:19:14.697998+0000","last_unstale":"2024-03-21T14:24:21.146373+0000","last_undegraded":"2024-03-21T14:24:21.146373+0000","last_fullsized":"2024-03-21T14:24:21.146373+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:05.738523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:05.738523+0000","last_clean_scrub_stamp":"2024-03-21T14:24:05.738523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:28.118856+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":877,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.150246+0000","last_change":"2024-03-21T14:22:48.925469+0000","last_active":"2024-03-21T14:24:21.150246+0000","last_peered":"2024-03-21T14:24:21.150246+0000","last_clean":"2024-03-21T14:24:21.150246+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:24:21.150246+0000","last_undegraded":"2024-03-21T14:24:21.150246+0000","last_fullsized":"2024-03-21T14:24:21.150246+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:48.925350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:23.381813+0000","last_clean_scrub_stamp":"2024-03-21T14:22:48.925350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885612+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"441'39","reported_seq":621,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.140732+0000","last_change":"2024-03-21T14:23:57.307302+0000","last_active":"2024-03-21T14:24:21.140732+0000","last_peered":"2024-03-21T14:24:21.140732+0000","last_clean":"2024-03-21T14:24:21.140732+0000","last_became_active":"2024-03-21T14:12:20.561790+0000","last_became_peered":"2024-03-21T14:12:20.561790+0000","last_unstale":"2024-03-21T14:24:21.140732+0000","last_undegraded":"2024-03-21T14:24:21.140732+0000","last_fullsized":"2024-03-21T14:24:21.140732+0000","mapping_epoch":224,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":225,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-21T14:23:57.307155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:57.307155+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:27.213668+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":232,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.163050+0000","last_change":"2024-03-21T14:22:50.356522+0000","last_active":"2024-03-21T14:24:21.163050+0000","last_peered":"2024-03-21T14:24:21.163050+0000","last_clean":"2024-03-21T14:24:21.163050+0000","last_became_active":"2024-03-21T14:19:14.675538+0000","last_became_peered":"2024-03-21T14:19:14.675538+0000","last_unstale":"2024-03-21T14:24:21.163050+0000","last_undegraded":"2024-03-21T14:24:21.163050+0000","last_fullsized":"2024-03-21T14:24:21.163050+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:50.356413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:22:50.356413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528231+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"457'14036","reported_seq":17086,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.994857+0000","last_change":"2024-03-21T14:24:01.812968+0000","last_active":"2024-03-21T14:24:22.994857+0000","last_peered":"2024-03-21T14:24:22.994857+0000","last_clean":"2024-03-21T14:24:22.994857+0000","last_became_active":"2024-03-21T14:21:41.923763+0000","last_became_peered":"2024-03-21T14:21:41.923763+0000","last_unstale":"2024-03-21T14:24:22.994857+0000","last_undegraded":"2024-03-21T14:24:22.994857+0000","last_fullsized":"2024-03-21T14:24:22.994857+0000","mapping_epoch":431,"log_start":"445'8500","ondisk_log_start":"445'8500","created":411,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'12558","last_scrub_stamp":"2024-03-21T14:24:01.812585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:24:01.812585+0000","objects_scrubbed":1490,"log_size":5536,"log_dups_size":0,"ondisk_log_size":5536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:06.735853+0000","scrub_duration":410,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14036,"num_write_kb":449089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1456,"num_bytes_recovered":95420416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":847,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153172+0000","last_change":"2024-03-21T14:23:54.372231+0000","last_active":"2024-03-21T14:24:21.153172+0000","last_peered":"2024-03-21T14:24:21.153172+0000","last_clean":"2024-03-21T14:24:21.153172+0000","last_became_active":"2024-03-21T14:20:24.819844+0000","last_became_peered":"2024-03-21T14:20:24.819844+0000","last_unstale":"2024-03-21T14:24:21.153172+0000","last_undegraded":"2024-03-21T14:24:21.153172+0000","last_fullsized":"2024-03-21T14:24:21.153172+0000","mapping_epoch":394,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":395,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:23:54.372111+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:23:54.372111+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:21.437316+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":371,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161585+0000","last_change":"2024-03-21T14:22:56.247550+0000","last_active":"2024-03-21T14:24:21.161585+0000","last_peered":"2024-03-21T14:24:21.161585+0000","last_clean":"2024-03-21T14:24:21.161585+0000","last_became_active":"2024-03-21T14:19:15.688238+0000","last_became_peered":"2024-03-21T14:19:15.688238+0000","last_unstale":"2024-03-21T14:24:21.161585+0000","last_undegraded":"2024-03-21T14:24:21.161585+0000","last_fullsized":"2024-03-21T14:24:21.161585+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:56.247428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:56.247428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":799,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.160080+0000","last_change":"2024-03-21T14:20:32.884624+0000","last_active":"2024-03-21T14:24:21.160080+0000","last_peered":"2024-03-21T14:24:21.160080+0000","last_clean":"2024-03-21T14:24:21.160080+0000","last_became_active":"2024-03-21T14:19:14.695943+0000","last_became_peered":"2024-03-21T14:19:14.695943+0000","last_unstale":"2024-03-21T14:24:21.160080+0000","last_undegraded":"2024-03-21T14:24:21.160080+0000","last_fullsized":"2024-03-21T14:24:21.160080+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:32.884514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:20:32.884514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":922,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.148395+0000","last_change":"2024-03-21T14:21:43.496288+0000","last_active":"2024-03-21T14:24:21.148395+0000","last_peered":"2024-03-21T14:24:21.148395+0000","last_clean":"2024-03-21T14:24:21.148395+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:24:21.148395+0000","last_undegraded":"2024-03-21T14:24:21.148395+0000","last_fullsized":"2024-03-21T14:24:21.148395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"236'25","last_scrub_stamp":"2024-03-21T14:21:43.496137+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:21:43.496137+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885103+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"307'11","reported_seq":887,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154792+0000","last_change":"2024-03-21T14:22:55.163828+0000","last_active":"2024-03-21T14:24:21.154792+0000","last_peered":"2024-03-21T14:24:21.154792+0000","last_clean":"2024-03-21T14:24:21.154792+0000","last_became_active":"2024-03-21T14:16:21.860849+0000","last_became_peered":"2024-03-21T14:16:21.860849+0000","last_unstale":"2024-03-21T14:24:21.154792+0000","last_undegraded":"2024-03-21T14:24:21.154792+0000","last_fullsized":"2024-03-21T14:24:21.154792+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"307'11","last_scrub_stamp":"2024-03-21T14:22:55.163691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:55.163691+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.409230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"457'13992","reported_seq":14096,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.990585+0000","last_change":"2024-03-21T14:20:59.110795+0000","last_active":"2024-03-21T14:24:22.990585+0000","last_peered":"2024-03-21T14:24:22.990585+0000","last_clean":"2024-03-21T14:24:22.990585+0000","last_became_active":"2024-03-21T14:20:59.110468+0000","last_became_peered":"2024-03-21T14:20:59.110468+0000","last_unstale":"2024-03-21T14:24:22.990585+0000","last_undegraded":"2024-03-21T14:24:22.990585+0000","last_fullsized":"2024-03-21T14:24:22.990585+0000","mapping_epoch":411,"log_start":"446'9600","ondisk_log_start":"446'9600","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4392,"log_dups_size":0,"ondisk_log_size":4392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13992,"num_write_kb":447744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":921,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152218+0000","last_change":"2024-03-21T14:20:52.764101+0000","last_active":"2024-03-21T14:24:21.152218+0000","last_peered":"2024-03-21T14:24:21.152218+0000","last_clean":"2024-03-21T14:24:21.152218+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:24:21.152218+0000","last_undegraded":"2024-03-21T14:24:21.152218+0000","last_fullsized":"2024-03-21T14:24:21.152218+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:20:52.763988+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:20:52.763988+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":871,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157384+0000","last_change":"2024-03-21T14:22:49.915554+0000","last_active":"2024-03-21T14:24:21.157384+0000","last_peered":"2024-03-21T14:24:21.157384+0000","last_clean":"2024-03-21T14:24:21.157384+0000","last_became_active":"2024-03-21T14:19:14.684876+0000","last_became_peered":"2024-03-21T14:19:14.684876+0000","last_unstale":"2024-03-21T14:24:21.157384+0000","last_undegraded":"2024-03-21T14:24:21.157384+0000","last_fullsized":"2024-03-21T14:24:21.157384+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:49.915445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:19:27.609702+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.915445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.901693+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":743,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155443+0000","last_change":"2024-03-21T14:23:55.482924+0000","last_active":"2024-03-21T14:24:21.155443+0000","last_peered":"2024-03-21T14:24:21.155443+0000","last_clean":"2024-03-21T14:24:21.155443+0000","last_became_active":"2024-03-21T14:19:34.692258+0000","last_became_peered":"2024-03-21T14:19:34.692258+0000","last_unstale":"2024-03-21T14:24:21.155443+0000","last_undegraded":"2024-03-21T14:24:21.155443+0000","last_fullsized":"2024-03-21T14:24:21.155443+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:23:55.482795+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:23:55.482795+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:56.035387+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"441'42","reported_seq":962,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152555+0000","last_change":"2024-03-21T14:22:49.399821+0000","last_active":"2024-03-21T14:24:21.152555+0000","last_peered":"2024-03-21T14:24:21.152555+0000","last_clean":"2024-03-21T14:24:21.152555+0000","last_became_active":"2024-03-21T14:21:26.768622+0000","last_became_peered":"2024-03-21T14:21:26.768622+0000","last_unstale":"2024-03-21T14:24:21.152555+0000","last_undegraded":"2024-03-21T14:24:21.152555+0000","last_fullsized":"2024-03-21T14:24:21.152555+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-21T14:22:49.399707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:22:49.399707+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":1131,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162146+0000","last_change":"2024-03-21T14:22:38.443967+0000","last_active":"2024-03-21T14:24:21.162146+0000","last_peered":"2024-03-21T14:24:21.162146+0000","last_clean":"2024-03-21T14:24:21.162146+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:24:21.162146+0000","last_undegraded":"2024-03-21T14:24:21.162146+0000","last_fullsized":"2024-03-21T14:24:21.162146+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:22:38.443825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:38.443825+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.409054+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"457'13932","reported_seq":14822,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.998933+0000","last_change":"2024-03-21T14:21:28.769176+0000","last_active":"2024-03-21T14:24:22.998933+0000","last_peered":"2024-03-21T14:24:22.998933+0000","last_clean":"2024-03-21T14:24:22.998933+0000","last_became_active":"2024-03-21T14:21:28.766274+0000","last_became_peered":"2024-03-21T14:21:28.766274+0000","last_unstale":"2024-03-21T14:24:22.998933+0000","last_undegraded":"2024-03-21T14:24:22.998933+0000","last_fullsized":"2024-03-21T14:24:22.998933+0000","mapping_epoch":425,"log_start":"446'9500","ondisk_log_start":"446'9500","created":411,"last_epoch_clean":426,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4432,"log_dups_size":0,"ondisk_log_size":4432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13932,"num_write_kb":445824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":59375616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":923,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153771+0000","last_change":"2024-03-21T14:21:19.302873+0000","last_active":"2024-03-21T14:24:21.153771+0000","last_peered":"2024-03-21T14:24:21.153771+0000","last_clean":"2024-03-21T14:24:21.153771+0000","last_became_active":"2024-03-21T14:19:14.682251+0000","last_became_peered":"2024-03-21T14:19:14.682251+0000","last_unstale":"2024-03-21T14:24:21.153771+0000","last_undegraded":"2024-03-21T14:24:21.153771+0000","last_fullsized":"2024-03-21T14:24:21.153771+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:21:19.302784+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:21:19.302784+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":871,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153176+0000","last_change":"2024-03-21T14:22:35.628602+0000","last_active":"2024-03-21T14:24:21.153176+0000","last_peered":"2024-03-21T14:24:21.153176+0000","last_clean":"2024-03-21T14:24:21.153176+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:24:21.153176+0000","last_undegraded":"2024-03-21T14:24:21.153176+0000","last_fullsized":"2024-03-21T14:24:21.153176+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:35.628491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:22:35.628491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":868,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154385+0000","last_change":"2024-03-21T14:21:39.060809+0000","last_active":"2024-03-21T14:24:21.154385+0000","last_peered":"2024-03-21T14:24:21.154385+0000","last_clean":"2024-03-21T14:24:21.154385+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:24:21.154385+0000","last_undegraded":"2024-03-21T14:24:21.154385+0000","last_fullsized":"2024-03-21T14:24:21.154385+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:39.060697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:21:39.060697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'50","reported_seq":187,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161104+0000","last_change":"2024-03-21T14:23:04.313764+0000","last_active":"2024-03-21T14:24:21.161104+0000","last_peered":"2024-03-21T14:24:21.161104+0000","last_clean":"2024-03-21T14:24:21.161104+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:24:21.161104+0000","last_undegraded":"2024-03-21T14:24:21.161104+0000","last_fullsized":"2024-03-21T14:24:21.161104+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'50","last_scrub_stamp":"2024-03-21T14:23:04.313411+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.313411+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.409200+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":843,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152225+0000","last_change":"2024-03-21T14:22:43.461461+0000","last_active":"2024-03-21T14:24:21.152225+0000","last_peered":"2024-03-21T14:24:21.152225+0000","last_clean":"2024-03-21T14:24:21.152225+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:24:21.152225+0000","last_undegraded":"2024-03-21T14:24:21.152225+0000","last_fullsized":"2024-03-21T14:24:21.152225+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:43.461338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:43.461338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"457'13746","reported_seq":13920,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.990229+0000","last_change":"2024-03-21T14:22:33.276757+0000","last_active":"2024-03-21T14:24:22.990229+0000","last_peered":"2024-03-21T14:24:22.990229+0000","last_clean":"2024-03-21T14:24:22.990229+0000","last_became_active":"2024-03-21T14:20:59.109191+0000","last_became_peered":"2024-03-21T14:20:59.109191+0000","last_unstale":"2024-03-21T14:24:22.990229+0000","last_undegraded":"2024-03-21T14:24:22.990229+0000","last_fullsized":"2024-03-21T14:24:22.990229+0000","mapping_epoch":411,"log_start":"447'9700","ondisk_log_start":"447'9700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"439'6212","last_scrub_stamp":"2024-03-21T14:22:33.276373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:33.276373+0000","objects_scrubbed":6148,"log_size":4046,"log_dups_size":0,"ondisk_log_size":4046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885370+0000","scrub_duration":2601,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13746,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":844,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.163262+0000","last_change":"2024-03-21T14:23:48.300168+0000","last_active":"2024-03-21T14:24:21.163262+0000","last_peered":"2024-03-21T14:24:21.163262+0000","last_clean":"2024-03-21T14:24:21.163262+0000","last_became_active":"2024-03-21T14:19:14.692919+0000","last_became_peered":"2024-03-21T14:19:14.692919+0000","last_unstale":"2024-03-21T14:24:21.163262+0000","last_undegraded":"2024-03-21T14:24:21.163262+0000","last_fullsized":"2024-03-21T14:24:21.163262+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:23:48.300043+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:23:48.300043+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:08.278032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":884,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.147556+0000","last_change":"2024-03-21T14:23:51.724637+0000","last_active":"2024-03-21T14:24:21.147556+0000","last_peered":"2024-03-21T14:24:21.147556+0000","last_clean":"2024-03-21T14:24:21.147556+0000","last_became_active":"2024-03-21T14:19:34.691121+0000","last_became_peered":"2024-03-21T14:19:34.691121+0000","last_unstale":"2024-03-21T14:24:21.147556+0000","last_undegraded":"2024-03-21T14:24:21.147556+0000","last_fullsized":"2024-03-21T14:24:21.147556+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:23:51.724512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:23:51.724512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:56.900973+0000","scrub_duration":230,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1121,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154593+0000","last_change":"2024-03-21T14:22:40.498837+0000","last_active":"2024-03-21T14:24:21.154593+0000","last_peered":"2024-03-21T14:24:21.154593+0000","last_clean":"2024-03-21T14:24:21.154593+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:24:21.154593+0000","last_undegraded":"2024-03-21T14:24:21.154593+0000","last_fullsized":"2024-03-21T14:24:21.154593+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:22:40.498694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:22:40.498694+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":150,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.161497+0000","last_change":"2024-03-21T14:21:54.066071+0000","last_active":"2024-03-21T14:24:21.161497+0000","last_peered":"2024-03-21T14:24:21.161497+0000","last_clean":"2024-03-21T14:24:21.161497+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:24:21.161497+0000","last_undegraded":"2024-03-21T14:24:21.161497+0000","last_fullsized":"2024-03-21T14:24:21.161497+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.065922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.065922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'39","reported_seq":954,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.149049+0000","last_change":"2024-03-21T14:23:48.535028+0000","last_active":"2024-03-21T14:24:21.149049+0000","last_peered":"2024-03-21T14:24:21.149049+0000","last_clean":"2024-03-21T14:24:21.149049+0000","last_became_active":"2024-03-21T14:21:19.713483+0000","last_became_peered":"2024-03-21T14:21:19.713483+0000","last_unstale":"2024-03-21T14:24:21.149049+0000","last_undegraded":"2024-03-21T14:24:21.149049+0000","last_fullsized":"2024-03-21T14:24:21.149049+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-21T14:23:48.534888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:48.534888+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:53.807572+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"457'13956","reported_seq":14060,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.995160+0000","last_change":"2024-03-21T14:20:59.240921+0000","last_active":"2024-03-21T14:24:22.995160+0000","last_peered":"2024-03-21T14:24:22.995160+0000","last_clean":"2024-03-21T14:24:22.995160+0000","last_became_active":"2024-03-21T14:20:59.240641+0000","last_became_peered":"2024-03-21T14:20:59.240641+0000","last_unstale":"2024-03-21T14:24:22.995160+0000","last_undegraded":"2024-03-21T14:24:22.995160+0000","last_fullsized":"2024-03-21T14:24:22.995160+0000","mapping_epoch":411,"log_start":"447'9700","ondisk_log_start":"447'9700","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:20:58.082826+0000","objects_scrubbed":0,"log_size":4256,"log_dups_size":0,"ondisk_log_size":4256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:26.408679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13956,"num_write_kb":446592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":861,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154693+0000","last_change":"2024-03-21T14:22:39.460174+0000","last_active":"2024-03-21T14:24:21.154693+0000","last_peered":"2024-03-21T14:24:21.154693+0000","last_clean":"2024-03-21T14:24:21.154693+0000","last_became_active":"2024-03-21T14:12:00.529051+0000","last_became_peered":"2024-03-21T14:12:00.529051+0000","last_unstale":"2024-03-21T14:24:21.154693+0000","last_undegraded":"2024-03-21T14:24:21.154693+0000","last_fullsized":"2024-03-21T14:24:21.154693+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:39.460024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:22:39.460024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":851,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154466+0000","last_change":"2024-03-21T14:21:51.322736+0000","last_active":"2024-03-21T14:24:21.154466+0000","last_peered":"2024-03-21T14:24:21.154466+0000","last_clean":"2024-03-21T14:24:21.154466+0000","last_became_active":"2024-03-21T14:21:26.767687+0000","last_became_peered":"2024-03-21T14:21:26.767687+0000","last_unstale":"2024-03-21T14:24:21.154466+0000","last_undegraded":"2024-03-21T14:24:21.154466+0000","last_fullsized":"2024-03-21T14:24:21.154466+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:51.322658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:21:51.322658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":868,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.152363+0000","last_change":"2024-03-21T14:22:31.537067+0000","last_active":"2024-03-21T14:24:21.152363+0000","last_peered":"2024-03-21T14:24:21.152363+0000","last_clean":"2024-03-21T14:24:21.152363+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:24:21.152363+0000","last_undegraded":"2024-03-21T14:24:21.152363+0000","last_fullsized":"2024-03-21T14:24:21.152363+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:31.536957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:31.536957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'27","reported_seq":915,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154054+0000","last_change":"2024-03-21T14:23:49.619677+0000","last_active":"2024-03-21T14:24:21.154054+0000","last_peered":"2024-03-21T14:24:21.154054+0000","last_clean":"2024-03-21T14:24:21.154054+0000","last_became_active":"2024-03-21T14:12:00.532784+0000","last_became_peered":"2024-03-21T14:12:00.532784+0000","last_unstale":"2024-03-21T14:24:21.154054+0000","last_undegraded":"2024-03-21T14:24:21.154054+0000","last_fullsized":"2024-03-21T14:24:21.154054+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-21T14:23:49.619535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:23:49.619535+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:09.544054+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"457'13700","reported_seq":13837,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.970195+0000","last_change":"2024-03-21T14:23:48.409332+0000","last_active":"2024-03-21T14:24:22.970195+0000","last_peered":"2024-03-21T14:24:22.970195+0000","last_clean":"2024-03-21T14:24:22.970195+0000","last_became_active":"2024-03-21T14:20:59.239396+0000","last_became_peered":"2024-03-21T14:20:59.239396+0000","last_unstale":"2024-03-21T14:24:22.970195+0000","last_undegraded":"2024-03-21T14:24:22.970195+0000","last_fullsized":"2024-03-21T14:24:22.970195+0000","mapping_epoch":411,"log_start":"445'8200","ondisk_log_start":"445'8200","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"451'11403","last_scrub_stamp":"2024-03-21T14:23:48.409068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:23:48.409068+0000","objects_scrubbed":2339,"log_size":5500,"log_dups_size":0,"ondisk_log_size":5500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:48.441234+0000","scrub_duration":1155,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13700,"num_write_kb":438400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":844,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.153998+0000","last_change":"2024-03-21T14:22:57.334699+0000","last_active":"2024-03-21T14:24:21.153998+0000","last_peered":"2024-03-21T14:24:21.153998+0000","last_clean":"2024-03-21T14:24:21.153998+0000","last_became_active":"2024-03-21T14:19:14.700353+0000","last_became_peered":"2024-03-21T14:19:14.700353+0000","last_unstale":"2024-03-21T14:24:21.153998+0000","last_undegraded":"2024-03-21T14:24:21.153998+0000","last_fullsized":"2024-03-21T14:24:21.153998+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:22:57.334563+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:22:57.334563+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.721763+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":663,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162273+0000","last_change":"2024-03-21T14:23:51.524801+0000","last_active":"2024-03-21T14:24:21.162273+0000","last_peered":"2024-03-21T14:24:21.162273+0000","last_clean":"2024-03-21T14:24:21.162273+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:24:21.162273+0000","last_undegraded":"2024-03-21T14:24:21.162273+0000","last_fullsized":"2024-03-21T14:24:21.162273+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:23:51.524562+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:23:51.524562+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:10.542182+0000","scrub_duration":183,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":867,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.150396+0000","last_change":"2024-03-21T14:21:54.619575+0000","last_active":"2024-03-21T14:24:21.150396+0000","last_peered":"2024-03-21T14:24:21.150396+0000","last_clean":"2024-03-21T14:24:21.150396+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:24:21.150396+0000","last_undegraded":"2024-03-21T14:24:21.150396+0000","last_fullsized":"2024-03-21T14:24:21.150396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_clean_scrub_stamp":"2024-03-21T14:21:54.617817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:24.885185+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":154,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.162968+0000","last_change":"2024-03-21T14:22:41.364253+0000","last_active":"2024-03-21T14:24:21.162968+0000","last_peered":"2024-03-21T14:24:21.162968+0000","last_clean":"2024-03-21T14:24:21.162968+0000","last_became_active":"2024-03-21T14:20:19.703587+0000","last_became_peered":"2024-03-21T14:20:19.703587+0000","last_unstale":"2024-03-21T14:24:21.162968+0000","last_undegraded":"2024-03-21T14:24:21.162968+0000","last_fullsized":"2024-03-21T14:24:21.162968+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:22:41.363995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:22:41.363995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:25.528202+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":917,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154142+0000","last_change":"2024-03-21T14:20:46.480561+0000","last_active":"2024-03-21T14:24:21.154142+0000","last_peered":"2024-03-21T14:24:21.154142+0000","last_clean":"2024-03-21T14:24:21.154142+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:24:21.154142+0000","last_undegraded":"2024-03-21T14:24:21.154142+0000","last_fullsized":"2024-03-21T14:24:21.154142+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-03-21T14:20:46.480457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:20:46.480457+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"457'13876","reported_seq":14046,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:22.975732+0000","last_change":"2024-03-21T14:22:59.010537+0000","last_active":"2024-03-21T14:24:22.975732+0000","last_peered":"2024-03-21T14:24:22.975732+0000","last_clean":"2024-03-21T14:24:22.975732+0000","last_became_active":"2024-03-21T14:20:59.386715+0000","last_became_peered":"2024-03-21T14:20:59.386715+0000","last_unstale":"2024-03-21T14:24:22.975732+0000","last_undegraded":"2024-03-21T14:24:22.975732+0000","last_fullsized":"2024-03-21T14:24:22.975732+0000","mapping_epoch":411,"log_start":"446'9316","ondisk_log_start":"446'9316","created":411,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'8088","last_scrub_stamp":"2024-03-21T14:22:59.010282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:20:58.082826+0000","last_clean_scrub_stamp":"2024-03-21T14:22:59.010282+0000","objects_scrubbed":5843,"log_size":4560,"log_dups_size":0,"ondisk_log_size":4560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:27.993189+0000","scrub_duration":1416,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13876,"num_write_kb":444032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1188,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155178+0000","last_change":"2024-03-21T14:23:04.602240+0000","last_active":"2024-03-21T14:24:21.155178+0000","last_peered":"2024-03-21T14:24:21.155178+0000","last_clean":"2024-03-21T14:24:21.155178+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:24:21.155178+0000","last_undegraded":"2024-03-21T14:24:21.155178+0000","last_fullsized":"2024-03-21T14:24:21.155178+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:23:04.602105+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:23:04.602105+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:32.873082+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":871,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.154244+0000","last_change":"2024-03-21T14:24:03.838361+0000","last_active":"2024-03-21T14:24:21.154244+0000","last_peered":"2024-03-21T14:24:21.154244+0000","last_clean":"2024-03-21T14:24:21.154244+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:24:21.154244+0000","last_undegraded":"2024-03-21T14:24:21.154244+0000","last_fullsized":"2024-03-21T14:24:21.154244+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:24:03.838256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:24:03.838256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:16.475809+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":895,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.160386+0000","last_change":"2024-03-21T14:23:53.300337+0000","last_active":"2024-03-21T14:24:21.160386+0000","last_peered":"2024-03-21T14:24:21.160386+0000","last_clean":"2024-03-21T14:24:21.160386+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:24:21.160386+0000","last_undegraded":"2024-03-21T14:24:21.160386+0000","last_fullsized":"2024-03-21T14:24:21.160386+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:23:53.300208+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:20:49.535652+0000","last_clean_scrub_stamp":"2024-03-21T14:23:53.300208+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:25:22.284511+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":879,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.155021+0000","last_change":"2024-03-21T14:21:46.110127+0000","last_active":"2024-03-21T14:24:21.155021+0000","last_peered":"2024-03-21T14:24:21.155021+0000","last_clean":"2024-03-21T14:24:21.155021+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:24:21.155021+0000","last_undegraded":"2024-03-21T14:24:21.155021+0000","last_fullsized":"2024-03-21T14:24:21.155021+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:21:46.110008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:21:46.110008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":922,"reported_epoch":457,"state":"active+clean","last_fresh":"2024-03-21T14:24:21.157040+0000","last_change":"2024-03-21T14:23:02.231025+0000","last_active":"2024-03-21T14:24:21.157040+0000","last_peered":"2024-03-21T14:24:21.157040+0000","last_clean":"2024-03-21T14:24:21.157040+0000","last_became_active":"2024-03-21T14:12:00.537029+0000","last_became_peered":"2024-03-21T14:12:00.537029+0000","last_unstale":"2024-03-21T14:24:21.157040+0000","last_undegraded":"2024-03-21T14:24:21.157040+0000","last_fullsized":"2024-03-21T14:24:21.157040+0000","mapping_epoch":210,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":211,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:23:02.230890+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:23:02.230890+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:24:21.516151+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":221760,"num_write_kb":7096257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3767,"num_bytes_recovered":246874112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71874,"ondisk_log_size":71874,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1247,"num_read_kb":1242,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1542,"num_read_kb":1299,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":176,"num_bytes_recovered":3812,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1057,"ondisk_log_size":1057,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103435,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898636,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846516,"statfs":{"total":95995035648,"available":95074832384,"internally_reserved":0,"allocated":1056768,"data_stored":703385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":351,"seq":1507533520962,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83732,"kb_used_data":652,"kb_used_omap":4,"kb_used_meta":83067,"kb_avail":93661420,"statfs":{"total":95995035648,"available":95909294080,"internally_reserved":0,"allocated":667648,"data_stored":372799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":85061284},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575147,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916264,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828888,"statfs":{"total":95995035648,"available":95056781312,"internally_reserved":0,"allocated":753664,"data_stored":376296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738672,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876124,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92869028,"statfs":{"total":95995035648,"available":95097884672,"internally_reserved":0,"allocated":745472,"data_stored":374929,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598292031,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":394892,"kb_used_data":13700,"kb_used_omap":7,"kb_used_meta":380984,"kb_avail":93350260,"statfs":{"total":95995035648,"available":95590666240,"internally_reserved":0,"allocated":14028800,"data_stored":13678576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":390127748},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182687,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822444,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92922708,"statfs":{"total":95995035648,"available":95152852992,"internally_reserved":0,"allocated":1347584,"data_stored":965560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":840825287},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019163,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":920416,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92824736,"statfs":{"total":95995035648,"available":95052529664,"internally_reserved":0,"allocated":745472,"data_stored":374906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692118,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":902940,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":92842212,"statfs":{"total":95995035648,"available":95070425088,"internally_reserved":0,"allocated":1331200,"data_stored":965925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:24:29.282 INFO:tasks.thrashosds.thrasher:Setting 6.10 to [0, 4, 3] 2024-03-21T14:24:29.282 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.10', '0', '4', '3'] 2024-03-21T14:24:29.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 6.10 0 4 3 2024-03-21T14:24:29.322 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:29 smithi118 ceph-mon[33862]: from='client.25967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:24:29.349 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-21T14:24:29.369 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:29.416 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:29.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:29.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:29 smithi003 ceph-mon[29576]: from='client.25967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:24:29.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:29 smithi003 ceph-mon[40510]: from='client.25967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:24:29.518 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:29.546 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:29.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:29.593 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:29.641 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-21T14:24:29.648 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:29.707 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:29.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:29.781 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:29.809 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:29.833 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:29.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:29.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:29.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:29.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:30.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:30.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.086 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:30.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:30.187 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:30.211 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:30.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:30.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:30.344 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:30.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:30.470 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:30.571 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:30.595 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:30.697 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:30.721 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:30.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:30.847 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:30.949 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:30.973 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:30.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:31.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:30 smithi003 ceph-mon[29576]: pgmap v1026: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 584 op/s 2024-03-21T14:24:31.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:30 smithi003 ceph-mon[40510]: pgmap v1026: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 584 op/s 2024-03-21T14:24:31.029 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:30 smithi118 ceph-mon[33862]: pgmap v1026: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 584 op/s 2024-03-21T14:24:31.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:31.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:31.224 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:31.307 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:31.308 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:31.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:31.350 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:31.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:31.451 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:31.475 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:31.576 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:31.610 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:31.632 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:31.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:31.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.737 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:31.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:31.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.862 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:31.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:31.901 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:24:31.963 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:31.983 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-21T14:24:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:31.989 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:31.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:32.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:31 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2548596824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:32.007 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:31 smithi003 ceph-mon[29576]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:32.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:31 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2548596824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:32.007 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:31 smithi003 ceph-mon[40510]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:32.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:32.114 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:31 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2548596824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:31 smithi118 ceph-mon[33862]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-21T14:24:32.216 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:32.226 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.b/config 2024-03-21T14:24:32.239 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:32.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:32.364 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:32.466 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:32.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:32.616 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:32.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:32.718 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:32.741 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:32.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:32.866 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2548596824' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:32 smithi118 ceph-mon[33862]: osdmap e458: 8 total, 8 up, 8 in 2024-03-21T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:32 smithi118 ceph-mon[33862]: 5.18 scrub starts 2024-03-21T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:32 smithi118 ceph-mon[33862]: 5.18 scrub ok 2024-03-21T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:32 smithi118 ceph-mon[33862]: pgmap v1028: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 508 op/s 2024-03-21T14:24:32.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:32.969 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:32.992 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:32.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:32.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:33.063 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2548596824' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[29576]: osdmap e458: 8 total, 8 up, 8 in 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[29576]: 5.18 scrub starts 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[29576]: 5.18 scrub ok 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[29576]: pgmap v1028: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 508 op/s 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2548596824' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[40510]: osdmap e458: 8 total, 8 up, 8 in 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[40510]: 5.18 scrub starts 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[40510]: 5.18 scrub ok 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[40510]: pgmap v1028: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 508 op/s 2024-03-21T14:24:33.064 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:33.094 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:33.118 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:33.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:33.220 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:33.244 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:33.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:33.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:33.370 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:33.472 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:33.496 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:33.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:33.597 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.623 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:33.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:33.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.750 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:33.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:33.851 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:33.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:33.876 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:33.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:33.908 INFO:teuthology.orchestra.run.smithi118.stdout:ERROR: (22) Invalid argument 2024-03-21T14:24:33.908 INFO:teuthology.orchestra.run.smithi118.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-21T14:24:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:33.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.000 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:34.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:34.101 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.128 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:34.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:33 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:24:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:33 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:24:34.229 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:34.252 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:34.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:33 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:24:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:33 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:24:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:33 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:24:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:33 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:24:34.354 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.378 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:34.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:34.479 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:34.503 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:34.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:34.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-21T14:24:34.605 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:34.628 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:34.730 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:34.754 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:34.856 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:34.880 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:34.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:34.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.007 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:35.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:35.108 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:35.131 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:35.153 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:34 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:24:35.153 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:34 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:24:35.153 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:34 smithi003 ceph-mon[29576]: pgmap v1029: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 35 op/s 2024-03-21T14:24:35.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:34 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:24:35.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:34 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:24:35.154 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:34 smithi003 ceph-mon[40510]: pgmap v1029: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 35 op/s 2024-03-21T14:24:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:34 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:24:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:34 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:24:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:34 smithi118 ceph-mon[33862]: pgmap v1029: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 35 op/s 2024-03-21T14:24:35.233 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:35.257 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:35.358 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:35.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.383 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:35.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:35.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:35.507 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:35.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:35.609 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:35.633 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:35.734 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:35.759 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:35.861 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:35.884 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:35.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:set 6.10 pg_upmap mapping to [0,4,3] 2024-03-21T14:24:35.986 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:36.010 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:36.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:36.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:36.135 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:36.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:35 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:24:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/517798227' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.10", "id": [0, 4, 3]}]: dispatch 2024-03-21T14:24:36.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:35 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:24:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/517798227' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.10", "id": [0, 4, 3]}]: dispatch 2024-03-21T14:24:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:35 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:24:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/517798227' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.10", "id": [0, 4, 3]}]: dispatch 2024-03-21T14:24:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:36.363 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:36.388 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:36.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:36.489 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:36.513 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:36.615 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:36.640 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:36.742 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:36.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:24:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:24:36.766 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:36.868 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:36.893 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:36.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:36.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:37.018 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:37.094 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/517798227' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.10", "id": [0, 4, 3]}]': finished 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[29576]: osdmap e459: 8 total, 8 up, 8 in 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[29576]: pgmap v1031: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/517798227' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.10", "id": [0, 4, 3]}]': finished 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[40510]: osdmap e459: 8 total, 8 up, 8 in 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[40510]: pgmap v1031: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:24:37.095 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:36 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:24:37.120 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:37.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.145 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:37.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:36 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:24:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:36 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/517798227' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.10", "id": [0, 4, 3]}]': finished 2024-03-21T14:24:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:36 smithi118 ceph-mon[33862]: osdmap e459: 8 total, 8 up, 8 in 2024-03-21T14:24:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:36 smithi118 ceph-mon[33862]: pgmap v1031: 158 pgs: 158 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:24:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:36 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:24:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:36 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:24:37.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:37.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.273 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:37.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:37.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:37.399 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:37.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:37.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:37.528 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:37.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:37.653 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:37.755 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:37.780 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:37.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:37.882 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:37.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:37.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:24:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:24:37.947 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'unique_pool_4' removed 2024-03-21T14:24:38.007 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:38.032 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:38.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:38.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.159 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:38.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:37 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:37 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:37 smithi118 ceph-mon[33862]: osdmap e460: 8 total, 8 up, 8 in 2024-03-21T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1790963406' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:24:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:37 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:24:38.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:24:38.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:24:38.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[29576]: osdmap e460: 8 total, 8 up, 8 in 2024-03-21T14:24:38.190 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1790963406' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:24:38.191 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:24:38.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:24:38.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:24:38.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[40510]: osdmap e460: 8 total, 8 up, 8 in 2024-03-21T14:24:38.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1790963406' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:24:38.191 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:37 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-21T14:24:38.259 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0 2024-03-21T14:24:38.283 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.0' 2024-03-21T14:24:38.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:24:38.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:38.408 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:38.510 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:38.534 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:38.636 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:38.659 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:38.760 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:38.787 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7: {} 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-21T14:24:38.831 INFO:teuthology.orchestra.run.smithi003.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-21T14:24:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-21T14:24:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-21T14:24:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-21T14:24:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-21T14:24:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-21T14:24:38.832 INFO:teuthology.orchestra.run.smithi003.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-21T14:24:38.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:38.912 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:38.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:39.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3 2024-03-21T14:24:39.030 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:39.040 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:39.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.3' 2024-03-21T14:24:39.142 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1 2024-03-21T14:24:39.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: 4.8 scrub starts 2024-03-21T14:24:39.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: 4.8 scrub ok 2024-03-21T14:24:39.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:39.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: osdmap e461: 8 total, 8 up, 8 in 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4248880382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[29576]: pgmap v1034: 142 pgs: 1 activating, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: 4.8 scrub starts 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: 4.8 scrub ok 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:39.144 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:24:39.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:24:39.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:24:39.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: osdmap e461: 8 total, 8 up, 8 in 2024-03-21T14:24:39.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4248880382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:39.145 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:38 smithi003 ceph-mon[40510]: pgmap v1034: 142 pgs: 1 activating, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:24:39.166 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.1' 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: 4.8 scrub starts 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: 4.8 scrub ok 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: osdmap e461: 8 total, 8 up, 8 in 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4248880382' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-21T14:24:39.221 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:38 smithi118 ceph-mon[33862]: pgmap v1034: 142 pgs: 1 activating, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-21T14:24:39.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2 2024-03-21T14:24:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.292 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: docker: command not found 2024-03-21T14:24:39.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi003 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.2' 2024-03-21T14:24:39.393 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4 2024-03-21T14:24:39.417 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.4' 2024-03-21T14:24:39.519 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5 2024-03-21T14:24:39.543 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.5' 2024-03-21T14:24:39.644 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6 2024-03-21T14:24:39.667 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.6' 2024-03-21T14:24:39.769 DEBUG:teuthology.orchestra.run.smithi118:> sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7 2024-03-21T14:24:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:24:39.793 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: docker: command not found 2024-03-21T14:24:39.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi118 with status 1: 'sudo docker kill -s 1 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd.7' 2024-03-21T14:24:39.832 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-21T14:24:39.845 INFO:tasks.thrashosds:joining thrashosds 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 1.6 scrub starts 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 1.6 scrub ok 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 6.17 scrub starts 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 6.17 scrub ok 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:24:40.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 1.6 scrub starts 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 1.6 scrub ok 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 6.17 scrub starts 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 6.17 scrub ok 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:24:40.118 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:39 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 1.6 scrub starts 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 1.6 scrub ok 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 6.17 scrub starts 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 6.17 scrub ok 2024-03-21T14:24:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:24:40.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:39 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:24:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[29576]: 6.3 scrub starts 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[29576]: 6.3 scrub ok 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[29576]: pgmap v1035: 142 pgs: 1 activating, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[40510]: 6.3 scrub starts 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[40510]: 6.3 scrub ok 2024-03-21T14:24:40.950 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:24:40.951 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:24:40.951 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:40 smithi003 ceph-mon[40510]: pgmap v1035: 142 pgs: 1 activating, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:24:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:40 smithi118 ceph-mon[33862]: 6.3 scrub starts 2024-03-21T14:24:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:40 smithi118 ceph-mon[33862]: 6.3 scrub ok 2024-03-21T14:24:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:40 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:24:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:40 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:24:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:40 smithi118 ceph-mon[33862]: pgmap v1035: 142 pgs: 1 activating, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:24:42.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 3.15 scrub starts 2024-03-21T14:24:42.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 3.15 scrub ok 2024-03-21T14:24:42.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 3.15 scrub starts 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 3.15 scrub ok 2024-03-21T14:24:42.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:24:42.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:24:42.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:24:42.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:24:42.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:24:42.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:24:42.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:41 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 3.15 scrub starts 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 3.15 scrub ok 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:24:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:41 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:24:42.747 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:24:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:24:42.747 INFO:teuthology.orchestra.run.smithi003.stdout: "num_ops": 0 2024-03-21T14:24:42.747 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 1.5 scrub starts 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: 1.5 scrub ok 2024-03-21T14:24:43.224 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[29576]: pgmap v1036: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 1.5 scrub starts 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: 1.5 scrub ok 2024-03-21T14:24:43.225 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:42 smithi003 ceph-mon[40510]: pgmap v1036: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 1.5 scrub starts 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: 1.5 scrub ok 2024-03-21T14:24:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:42 smithi118 ceph-mon[33862]: pgmap v1036: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 828 B/s rd, 0 op/s 2024-03-21T14:24:43.472 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-21T14:24:43.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-21T14:24:43.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-21T14:24:43.727 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:43.975 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:24:44.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[29576]: 5.5 scrub starts 2024-03-21T14:24:44.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[29576]: 5.5 scrub ok 2024-03-21T14:24:44.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[29576]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-21T14:24:44.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3600816219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:44.170 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:44.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[40510]: 5.5 scrub starts 2024-03-21T14:24:44.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[40510]: 5.5 scrub ok 2024-03-21T14:24:44.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[40510]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-21T14:24:44.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3600816219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:44.171 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:43 smithi118 ceph-mon[33862]: 5.5 scrub starts 2024-03-21T14:24:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:43 smithi118 ceph-mon[33862]: 5.5 scrub ok 2024-03-21T14:24:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:43 smithi118 ceph-mon[33862]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-21T14:24:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:43 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3600816219' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[29576]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[29576]: osdmap e462: 8 total, 8 up, 8 in 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[29576]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[40510]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[40510]: osdmap e462: 8 total, 8 up, 8 in 2024-03-21T14:24:45.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:44 smithi003 ceph-mon[40510]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:24:45.259 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:44 smithi118 ceph-mon[33862]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-21T14:24:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:44 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:24:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:44 smithi118 ceph-mon[33862]: osdmap e462: 8 total, 8 up, 8 in 2024-03-21T14:24:45.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:44 smithi118 ceph-mon[33862]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:24:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:45 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:24:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:45 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:24:46.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:45 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:24:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:45 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:24:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:45 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:24:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:45 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:24:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:24:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:24:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:24:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:24:47.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-mon[29576]: pgmap v1039: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:24:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:24:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:24:47.183 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:46 smithi003 ceph-mon[40510]: pgmap v1039: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:24:47.219 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-21T14:24:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:46 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:24:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:46 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:24:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:46 smithi118 ceph-mon[33862]: pgmap v1039: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:24:47.826 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:24:47.826 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [], 2024-03-21T14:24:47.826 INFO:teuthology.orchestra.run.smithi003.stdout: "complaint_time": 30, 2024-03-21T14:24:47.826 INFO:teuthology.orchestra.run.smithi003.stdout: "num_blocked_ops": 0 2024-03-21T14:24:47.826 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:24:47.835 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:47.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:24:47.983 INFO:teuthology.orchestra.run.smithi003.stderr:set pool 1 pgp_num to 11 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[29576]: 6.6 scrub starts 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[29576]: 6.6 scrub ok 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/773408330' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[40510]: 6.6 scrub starts 2024-03-21T14:24:48.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[40510]: 6.6 scrub ok 2024-03-21T14:24:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/773408330' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-21T14:24:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-21T14:24:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:48.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-mon[33862]: 6.6 scrub starts 2024-03-21T14:24:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-mon[33862]: 6.6 scrub ok 2024-03-21T14:24:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/773408330' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-21T14:24:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-21T14:24:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:24:48.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:24:48.629 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: 5.15 scrub starts 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: 5.15 scrub ok 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T14:24:49.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[29576]: pgmap v1041: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: 5.15 scrub starts 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: 5.15 scrub ok 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T14:24:49.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:48 smithi003 ceph-mon[40510]: pgmap v1041: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:24:49.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: 5.15 scrub starts 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: 5.15 scrub ok 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: osdmap e463: 8 total, 8 up, 8 in 2024-03-21T14:24:49.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:48 smithi118 ceph-mon[33862]: pgmap v1041: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:24:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[29576]: 1.4 scrub starts 2024-03-21T14:24:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[29576]: 1.4 scrub ok 2024-03-21T14:24:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:24:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[29576]: 5.6 deep-scrub starts 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[29576]: 5.6 deep-scrub ok 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[40510]: 1.4 scrub starts 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[40510]: 1.4 scrub ok 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[40510]: 5.6 deep-scrub starts 2024-03-21T14:24:50.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:49 smithi003 ceph-mon[40510]: 5.6 deep-scrub ok 2024-03-21T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:49 smithi118 ceph-mon[33862]: 1.4 scrub starts 2024-03-21T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:49 smithi118 ceph-mon[33862]: 1.4 scrub ok 2024-03-21T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:49 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:49 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:49 smithi118 ceph-mon[33862]: 5.6 deep-scrub starts 2024-03-21T14:24:50.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:49 smithi118 ceph-mon[33862]: 5.6 deep-scrub ok 2024-03-21T14:24:50.905 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[29576]: 1.9 scrub starts 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[29576]: 1.9 scrub ok 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[29576]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[29576]: pgmap v1043: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-03-21T14:24:51.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[40510]: 1.9 scrub starts 2024-03-21T14:24:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[40510]: 1.9 scrub ok 2024-03-21T14:24:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:24:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:24:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[40510]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T14:24:51.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:50 smithi003 ceph-mon[40510]: pgmap v1043: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-03-21T14:24:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:50 smithi118 ceph-mon[33862]: 1.9 scrub starts 2024-03-21T14:24:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:50 smithi118 ceph-mon[33862]: 1.9 scrub ok 2024-03-21T14:24:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:50 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:24:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:50 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:24:51.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:50 smithi118 ceph-mon[33862]: osdmap e464: 8 total, 8 up, 8 in 2024-03-21T14:24:51.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:50 smithi118 ceph-mon[33862]: pgmap v1043: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 835 B/s rd, 0 op/s 2024-03-21T14:24:51.560 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:52.016 INFO:teuthology.orchestra.run.smithi003.stderr:noscrub is unset 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3128065033' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:52 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3128065033' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:24:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:51 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:24:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:51 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3128065033' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:51 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-21T14:24:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:51 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:24:52.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:51 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:24:53.130 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a: {} 2024-03-21T14:24:53.141 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b: {} 2024-03-21T14:24:53.141 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[40510]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[40510]: pgmap v1045: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[29576]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T14:24:53.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:53 smithi003 ceph-mon[29576]: pgmap v1045: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:24:53.151 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c: {} 2024-03-21T14:24:53.152 INFO:teuthology.orchestra.run.smithi003.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T14:24:53.153 INFO:teuthology.orchestra.run.smithi003.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T14:24:53.153 INFO:teuthology.orchestra.run.smithi003.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-21T14:24:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:53 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:24:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:53 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:24:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:53 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-21T14:24:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:53 smithi118 ceph-mon[33862]: osdmap e465: 8 total, 8 up, 8 in 2024-03-21T14:24:53.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:53 smithi118 ceph-mon[33862]: pgmap v1045: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:24:53.640 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-21T14:24:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:54 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:24:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:54 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:24:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:54 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/953993908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T14:24:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:54 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:24:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:54 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:24:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:54 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/953993908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T14:24:54.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:54 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:24:54.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:54 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:24:54.332 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:54 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/953993908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-21T14:24:54.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-21T14:24:54.795 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:55.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:55 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:24:55.344 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:55 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:24:55.345 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:55 smithi118 ceph-mon[33862]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T14:24:55.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:55 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:24:55.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:55 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:24:55.349 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:55 smithi003 ceph-mon[29576]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T14:24:55.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:55 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:24:55.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:55 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:24:55.349 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:55 smithi003 ceph-mon[40510]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T14:24:55.506 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:24:56.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:56 smithi003 ceph-mon[29576]: 1.3 scrub starts 2024-03-21T14:24:56.314 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:56 smithi003 ceph-mon[29576]: 1.3 scrub ok 2024-03-21T14:24:56.314 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:56 smithi003 ceph-mon[40510]: 1.3 scrub starts 2024-03-21T14:24:56.314 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:56 smithi003 ceph-mon[40510]: 1.3 scrub ok 2024-03-21T14:24:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:56 smithi118 ceph-mon[33862]: 1.3 scrub starts 2024-03-21T14:24:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:56 smithi118 ceph-mon[33862]: 1.3 scrub ok 2024-03-21T14:24:56.569 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:24:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:24:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:24:56.569 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-03-21T14:24:56.569 INFO:teuthology.orchestra.run.smithi003.stdout: "size": 20, 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 600, 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "ops": [ 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250798 10.9 10:968a5cf1:::benchmark_data_smithi003_160672_object97422:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.736338+0000", 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 315.83088606899997, 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.053511334, 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.570 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250798 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736338+0000", 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736338+0000", 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.571 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736339+0000", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.689e-06 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736344+0000", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1250000000000001e-06 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736356+0000", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1440000000000001e-05 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736372+0000", 2024-03-21T14:24:56.572 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6345e-05 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736675+0000", 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00030272300000000002 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.736685+0000", 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6549999999999999e-06 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.789762+0000", 2024-03-21T14:24:56.573 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0530769639999999 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.789849+0000", 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7392999999999995e-05 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250817 10.9 10:988ed933:::benchmark_data_smithi003_160672_object97441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.741969+0000", 2024-03-21T14:24:56.574 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 315.82525440000001, 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.056560226, 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250817 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741969+0000", 2024-03-21T14:24:56.575 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741969+0000", 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741971+0000", 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.542e-06 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.741985+0000", 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3771000000000001e-05 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742001+0000", 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5928999999999999e-05 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742009+0000", 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9069999999999996e-06 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.577 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742065+0000", 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5371000000000002e-05 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.742075+0000", 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8309999999999996e-06 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.790296+0000", 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.048221058 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.578 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.790474+0000", 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000178025 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.790922+0000", 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000448007 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798262+0000", 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0073399989999999998 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.579 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798381+0000", 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000118768 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798419+0000", 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8210000000000002e-05 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798473+0000", 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4722999999999998e-05 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.580 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798530+0000", 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6085000000000003e-05 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250819 10.9 10:9b87e9d2:::benchmark_data_smithi003_160672_object97443:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.743601+0000", 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 315.82362264300002, 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0550646159999999, 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250819 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743601+0000", 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.582 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743601+0000", 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743602+0000", 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.052e-06 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743610+0000", 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5719999999999997e-06 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.583 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743621+0000", 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1430000000000001e-05 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743630+0000", 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4919999999999993e-06 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743695+0000", 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5195000000000002e-05 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.584 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.743703+0000", 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0290000000000005e-06 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.793749+0000", 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.050046029 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.793880+0000", 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013149 2024-03-21T14:24:56.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.794024+0000", 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014386600000000001 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798328+0000", 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0043040869999999998 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798542+0000", 2024-03-21T14:24:56.586 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00021386500000000001 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798580+0000", 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8172e-05 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798627+0000", 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.6836999999999999e-05 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798666+0000", 2024-03-21T14:24:56.587 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8500000000000001e-05 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:250826 10.9 10:96022229:::benchmark_data_smithi003_160672_object97450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:19:40.752788+0000", 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 315.81443599800002, 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0512797460000001, 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.588 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 250826 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752788+0000", 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752788+0000", 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.589 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752789+0000", 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4670000000000001e-06 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752794+0000", 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.498e-06 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752809+0000", 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5284000000000001e-05 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.590 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752818+0000", 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.0850000000000003e-06 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752981+0000", 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000163287 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:40.752991+0000", 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.6350000000000001e-06 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.591 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.796127+0000", 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0431358509999999 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.796244+0000", 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011756000000000001 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 5,7", 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.796365+0000", 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120338 2024-03-21T14:24:56.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798676+0000", 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0023114519999999999 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.798715+0000", 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8964999999999997e-05 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.803956+0000", 2024-03-21T14:24:56.593 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0052408039999999999 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.804019+0000", 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3431999999999998e-05 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:19:41.804067+0000", 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8087999999999998e-05 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.594 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276756 10.a 10:5a01975e:::benchmark_data_smithi003_160672_object123380:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.951630+0000", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.61559329800002, 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0536734160000001, 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276756 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951630+0000", 2024-03-21T14:24:56.595 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951630+0000", 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.596 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951631+0000", 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9999999999999996e-07 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951636+0000", 2024-03-21T14:24:56.597 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0880000000000002e-06 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951649+0000", 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2384e-05 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951657+0000", 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6789999999999994e-06 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.598 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.963113+0000", 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011455051000000001 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op must wait for map", 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.963120+0000", 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2479999999999999e-06 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.966072+0000", 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0029518650000000001 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.599 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.966083+0000", 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.097e-05 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005133+0000", 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.039050142 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005304+0000", 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000171089 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276757 10.a 10:5609b524:::benchmark_data_smithi003_160672_object123381:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.951674+0000", 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.615549316, 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.05404067, 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276757 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.601 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951674+0000", 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951674+0000", 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951675+0000", 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8100000000000003e-07 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.602 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951678+0000", 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.881e-06 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951684+0000", 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5869999999999998e-06 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951692+0000", 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3180000000000001e-06 2024-03-21T14:24:56.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969235+0000", 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017543751 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969244+0000", 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.1589999999999993e-06 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005666+0000", 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.036422465 2024-03-21T14:24:56.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005715+0000", 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.9027999999999999e-05 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276763 10.a 10:55e751fa:::benchmark_data_smithi003_160672_object123387:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.61552386599999, 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0682812049999999, 2024-03-21T14:24:56.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "reached pg", 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276763 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.606 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951700+0000", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.3000000000000001e-07 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951703+0000", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2259999999999999e-06 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951708+0000", 2024-03-21T14:24:56.607 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7610000000000004e-06 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.951714+0000", 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2480000000000004e-06 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969268+0000", 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.017554311 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.969273+0000", 2024-03-21T14:24:56.608 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.6620000000000002e-06 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005800+0000", 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0365267819999999 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for missing object", 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005888+0000", 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7812999999999994e-05 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.019908+0000", 2024-03-21T14:24:56.609 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.014019723 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.019981+0000", 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.3249000000000003e-05 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276784 10.a 10:58b60665:::benchmark_data_smithi003_160672_object123408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.971287+0000", 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.59593720999999, 2024-03-21T14:24:56.610 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0421002779999999, 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276784 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971287+0000", 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.611 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971287+0000", 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971288+0000", 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5600000000000001e-06 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971297+0000", 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.3209999999999994e-06 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.612 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971308+0000", 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0791000000000001e-05 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971317+0000", 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4980000000000006e-06 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971370+0000", 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3062999999999997e-05 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.613 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971382+0000", 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2031000000000001e-05 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.005911+0000", 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0345289339999999 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.006009+0000", 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.8376999999999997e-05 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.614 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.006443+0000", 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00043439599999999998 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.011607+0000", 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0051635889999999997 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013084+0000", 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0014772990000000001 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.615 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013233+0000", 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00014816300000000001 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013291+0000", 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.8937999999999997e-05 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013387+0000", 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5317999999999998e-05 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.616 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276785 10.a 10:563d6131:::benchmark_data_smithi003_160672_object123409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.971785+0000", 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.59543839499997, 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0526706690000001, 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276785 2024-03-21T14:24:56.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971785+0000", 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971785+0000", 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971786+0000", 2024-03-21T14:24:56.618 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9299999999999996e-07 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971791+0000", 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.831e-06 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971802+0000", 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0675e-05 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971810+0000", 2024-03-21T14:24:56.619 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.943e-06 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971862+0000", 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1919999999999998e-05 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.971870+0000", 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.2279999999999998e-06 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013436+0000", 2024-03-21T14:24:56.620 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0415666139999999 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013561+0000", 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012478000000000001 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.013641+0000", 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.0142999999999995e-05 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.019676+0000", 2024-03-21T14:24:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0060351420000000003 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020014+0000", 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00033725200000000002 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024299+0000", 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042853709999999996 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024349+0000", 2024-03-21T14:24:56.622 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0249e-05 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024456+0000", 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000106628 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276810 10.a 10:595ca004:::benchmark_data_smithi003_160672_object123434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:05.996981+0000", 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.57024295100001, 2024-03-21T14:24:56.623 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.027762557, 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276810 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996981+0000", 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.624 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996981+0000", 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996982+0000", 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2300000000000001e-06 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996987+0000", 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4199999999999998e-06 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.625 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.996998+0000", 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.1015e-05 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.997007+0000", 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.4689999999999997e-06 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.997196+0000", 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000189192 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.626 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:05.997206+0000", 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.995e-06 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.015289+0000", 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.018083045 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.015422+0000", 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013287600000000001 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.627 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.015557+0000", 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013471300000000001 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024466+0000", 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0089097580000000003 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024517+0000", 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0138e-05 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.628 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024582+0000", 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5592999999999994e-05 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024627+0000", 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4487000000000003e-05 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024743+0000", 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116626 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.629 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276838 10.a 10:51795154:::benchmark_data_smithi003_160672_object123462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.018961+0000", 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.54826295999999, 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.006085809, 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276838 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018961+0000", 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018961+0000", 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018962+0000", 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2500000000000001e-06 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.631 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018967+0000", 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2159999999999997e-06 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018979+0000", 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2109e-05 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.018988+0000", 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5599999999999994e-06 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.632 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.019049+0000", 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1373999999999999e-05 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.019059+0000", 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.5719999999999992e-06 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.017466+0000", 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99840741600000005 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.633 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.017571+0000", 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000105098 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.017686+0000", 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000114448 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024473+0000", 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0067870939999999996 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.634 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024548+0000", 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5094999999999998e-05 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024896+0000", 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00034793900000000002 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024931+0000", 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.5045000000000001e-05 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.635 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025047+0000", 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000115593 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276844 10.a 10:5e872913:::benchmark_data_smithi003_160672_object123468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.54136519799999, 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99941103799999997, 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.636 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276844 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025859+0000", 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3799999999999996e-07 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025864+0000", 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.5329999999999996e-06 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025873+0000", 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.6670000000000002e-06 2024-03-21T14:24:56.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025880+0000", 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.5059999999999998e-06 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025944+0000", 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.4313000000000006e-05 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.025953+0000", 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9809999999999992e-06 2024-03-21T14:24:56.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020032+0000", 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99407909500000002 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020142+0000", 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00010915 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.020275+0000", 2024-03-21T14:24:56.640 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013345800000000001 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024789+0000", 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0045142899999999998 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.024851+0000", 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.1876000000000002e-05 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025143+0000", 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00029137500000000002 2024-03-21T14:24:56.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025181+0000", 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 3.8148999999999997e-05 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.025270+0000", 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.8807000000000005e-05 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.642 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276859 10.a 10:5e9d8ab9:::benchmark_data_smithi003_160672_object123483:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.042891+0000", 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.52433230899999, 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99221122399999995, 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276859 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.643 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042891+0000", 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042891+0000", 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042892+0000", 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.2300000000000001e-07 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.644 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042895+0000", 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.8660000000000002e-06 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042902+0000", 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0040000000000002e-06 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.042908+0000", 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.0469999999999999e-06 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.645 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.043025+0000", 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011754 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.043032+0000", 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1080000000000004e-06 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.027880+0000", 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98484717399999999 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.646 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.028056+0000", 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00017604299999999999 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.028177+0000", 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000120949 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034673+0000", 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0064963210000000002 2024-03-21T14:24:56.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034918+0000", 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00024549199999999999 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034966+0000", 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7184000000000001e-05 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.035013+0000", 2024-03-21T14:24:56.648 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.7333000000000002e-05 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.035103+0000", 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.9640000000000002e-05 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276865 10.a 10:5935ebfb:::benchmark_data_smithi003_160672_object123489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:06.046907+0000", 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 290.52031712000002, 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.99829890899999996, 2024-03-21T14:24:56.649 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276865 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046907+0000", 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.650 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046907+0000", 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046908+0000", 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2100000000000001e-06 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046913+0000", 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4550000000000003e-06 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.651 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046923+0000", 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2909999999999996e-06 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046939+0000", 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.6055000000000001e-05 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.046992+0000", 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.3052e-05 2024-03-21T14:24:56.652 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:06.047002+0000", 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.9229999999999997e-06 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.030224+0000", 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.98322199099999996 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.030332+0000", 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000108626 2024-03-21T14:24:56.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 0,5", 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.030446+0000", 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00011424799999999999 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.034880+0000", 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.004433233 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.035146+0000", 2024-03-21T14:24:56.654 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00026675700000000001 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.044794+0000", 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0096480620000000007 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.045083+0000", 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00028866800000000001 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.045206+0000", 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00012233799999999999 2024-03-21T14:24:56.655 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276950 10.3 10:c59a8dcb:::benchmark_data_smithi003_160672_object123574:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:07.058797+0000", 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 289.50842700599998, 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.016286622, 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276950 2024-03-21T14:24:56.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058797+0000", 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058797+0000", 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058798+0000", 2024-03-21T14:24:56.657 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2619999999999999e-06 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058805+0000", 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.2969999999999998e-06 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058820+0000", 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4528999999999999e-05 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.658 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058828+0000", 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.5650000000000002e-06 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058897+0000", 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.8546000000000004e-05 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.058906+0000", 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.2809999999999997e-06 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.659 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.006938+0000", 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94803164699999998 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.007121+0000", 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018341200000000001 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.007251+0000", 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013005800000000001 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.660 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009281+0000", 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0020301920000000001 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.013085+0000", 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.003803374 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.074976+0000", 2024-03-21T14:24:56.661 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.061891310999999997 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.075032+0000", 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5872999999999999e-05 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.662 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.075083+0000", 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1275e-05 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.15666.0:276968 10.3 10:cda049ff:::benchmark_data_smithi003_160672_object123592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e384)", 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:20:07.071761+0000", 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 289.49546294599998, 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.01005374, 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.663 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.15666", 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/2011550766", 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 276968 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071761+0000", 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071761+0000", 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071762+0000", 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.125e-06 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071767+0000", 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1279999999999999e-06 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071777+0000", 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 9.7289999999999995e-06 2024-03-21T14:24:56.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071785+0000", 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.9289999999999998e-06 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071855+0000", 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0745000000000001e-05 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:07.071864+0000", 2024-03-21T14:24:56.666 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.7770000000000008e-06 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009467+0000", 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.93760276399999998 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009583+0000", 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000116512 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,5", 2024-03-21T14:24:56.667 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.009655+0000", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1285999999999995e-05 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.013140+0000", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034857109999999998 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.013185+0000", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4195e-05 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.081693+0000", 2024-03-21T14:24:56.668 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.068508821999999997 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.081763+0000", 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9262000000000002e-05 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:20:08.081815+0000", 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.1755000000000002e-05 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.669 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23439 11.d 11:b7606483:::benchmark_data_smithi003_188959_object23438:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.811936+0000", 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 210.755288175, 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3735996780000002, 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23439 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.670 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811936+0000", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811936+0000", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811937+0000", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3689999999999999e-06 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.671 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811981+0000", 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.4029999999999997e-05 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811991+0000", 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0562e-05 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.811998+0000", 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.2249999999999997e-06 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.672 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.812063+0000", 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.5409999999999999e-05 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.812076+0000", 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.3271000000000001e-05 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759898+0000", 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.94782209299999998 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.673 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759911+0000", 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2788e-05 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.179898+0000", 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4199869139999999 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.180053+0000", 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00015448099999999999 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.674 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.180234+0000", 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018103499999999999 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184334+0000", 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0041007659999999996 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184354+0000", 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.9066e-05 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.675 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184409+0000", 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4957000000000003e-05 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.184451+0000", 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.2542e-05 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.185535+0000", 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010841690000000001 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23501 11.d 11:b959049b:::benchmark_data_smithi003_188959_object23500:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.847823+0000", 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 210.71940089700001, 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3399457090000002, 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23501 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.677 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847823+0000", 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847823+0000", 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847824+0000", 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.2559999999999999e-06 2024-03-21T14:24:56.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847847+0000", 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2459999999999998e-05 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847857+0000", 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0543000000000001e-05 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847864+0000", 2024-03-21T14:24:56.679 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1690000000000003e-06 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847934+0000", 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.9986000000000004e-05 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.847944+0000", 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0173000000000001e-05 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759953+0000", 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.91200873100000002 2024-03-21T14:24:56.680 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759961+0000", 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.79e-06 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.182716+0000", 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.422755558 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.182884+0000", 2024-03-21T14:24:56.681 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016775700000000001 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.183093+0000", 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00020857499999999999 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186539+0000", 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0034464610000000001 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186560+0000", 2024-03-21T14:24:56.682 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.0954999999999998e-05 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186611+0000", 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.0311e-05 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.186682+0000", 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.1217999999999995e-05 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.187769+0000", 2024-03-21T14:24:56.683 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0010867660000000001 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23540 11.d 11:b1a32943:::benchmark_data_smithi003_188959_object23539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.877480+0000", 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 210.68974349999999, 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3302135609999999, 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:56.684 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23540 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877480+0000", 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877480+0000", 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.685 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877481+0000", 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.238e-06 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877504+0000", 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.2674e-05 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877518+0000", 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4299e-05 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.686 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877526+0000", 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.6990000000000003e-06 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877589+0000", 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 6.3091999999999998e-05 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.877600+0000", 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.0730999999999999e-05 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.687 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759982+0000", 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.882382377 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.759989+0000", 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.0210000000000002e-06 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.187818+0000", 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.42782864 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.688 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.188000+0000", 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00018164200000000001 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.188187+0000", 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.000187484 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.192410+0000", 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0042227740000000003 2024-03-21T14:24:56.689 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.196014+0000", 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.0036038210000000001 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.206195+0000", 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.010181638 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.206251+0000", 2024-03-21T14:24:56.690 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.5529000000000003e-05 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.207694+0000", 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.001442902 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "description": "osd_op(client.25811.0:23556 11.d 11:b802ef67:::benchmark_data_smithi003_188959_object23555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "initiated_at": "2024-03-21T14:21:25.893032+0000", 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "age": 210.674192119, 2024-03-21T14:24:56.691 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 2.3156643900000002, 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type_data": { 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client_info": { 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client": "client.25811", 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "client_addr": "172.21.15.3:0/4091728945", 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "tid": 23556 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "events": [ 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "initiated", 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893032+0000", 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.692 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "header_read", 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893032+0000", 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "throttled", 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893033+0000", 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.155e-06 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "all_read", 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893051+0000", 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.8555000000000001e-05 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.693 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "dispatched", 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893066+0000", 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.5079000000000001e-05 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "queued_for_pg", 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893074+0000", 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 7.4900000000000003e-06 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893128+0000", 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.4055000000000003e-05 2024-03-21T14:24:56.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:25.893136+0000", 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.354e-06 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.760015+0000", 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.86687844599999997 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for peered", 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:26.760023+0000", 2024-03-21T14:24:56.695 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 8.3459999999999994e-06 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "reached_pg", 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.189354+0000", 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 1.4293309910000001 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "started", 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.189494+0000", 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00013979899999999999 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "waiting for subops from 4,6", 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.189657+0000", 2024-03-21T14:24:56.696 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00016322900000000001 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.196065+0000", 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.006407968 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "sub_op_commit_rec", 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.196114+0000", 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 4.8859000000000001e-05 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "op_commit", 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.207711+0000", 2024-03-21T14:24:56.697 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.011597077000000001 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "commit_sent", 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.207768+0000", 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 5.7327999999999997e-05 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: "event": "done", 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: "time": "2024-03-21T14:21:28.208696+0000", 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: "duration": 0.00092765899999999995 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-03-21T14:24:56.698 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-03-21T14:24:56.699 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: 3.0 deep-scrub starts 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: 3.0 deep-scrub ok 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:24:57.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: 3.0 deep-scrub starts 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: 3.0 deep-scrub ok 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:24:57.361 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:57 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:24:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: 3.0 deep-scrub starts 2024-03-21T14:24:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: 3.0 deep-scrub ok 2024-03-21T14:24:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:24:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:24:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: osdmap e466: 8 total, 8 up, 8 in 2024-03-21T14:24:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 837 B/s rd, 0 op/s 2024-03-21T14:24:57.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:24:57.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:24:57.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:24:57.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:24:57.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:24:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:24:58.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:24:58.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:24:58.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[29576]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T14:24:58.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[29576]: 3.e scrub starts 2024-03-21T14:24:58.236 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[29576]: 3.e scrub ok 2024-03-21T14:24:58.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:24:58.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:24:58.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[40510]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T14:24:58.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[40510]: 3.e scrub starts 2024-03-21T14:24:58.237 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:58 smithi003 ceph-mon[40510]: 3.e scrub ok 2024-03-21T14:24:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:58 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:24:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:58 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:24:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:58 smithi118 ceph-mon[33862]: osdmap e467: 8 total, 8 up, 8 in 2024-03-21T14:24:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:58 smithi118 ceph-mon[33862]: 3.e scrub starts 2024-03-21T14:24:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:58 smithi118 ceph-mon[33862]: 3.e scrub ok 2024-03-21T14:24:59.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:59 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:59.434 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:59 smithi118 ceph-mon[33862]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T14:24:59.434 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:59 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:24:59.434 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:24:59 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:24:59.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:59.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[29576]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T14:24:59.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:24:59.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:24:59.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:24:59.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[40510]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-21T14:24:59.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:24:59.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:24:59 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:25:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:25:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: 5.1d scrub starts 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: 5.1d scrub ok 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/104045512' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: 5.1d scrub starts 2024-03-21T14:25:00.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: 5.1d scrub ok 2024-03-21T14:25:00.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:25:00.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:25:00.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T14:25:00.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:25:00.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:25:00.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:00 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/104045512' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:25:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:25:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: 5.1d scrub starts 2024-03-21T14:25:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: 5.1d scrub ok 2024-03-21T14:25:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:25:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:25:00.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: osdmap e468: 8 total, 8 up, 8 in 2024-03-21T14:25:00.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:25:00.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:25:00.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:00 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/104045512' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:00.801 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.0 to 1 (10000) 2024-03-21T14:25:00.806 INFO:teuthology.orchestra.run.smithi003.stderr:nodeep-scrub is unset 2024-03-21T14:25:01.391 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: 1.2 scrub starts 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: 1.2 scrub ok 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1195756860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/104045512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[29576]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: 1.2 scrub starts 2024-03-21T14:25:01.392 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: 1.2 scrub ok 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1195756860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/104045512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:25:01.393 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:01 smithi003 ceph-mon[40510]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: 1.2 scrub starts 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: 1.2 scrub ok 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1195756860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/104045512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-21T14:25:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:01 smithi118 ceph-mon[33862]: osdmap e469: 8 total, 8 up, 8 in 2024-03-21T14:25:01.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-21T14:25:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:02 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:25:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:02 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:25:02.496 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:02 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:25:02.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:02 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:25:02.497 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:02 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:25:02.497 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:02 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:25:02.642 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:03 smithi118 ceph-mon[33862]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:03 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:25:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:03 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:25:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:03 smithi118 ceph-mon[33862]: 6.9 scrub starts 2024-03-21T14:25:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:03 smithi118 ceph-mon[33862]: 6.9 scrub ok 2024-03-21T14:25:03.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[29576]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[29576]: 6.9 scrub starts 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[29576]: 6.9 scrub ok 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[40510]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:03.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:25:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:25:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[40510]: 6.9 scrub starts 2024-03-21T14:25:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[40510]: 6.9 scrub ok 2024-03-21T14:25:03.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:04.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:04 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:25:04.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:04 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:25:04.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:04 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:25:04.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:04 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:25:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:04 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:25:04.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:04 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:25:05.169 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-21T14:25:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:05 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:25:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:05 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:25:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:05 smithi118 ceph-mon[33862]: pgmap v1055: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:25:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:05 smithi118 ceph-mon[33862]: 6.13 deep-scrub starts 2024-03-21T14:25:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:05 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2881446826' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:05.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:05 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:05.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:25:05.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:25:05.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[29576]: pgmap v1055: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:25:05.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[29576]: 6.13 deep-scrub starts 2024-03-21T14:25:05.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2881446826' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:05.478 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:05.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:25:05.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:25:05.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[40510]: pgmap v1055: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:25:05.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[40510]: 6.13 deep-scrub starts 2024-03-21T14:25:05.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2881446826' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:05.479 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:05 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-21T14:25:05.905 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-21T14:25:06.161 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:06.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: 6.13 deep-scrub ok 2024-03-21T14:25:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:25:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:25:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:25:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T14:25:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:25:06.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:06 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: 6.13 deep-scrub ok 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:25:06.474 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: 6.13 deep-scrub ok 2024-03-21T14:25:06.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:25:06.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:25:06.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-21T14:25:06.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: osdmap e470: 8 total, 8 up, 8 in 2024-03-21T14:25:06.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:25:06.475 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:25:06.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:25:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:25:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: 3.4 scrub starts 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: 3.4 scrub ok 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T14:25:07.429 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[29576]: 6.1f scrub starts 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: 3.4 scrub starts 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: 3.4 scrub ok 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:07 smithi003 ceph-mon[40510]: 6.1f scrub starts 2024-03-21T14:25:07.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: 3.4 scrub starts 2024-03-21T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: 3.4 scrub ok 2024-03-21T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-21T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: osdmap e471: 8 total, 8 up, 8 in 2024-03-21T14:25:07.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-mon[33862]: 6.1f scrub starts 2024-03-21T14:25:07.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:25:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 6.1f scrub ok 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 1.2 scrub starts 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 1.2 scrub ok 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 5.16 deep-scrub starts 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 5.16 deep-scrub ok 2024-03-21T14:25:08.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:25:08.432 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:08 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 6.1f scrub ok 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 1.2 scrub starts 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 1.2 scrub ok 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 5.16 deep-scrub starts 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 5.16 deep-scrub ok 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:25:08.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 6.1f scrub ok 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 1.2 scrub starts 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 1.2 scrub ok 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 5.16 deep-scrub starts 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 5.16 deep-scrub ok 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:25:08.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:08 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:25:09.187 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.1 to 1 (10000) 2024-03-21T14:25:09.503 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[29576]: 1.1 scrub starts 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[29576]: 1.1 scrub ok 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[29576]: pgmap v1059: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.4 KiB/s rd, 60 KiB/s wr, 10 op/s; 57 KiB/s, 0 objects/s recovering 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2730955407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[40510]: 1.1 scrub starts 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[40510]: 1.1 scrub ok 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[40510]: pgmap v1059: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.4 KiB/s rd, 60 KiB/s wr, 10 op/s; 57 KiB/s, 0 objects/s recovering 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2730955407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:09.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:09 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:09 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:09 smithi118 ceph-mon[33862]: 1.1 scrub starts 2024-03-21T14:25:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:09 smithi118 ceph-mon[33862]: 1.1 scrub ok 2024-03-21T14:25:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:09 smithi118 ceph-mon[33862]: pgmap v1059: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.4 KiB/s rd, 60 KiB/s wr, 10 op/s; 57 KiB/s, 0 objects/s recovering 2024-03-21T14:25:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:09 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2730955407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:09.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:09 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:09.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-21T14:25:10.176 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: 1.6 scrub starts 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: 1.6 scrub ok 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: 4.4 scrub starts 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: 4.4 scrub ok 2024-03-21T14:25:10.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[29576]: pgmap v1061: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 9.3 KiB/s rd, 73 KiB/s wr, 11 op/s; 70 KiB/s, 0 objects/s recovering 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: 1.6 scrub starts 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: 1.6 scrub ok 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: 4.4 scrub starts 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: 4.4 scrub ok 2024-03-21T14:25:10.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:10 smithi003 ceph-mon[40510]: pgmap v1061: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 9.3 KiB/s rd, 73 KiB/s wr, 11 op/s; 70 KiB/s, 0 objects/s recovering 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: 1.6 scrub starts 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: 1.6 scrub ok 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: osdmap e472: 8 total, 8 up, 8 in 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: 4.4 scrub starts 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: 4.4 scrub ok 2024-03-21T14:25:10.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:10 smithi118 ceph-mon[33862]: pgmap v1061: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 9.3 KiB/s rd, 73 KiB/s wr, 11 op/s; 70 KiB/s, 0 objects/s recovering 2024-03-21T14:25:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:25:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:25:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[29576]: 6.12 deep-scrub starts 2024-03-21T14:25:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[29576]: 6.12 deep-scrub ok 2024-03-21T14:25:11.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[29576]: 4.12 deep-scrub starts 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[29576]: 4.12 deep-scrub ok 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[40510]: 6.12 deep-scrub starts 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[40510]: 6.12 deep-scrub ok 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[40510]: 4.12 deep-scrub starts 2024-03-21T14:25:11.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:11 smithi003 ceph-mon[40510]: 4.12 deep-scrub ok 2024-03-21T14:25:11.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:11 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:25:11.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:11 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:25:11.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:11 smithi118 ceph-mon[33862]: 6.12 deep-scrub starts 2024-03-21T14:25:11.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:11 smithi118 ceph-mon[33862]: 6.12 deep-scrub ok 2024-03-21T14:25:11.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:11 smithi118 ceph-mon[33862]: 4.12 deep-scrub starts 2024-03-21T14:25:11.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:11 smithi118 ceph-mon[33862]: 4.12 deep-scrub ok 2024-03-21T14:25:12.197 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: 6.2 scrub starts 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: 6.2 scrub ok 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: 6.19 scrub ok 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: pgmap v1062: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.1 KiB/s rd, 63 KiB/s wr, 10 op/s; 60 KiB/s, 0 objects/s recovering 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1882694461' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:12.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: 6.2 scrub starts 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: 6.2 scrub ok 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: 6.19 scrub ok 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: pgmap v1062: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.1 KiB/s rd, 63 KiB/s wr, 10 op/s; 60 KiB/s, 0 objects/s recovering 2024-03-21T14:25:12.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:12 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1882694461' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: 6.2 scrub starts 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: 6.2 scrub ok 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: 6.19 scrub ok 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: pgmap v1062: 142 pgs: 142 active+clean; 687 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 8.1 KiB/s rd, 63 KiB/s wr, 10 op/s; 60 KiB/s, 0 objects/s recovering 2024-03-21T14:25:12.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:12 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1882694461' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-21T14:25:12.857 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-21T14:25:13.105 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:13.448 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:13 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1882694461' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:25:13.449 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:13 smithi003 ceph-mon[29576]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T14:25:13.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:13 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1882694461' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:25:13.449 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:13 smithi003 ceph-mon[40510]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T14:25:13.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:13 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1882694461' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-21T14:25:13.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:13 smithi118 ceph-mon[33862]: osdmap e473: 8 total, 8 up, 8 in 2024-03-21T14:25:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:14 smithi003 ceph-mon[29576]: 5.e scrub starts 2024-03-21T14:25:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:14 smithi003 ceph-mon[29576]: 5.e scrub ok 2024-03-21T14:25:14.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:14 smithi003 ceph-mon[29576]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-21T14:25:14.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:14 smithi003 ceph-mon[40510]: 5.e scrub starts 2024-03-21T14:25:14.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:14 smithi003 ceph-mon[40510]: 5.e scrub ok 2024-03-21T14:25:14.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:14 smithi003 ceph-mon[40510]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-21T14:25:14.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:14 smithi118 ceph-mon[33862]: 5.e scrub starts 2024-03-21T14:25:14.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:14 smithi118 ceph-mon[33862]: 5.e scrub ok 2024-03-21T14:25:14.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:14 smithi118 ceph-mon[33862]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 645 B/s rd, 0 op/s 2024-03-21T14:25:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:25:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:25:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[29576]: 3.f deep-scrub starts 2024-03-21T14:25:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[29576]: 3.f deep-scrub ok 2024-03-21T14:25:15.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2392689368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:25:15.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:25:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:25:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[40510]: 3.f deep-scrub starts 2024-03-21T14:25:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[40510]: 3.f deep-scrub ok 2024-03-21T14:25:15.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:15 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2392689368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:25:15.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:15 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:25:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:15 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:25:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:15 smithi118 ceph-mon[33862]: 3.f deep-scrub starts 2024-03-21T14:25:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:15 smithi118 ceph-mon[33862]: 3.f deep-scrub ok 2024-03-21T14:25:15.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:15 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2392689368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:25:15.806 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.3 to 1 (10000) 2024-03-21T14:25:16.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: 3.3 deep-scrub starts 2024-03-21T14:25:16.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: 3.3 deep-scrub ok 2024-03-21T14:25:16.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:25:16.490 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2392689368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: 3.13 deep-scrub starts 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: 3.13 deep-scrub ok 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[29576]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: 3.3 deep-scrub starts 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: 3.3 deep-scrub ok 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2392689368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T14:25:16.491 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: 3.13 deep-scrub starts 2024-03-21T14:25:16.492 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: 3.13 deep-scrub ok 2024-03-21T14:25:16.492 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-mon[40510]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:16.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-21T14:25:16.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: 3.3 deep-scrub starts 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: 3.3 deep-scrub ok 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2392689368' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: osdmap e474: 8 total, 8 up, 8 in 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: 3.13 deep-scrub starts 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: 3.13 deep-scrub ok 2024-03-21T14:25:16.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:16 smithi118 ceph-mon[33862]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:16.753 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:25:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:25:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:25:16.913 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:17.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: 5.1c scrub starts 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: 5.1c scrub ok 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: 5.1c scrub starts 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: 5.1c scrub ok 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:25:17.299 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:25:17.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:25:17.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:17.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:17.300 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:17 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: 5.1c scrub starts 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: 5.1c scrub ok 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:17.612 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:25:17.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:25:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:25:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:25:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:25:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:18.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:25:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:25:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:18.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:18.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:18 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:25:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:18 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:25:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:18.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:19.227 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[29576]: 3.19 scrub starts 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[29576]: 3.19 scrub ok 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/757101666' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[29576]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[40510]: 3.19 scrub starts 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[40510]: 3.19 scrub ok 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/757101666' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:25:19.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:19 smithi003 ceph-mon[40510]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:19 smithi118 ceph-mon[33862]: 3.19 scrub starts 2024-03-21T14:25:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:19 smithi118 ceph-mon[33862]: 3.19 scrub ok 2024-03-21T14:25:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:19 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/757101666' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-21T14:25:19.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:19 smithi118 ceph-mon[33862]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:19.970 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-21T14:25:20.226 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:20.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[29576]: 4.1 scrub starts 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[29576]: 4.1 scrub ok 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/757101666' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[29576]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[29576]: 5.4 deep-scrub starts 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[29576]: 5.4 deep-scrub ok 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[40510]: 4.1 scrub starts 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[40510]: 4.1 scrub ok 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/757101666' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[40510]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T14:25:20.483 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[40510]: 5.4 deep-scrub starts 2024-03-21T14:25:20.484 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:20 smithi003 ceph-mon[40510]: 5.4 deep-scrub ok 2024-03-21T14:25:20.680 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:20 smithi118 ceph-mon[33862]: 4.1 scrub starts 2024-03-21T14:25:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:20 smithi118 ceph-mon[33862]: 4.1 scrub ok 2024-03-21T14:25:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:20 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/757101666' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-21T14:25:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:20 smithi118 ceph-mon[33862]: osdmap e475: 8 total, 8 up, 8 in 2024-03-21T14:25:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:20 smithi118 ceph-mon[33862]: 5.4 deep-scrub starts 2024-03-21T14:25:20.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:20 smithi118 ceph-mon[33862]: 5.4 deep-scrub ok 2024-03-21T14:25:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:21 smithi003 ceph-mon[29576]: 4.f scrub starts 2024-03-21T14:25:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:21 smithi003 ceph-mon[29576]: 4.f scrub ok 2024-03-21T14:25:21.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:21 smithi003 ceph-mon[29576]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-21T14:25:21.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:21 smithi003 ceph-mon[40510]: 4.f scrub starts 2024-03-21T14:25:21.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:21 smithi003 ceph-mon[40510]: 4.f scrub ok 2024-03-21T14:25:21.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:21 smithi003 ceph-mon[40510]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-21T14:25:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:21 smithi118 ceph-mon[33862]: 4.f scrub starts 2024-03-21T14:25:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:21 smithi118 ceph-mon[33862]: 4.f scrub ok 2024-03-21T14:25:21.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:21 smithi118 ceph-mon[33862]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 988 B/s rd, 0 op/s 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 3.b scrub starts 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 3.b scrub ok 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 1.8 scrub starts 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[29576]: 1.8 scrub ok 2024-03-21T14:25:22.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 3.b scrub starts 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 3.b scrub ok 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 1.8 scrub starts 2024-03-21T14:25:22.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:22 smithi003 ceph-mon[40510]: 1.8 scrub ok 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 3.b scrub starts 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 3.b scrub ok 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 1.8 scrub starts 2024-03-21T14:25:22.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:22 smithi118 ceph-mon[33862]: 1.8 scrub ok 2024-03-21T14:25:23.252 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.4 to 1 (10000) 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: 3.10 scrub starts 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: 3.10 scrub ok 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: 1.0 scrub starts 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: 1.0 scrub ok 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3805120058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:23.504 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[29576]: pgmap v1072: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: 3.10 scrub starts 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: 3.10 scrub ok 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: 1.0 scrub starts 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: 1.0 scrub ok 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3805120058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:23.505 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:23.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:25:23.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:25:23.506 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:23 smithi003 ceph-mon[40510]: pgmap v1072: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: 3.10 scrub starts 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: 3.10 scrub ok 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: 1.0 scrub starts 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: 1.0 scrub ok 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3805120058' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:25:23.681 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:25:23.682 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:23 smithi118 ceph-mon[33862]: pgmap v1072: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 744 B/s rd, 0 op/s 2024-03-21T14:25:24.013 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-21T14:25:24.264 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:24.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:25:24.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:25:24.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:25:24.498 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[29576]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T14:25:24.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:25:24.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:25:24.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:25:24.499 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:24 smithi003 ceph-mon[40510]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T14:25:24.582 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:24 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:25:24.582 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:24 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:25:24.582 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:24 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:25:24.582 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:24 smithi118 ceph-mon[33862]: osdmap e476: 8 total, 8 up, 8 in 2024-03-21T14:25:25.914 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 3.8 scrub starts 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[29576]: 3.8 scrub ok 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:25:25.915 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:25:25.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:25:25.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:25:25.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:25:25.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:25:25.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 3.8 scrub starts 2024-03-21T14:25:25.916 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:25 smithi003 ceph-mon[40510]: 3.8 scrub ok 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 3.8 scrub starts 2024-03-21T14:25:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:25 smithi118 ceph-mon[33862]: 3.8 scrub ok 2024-03-21T14:25:26.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:25:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:25:26.820 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/779905194' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/779905194' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:27.127 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:26 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:27.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:26 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:25:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:26 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:25:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/779905194' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:26 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-21T14:25:27.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-21T14:25:27.722 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:27.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:25:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:25:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:27 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:25:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:27 smithi118 ceph-mon[33862]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T14:25:27.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:27 smithi118 ceph-mon[33862]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 660 B/s rd, 0 op/s 2024-03-21T14:25:28.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:27 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:25:28.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:27 smithi003 ceph-mon[29576]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T14:25:28.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:27 smithi003 ceph-mon[29576]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 660 B/s rd, 0 op/s 2024-03-21T14:25:28.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:27 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-21T14:25:28.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:27 smithi003 ceph-mon[40510]: osdmap e477: 8 total, 8 up, 8 in 2024-03-21T14:25:28.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:27 smithi003 ceph-mon[40510]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 660 B/s rd, 0 op/s 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[29576]: 5.1 scrub starts 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[29576]: 5.1 scrub ok 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[29576]: 6.d scrub starts 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[29576]: 6.d scrub ok 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[40510]: 5.1 scrub starts 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[40510]: 5.1 scrub ok 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:29.116 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[40510]: 6.d scrub starts 2024-03-21T14:25:29.117 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:28 smithi003 ceph-mon[40510]: 6.d scrub ok 2024-03-21T14:25:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:28 smithi118 ceph-mon[33862]: 5.1 scrub starts 2024-03-21T14:25:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:28 smithi118 ceph-mon[33862]: 5.1 scrub ok 2024-03-21T14:25:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:28 smithi118 ceph-mon[33862]: 6.d scrub starts 2024-03-21T14:25:29.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:28 smithi118 ceph-mon[33862]: 6.d scrub ok 2024-03-21T14:25:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.5 to 1 (10000) 2024-03-21T14:25:30.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 4.1a scrub starts 2024-03-21T14:25:30.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 4.1a scrub ok 2024-03-21T14:25:30.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 6.1d scrub starts 2024-03-21T14:25:30.151 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 6.a scrub starts 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 6.1d scrub ok 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 6.a scrub ok 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/644970591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 4.1a scrub starts 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 4.1a scrub ok 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 6.1d scrub starts 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 6.a scrub starts 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 6.1d scrub ok 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 6.a scrub ok 2024-03-21T14:25:30.152 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:25:30.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/644970591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:25:30.153 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:29 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:25:30.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 4.1a scrub starts 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 4.1a scrub ok 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 6.1d scrub starts 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 6.a scrub starts 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 6.1d scrub ok 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 6.a scrub ok 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: pgmap v1077: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/644970591' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:25:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:29 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:25:30.596 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-21T14:25:30.849 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:31.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 2.1 deep-scrub starts 2024-03-21T14:25:31.086 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 2.1 deep-scrub ok 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 3.1d scrub starts 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 3.1d scrub ok 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 4.1f deep-scrub starts 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 4.1f deep-scrub ok 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/644970591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 2.1 deep-scrub starts 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 2.1 deep-scrub ok 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 3.1d scrub starts 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 3.1d scrub ok 2024-03-21T14:25:31.087 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 4.1f deep-scrub starts 2024-03-21T14:25:31.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 4.1f deep-scrub ok 2024-03-21T14:25:31.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/644970591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:25:31.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T14:25:31.088 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:30 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:25:31.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 2.1 deep-scrub starts 2024-03-21T14:25:31.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 2.1 deep-scrub ok 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 3.1d scrub starts 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 3.1d scrub ok 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 4.1f deep-scrub starts 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 4.1f deep-scrub ok 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/644970591' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: osdmap e478: 8 total, 8 up, 8 in 2024-03-21T14:25:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:30 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: 5.9 deep-scrub starts 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: 5.9 deep-scrub ok 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: 6.f scrub starts 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: 6.f scrub ok 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: 5.19 scrub starts 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: 5.19 scrub ok 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[29576]: pgmap v1079: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-03-21T14:25:32.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: 5.9 deep-scrub starts 2024-03-21T14:25:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: 5.9 deep-scrub ok 2024-03-21T14:25:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: 6.f scrub starts 2024-03-21T14:25:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: 6.f scrub ok 2024-03-21T14:25:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: 5.19 scrub starts 2024-03-21T14:25:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: 5.19 scrub ok 2024-03-21T14:25:32.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:31 smithi003 ceph-mon[40510]: pgmap v1079: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-03-21T14:25:32.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: 5.9 deep-scrub starts 2024-03-21T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: 5.9 deep-scrub ok 2024-03-21T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: 6.f scrub starts 2024-03-21T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: 6.f scrub ok 2024-03-21T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: 5.19 scrub starts 2024-03-21T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: 5.19 scrub ok 2024-03-21T14:25:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:31 smithi118 ceph-mon[33862]: pgmap v1079: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 661 B/s rd, 0 op/s 2024-03-21T14:25:32.834 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-21T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:32 smithi118 ceph-mon[33862]: 5.12 deep-scrub starts 2024-03-21T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:32 smithi118 ceph-mon[33862]: 5.12 deep-scrub ok 2024-03-21T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:32 smithi118 ceph-mon[33862]: 5.0 scrub starts 2024-03-21T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:32 smithi118 ceph-mon[33862]: 5.0 scrub ok 2024-03-21T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3025385528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[29576]: 5.12 deep-scrub starts 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[29576]: 5.12 deep-scrub ok 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[29576]: 5.0 scrub starts 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[29576]: 5.0 scrub ok 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3025385528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:25:33.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[40510]: 5.12 deep-scrub starts 2024-03-21T14:25:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[40510]: 5.12 deep-scrub ok 2024-03-21T14:25:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[40510]: 5.0 scrub starts 2024-03-21T14:25:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[40510]: 5.0 scrub ok 2024-03-21T14:25:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3025385528' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-21T14:25:33.687 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-21T14:25:33.950 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: 1.a scrub starts 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: 1.a scrub ok 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: 6.1a scrub starts 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: 6.1a scrub ok 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3025385528' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-03-21T14:25:34.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[29576]: 5.3 deep-scrub starts 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: 1.a scrub starts 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: 1.a scrub ok 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: 6.1a scrub starts 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: 6.1a scrub ok 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3025385528' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-03-21T14:25:34.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:33 smithi003 ceph-mon[40510]: 5.3 deep-scrub starts 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: 1.a scrub starts 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: 1.a scrub ok 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: 6.1a scrub starts 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: 6.1a scrub ok 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3025385528' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: osdmap e479: 8 total, 8 up, 8 in 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: pgmap v1081: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 829 B/s rd, 0 op/s 2024-03-21T14:25:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:33 smithi118 ceph-mon[33862]: 5.3 deep-scrub starts 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 5.d scrub starts 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 5.d scrub ok 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 4.5 scrub starts 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 4.5 scrub ok 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 5.3 deep-scrub ok 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:25:35.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 5.d scrub starts 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 5.d scrub ok 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 4.5 scrub starts 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 4.5 scrub ok 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 5.3 deep-scrub ok 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:25:35.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:34 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 5.d scrub starts 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 5.d scrub ok 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 4.5 scrub starts 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 4.5 scrub ok 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 5.3 deep-scrub ok 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: osdmap e480: 8 total, 8 up, 8 in 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:25:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:34 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:25:35.849 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.7 to 1 (10000) 2024-03-21T14:25:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:35 smithi118 ceph-mon[33862]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:35 smithi118 ceph-mon[33862]: 6.18 scrub starts 2024-03-21T14:25:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3436933828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:35 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[29576]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[29576]: 6.18 scrub starts 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3436933828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[40510]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[40510]: 6.18 scrub starts 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3436933828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:35 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:36.626 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:25:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:25:36.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-21T14:25:36.911 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[40510]: 6.4 scrub starts 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[40510]: 6.4 scrub ok 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[40510]: 6.18 scrub ok 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[40510]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[29576]: 6.4 scrub starts 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[29576]: 6.4 scrub ok 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[29576]: 6.18 scrub ok 2024-03-21T14:25:37.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:25:37.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:36 smithi003 ceph-mon[29576]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T14:25:37.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:36 smithi118 ceph-mon[33862]: 6.4 scrub starts 2024-03-21T14:25:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:36 smithi118 ceph-mon[33862]: 6.4 scrub ok 2024-03-21T14:25:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:36 smithi118 ceph-mon[33862]: 6.18 scrub ok 2024-03-21T14:25:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:36 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:25:37.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:36 smithi118 ceph-mon[33862]: osdmap e481: 8 total, 8 up, 8 in 2024-03-21T14:25:37.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:37 smithi118 ceph-mon[33862]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:37.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:25:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:25:38.167 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:37 smithi003 ceph-mon[29576]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:38.168 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:37 smithi003 ceph-mon[40510]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:38 smithi118 ceph-mon[33862]: 6.15 scrub starts 2024-03-21T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:38 smithi118 ceph-mon[33862]: 6.1c scrub starts 2024-03-21T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:38 smithi118 ceph-mon[33862]: 6.1c scrub ok 2024-03-21T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:38 smithi118 ceph-mon[33862]: 6.0 scrub starts 2024-03-21T14:25:39.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:38 smithi118 ceph-mon[33862]: 6.0 scrub ok 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[29576]: 6.15 scrub starts 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[29576]: 6.1c scrub starts 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[29576]: 6.1c scrub ok 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[29576]: 6.0 scrub starts 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[29576]: 6.0 scrub ok 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[40510]: 6.15 scrub starts 2024-03-21T14:25:39.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[40510]: 6.1c scrub starts 2024-03-21T14:25:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[40510]: 6.1c scrub ok 2024-03-21T14:25:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[40510]: 6.0 scrub starts 2024-03-21T14:25:39.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:38 smithi003 ceph-mon[40510]: 6.0 scrub ok 2024-03-21T14:25:39.872 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 6.15 scrub ok 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 3.2 scrub starts 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 3.2 scrub ok 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 4.1b scrub starts 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 4.1b scrub ok 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 6.1 scrub starts 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: 6.1 scrub ok 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3862312545' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:40.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:39 smithi118 ceph-mon[33862]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s; 3 B/s, 0 objects/s recovering 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 6.15 scrub ok 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 3.2 scrub starts 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 3.2 scrub ok 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 4.1b scrub starts 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 4.1b scrub ok 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 6.1 scrub starts 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: 6.1 scrub ok 2024-03-21T14:25:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3862312545' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[29576]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s; 3 B/s, 0 objects/s recovering 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 6.15 scrub ok 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 3.2 scrub starts 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 3.2 scrub ok 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 4.1b scrub starts 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 4.1b scrub ok 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 6.1 scrub starts 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: 6.1 scrub ok 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3862312545' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-21T14:25:40.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:39 smithi003 ceph-mon[40510]: pgmap v1086: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2.1 KiB/s rd, 2 op/s; 3 B/s, 0 objects/s recovering 2024-03-21T14:25:40.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-21T14:25:40.981 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: 4.c scrub starts 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: 4.c scrub ok 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: 5.7 scrub starts 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: 5.7 scrub ok 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: 4.c scrub starts 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: 4.c scrub ok 2024-03-21T14:25:41.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:25:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T14:25:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: 5.7 scrub starts 2024-03-21T14:25:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: 5.7 scrub ok 2024-03-21T14:25:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:25:41.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:40 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: 4.c scrub starts 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: 4.c scrub ok 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: osdmap e482: 8 total, 8 up, 8 in 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: 5.7 scrub starts 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: 5.7 scrub ok 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:25:41.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:40 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:25:42.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:41 smithi118 ceph-mon[33862]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T14:25:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:41 smithi118 ceph-mon[33862]: 3.11 scrub starts 2024-03-21T14:25:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:41 smithi118 ceph-mon[33862]: 3.11 scrub ok 2024-03-21T14:25:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:41 smithi118 ceph-mon[33862]: pgmap v1089: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-03-21T14:25:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:41 smithi118 ceph-mon[33862]: 3.1b scrub starts 2024-03-21T14:25:42.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:41 smithi118 ceph-mon[33862]: 3.1b scrub ok 2024-03-21T14:25:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[29576]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T14:25:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[29576]: 3.11 scrub starts 2024-03-21T14:25:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[29576]: 3.11 scrub ok 2024-03-21T14:25:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[29576]: pgmap v1089: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-03-21T14:25:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[29576]: 3.1b scrub starts 2024-03-21T14:25:42.184 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[29576]: 3.1b scrub ok 2024-03-21T14:25:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[40510]: osdmap e483: 8 total, 8 up, 8 in 2024-03-21T14:25:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[40510]: 3.11 scrub starts 2024-03-21T14:25:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[40510]: 3.11 scrub ok 2024-03-21T14:25:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[40510]: pgmap v1089: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 3 B/s, 0 objects/s recovering 2024-03-21T14:25:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[40510]: 3.1b scrub starts 2024-03-21T14:25:42.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:41 smithi003 ceph-mon[40510]: 3.1b scrub ok 2024-03-21T14:25:43.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:42 smithi118 ceph-mon[33862]: 4.e scrub starts 2024-03-21T14:25:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:42 smithi118 ceph-mon[33862]: 4.e scrub ok 2024-03-21T14:25:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:42 smithi118 ceph-mon[33862]: 6.16 scrub starts 2024-03-21T14:25:43.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:42 smithi118 ceph-mon[33862]: 6.16 scrub ok 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[29576]: 4.e scrub starts 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[29576]: 4.e scrub ok 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[29576]: 6.16 scrub starts 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[29576]: 6.16 scrub ok 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[40510]: 4.e scrub starts 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[40510]: 4.e scrub ok 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[40510]: 6.16 scrub starts 2024-03-21T14:25:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:42 smithi003 ceph-mon[40510]: 6.16 scrub ok 2024-03-21T14:25:43.907 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.6 to 1 (10000) 2024-03-21T14:25:44.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: 4.10 scrub starts 2024-03-21T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: 4.10 scrub ok 2024-03-21T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1728213119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: pgmap v1090: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2 B/s, 0 objects/s recovering 2024-03-21T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: 3.5 scrub starts 2024-03-21T14:25:44.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:43 smithi118 ceph-mon[33862]: 3.5 scrub ok 2024-03-21T14:25:44.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: 4.10 scrub starts 2024-03-21T14:25:44.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: 4.10 scrub ok 2024-03-21T14:25:44.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1728213119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: pgmap v1090: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2 B/s, 0 objects/s recovering 2024-03-21T14:25:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: 3.5 scrub starts 2024-03-21T14:25:44.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[29576]: 3.5 scrub ok 2024-03-21T14:25:44.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: 4.10 scrub starts 2024-03-21T14:25:44.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: 4.10 scrub ok 2024-03-21T14:25:44.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1728213119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:44.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:44.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: pgmap v1090: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 2 B/s, 0 objects/s recovering 2024-03-21T14:25:44.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: 3.5 scrub starts 2024-03-21T14:25:44.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:43 smithi003 ceph-mon[40510]: 3.5 scrub ok 2024-03-21T14:25:44.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[29576]: 5.11 scrub starts 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[29576]: 5.11 scrub ok 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[29576]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[29576]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[40510]: 5.11 scrub starts 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[40510]: 5.11 scrub ok 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:25:45.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[40510]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T14:25:45.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:44 smithi003 ceph-mon[40510]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T14:25:45.018 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:45.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:44 smithi118 ceph-mon[33862]: 5.11 scrub starts 2024-03-21T14:25:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:44 smithi118 ceph-mon[33862]: 5.11 scrub ok 2024-03-21T14:25:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:44 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:25:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:44 smithi118 ceph-mon[33862]: osdmap e484: 8 total, 8 up, 8 in 2024-03-21T14:25:45.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:44 smithi118 ceph-mon[33862]: osdmap e485: 8 total, 8 up, 8 in 2024-03-21T14:25:46.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:45 smithi118 ceph-mon[33862]: 4.0 scrub starts 2024-03-21T14:25:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:45 smithi118 ceph-mon[33862]: 4.0 scrub ok 2024-03-21T14:25:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:45 smithi118 ceph-mon[33862]: pgmap v1093: 142 pgs: 6 peering, 2 unknown, 1 active+clean+scrubbing, 133 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:45 smithi118 ceph-mon[33862]: 4.3 deep-scrub starts 2024-03-21T14:25:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:45 smithi118 ceph-mon[33862]: 4.3 deep-scrub ok 2024-03-21T14:25:46.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:45 smithi118 ceph-mon[33862]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T14:25:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[29576]: 4.0 scrub starts 2024-03-21T14:25:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[29576]: 4.0 scrub ok 2024-03-21T14:25:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[29576]: pgmap v1093: 142 pgs: 6 peering, 2 unknown, 1 active+clean+scrubbing, 133 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[29576]: 4.3 deep-scrub starts 2024-03-21T14:25:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[29576]: 4.3 deep-scrub ok 2024-03-21T14:25:46.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[29576]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T14:25:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[40510]: 4.0 scrub starts 2024-03-21T14:25:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[40510]: 4.0 scrub ok 2024-03-21T14:25:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[40510]: pgmap v1093: 142 pgs: 6 peering, 2 unknown, 1 active+clean+scrubbing, 133 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail 2024-03-21T14:25:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[40510]: 4.3 deep-scrub starts 2024-03-21T14:25:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[40510]: 4.3 deep-scrub ok 2024-03-21T14:25:46.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:45 smithi003 ceph-mon[40510]: osdmap e486: 8 total, 8 up, 8 in 2024-03-21T14:25:46.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:25:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:25:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:46 smithi118 ceph-mon[33862]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T14:25:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:46 smithi118 ceph-mon[33862]: 3.d scrub starts 2024-03-21T14:25:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:46 smithi118 ceph-mon[33862]: 3.d scrub ok 2024-03-21T14:25:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:46 smithi118 ceph-mon[33862]: 4.8 deep-scrub starts 2024-03-21T14:25:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:46 smithi118 ceph-mon[33862]: 4.8 deep-scrub ok 2024-03-21T14:25:47.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:46 smithi118 ceph-mon[33862]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[29576]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[29576]: 3.d scrub starts 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[29576]: 3.d scrub ok 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[29576]: 4.8 deep-scrub starts 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[29576]: 4.8 deep-scrub ok 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[29576]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T14:25:47.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[40510]: Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-21T14:25:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[40510]: 3.d scrub starts 2024-03-21T14:25:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[40510]: 3.d scrub ok 2024-03-21T14:25:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[40510]: 4.8 deep-scrub starts 2024-03-21T14:25:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[40510]: 4.8 deep-scrub ok 2024-03-21T14:25:47.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:46 smithi003 ceph-mon[40510]: osdmap e487: 8 total, 8 up, 8 in 2024-03-21T14:25:47.810 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-21T14:25:47.902 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:25:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: pgmap v1096: 142 pgs: 7 peering, 2 unknown, 1 active+clean+scrubbing, 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 54 B/s, 3 objects/s recovering 2024-03-21T14:25:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4051511880' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: 4.b scrub starts 2024-03-21T14:25:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: 4.b scrub ok 2024-03-21T14:25:48.142 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/4051511880' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[29576]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: pgmap v1096: 142 pgs: 7 peering, 2 unknown, 1 active+clean+scrubbing, 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 54 B/s, 3 objects/s recovering 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4051511880' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: 4.b scrub starts 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: 4.b scrub ok 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:48.143 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/4051511880' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:25:48.144 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:47 smithi003 ceph-mon[40510]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T14:25:48.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: pgmap v1096: 142 pgs: 7 peering, 2 unknown, 1 active+clean+scrubbing, 132 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 54 B/s, 3 objects/s recovering 2024-03-21T14:25:48.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4051511880' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-21T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: 4.b scrub starts 2024-03-21T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: 4.b scrub ok 2024-03-21T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/4051511880' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-21T14:25:48.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:47 smithi118 ceph-mon[33862]: osdmap e488: 8 total, 8 up, 8 in 2024-03-21T14:25:48.504 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-21T14:25:48.756 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:49.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:48 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:49.221 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:48 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:49.221 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:48 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:50.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:49 smithi118 ceph-mon[33862]: 6.c scrub starts 2024-03-21T14:25:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:49 smithi118 ceph-mon[33862]: 6.c scrub ok 2024-03-21T14:25:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:49 smithi118 ceph-mon[33862]: 6.e scrub starts 2024-03-21T14:25:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:49 smithi118 ceph-mon[33862]: 6.e scrub ok 2024-03-21T14:25:50.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:49 smithi118 ceph-mon[33862]: pgmap v1098: 142 pgs: 9 peering, 2 unknown, 131 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 136 B/s, 8 objects/s recovering 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[40510]: 6.c scrub starts 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[40510]: 6.c scrub ok 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[40510]: 6.e scrub starts 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[40510]: 6.e scrub ok 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[40510]: pgmap v1098: 142 pgs: 9 peering, 2 unknown, 131 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 136 B/s, 8 objects/s recovering 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[29576]: 6.c scrub starts 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[29576]: 6.c scrub ok 2024-03-21T14:25:50.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[29576]: 6.e scrub starts 2024-03-21T14:25:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[29576]: 6.e scrub ok 2024-03-21T14:25:50.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:49 smithi003 ceph-mon[29576]: pgmap v1098: 142 pgs: 9 peering, 2 unknown, 131 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 136 B/s, 8 objects/s recovering 2024-03-21T14:25:50.927 INFO:teuthology.orchestra.run.smithi003.stderr:reweighted osd.2 to 1 (10000) 2024-03-21T14:25:51.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:50 smithi118 ceph-mon[33862]: 1.7 scrub starts 2024-03-21T14:25:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:50 smithi118 ceph-mon[33862]: 1.7 scrub ok 2024-03-21T14:25:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:50 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3328865580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:51.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:50 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:51.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[29576]: 1.7 scrub starts 2024-03-21T14:25:51.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[29576]: 1.7 scrub ok 2024-03-21T14:25:51.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3328865580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:51.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:51.234 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[40510]: 1.7 scrub starts 2024-03-21T14:25:51.234 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[40510]: 1.7 scrub ok 2024-03-21T14:25:51.234 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3328865580' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:51.234 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:50 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:51.685 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-21T14:25:51.934 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 5.1a scrub starts 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 5.1a scrub ok 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 4.11 scrub starts 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 4.11 scrub ok 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 4.9 deep-scrub starts 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 4.9 deep-scrub ok 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 5.15 deep-scrub starts 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: 5.15 deep-scrub ok 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:25:52.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T14:25:52.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:51 smithi118 ceph-mon[33862]: pgmap v1100: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 987 B/s rd, 0 op/s; 181 B/s, 10 objects/s recovering 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 5.1a scrub starts 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 5.1a scrub ok 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 4.11 scrub starts 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 4.11 scrub ok 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 4.9 deep-scrub starts 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 4.9 deep-scrub ok 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 5.15 deep-scrub starts 2024-03-21T14:25:52.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: 5.15 deep-scrub ok 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[29576]: pgmap v1100: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 987 B/s rd, 0 op/s; 181 B/s, 10 objects/s recovering 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 5.1a scrub starts 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 5.1a scrub ok 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 4.11 scrub starts 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 4.11 scrub ok 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 4.9 deep-scrub starts 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 4.9 deep-scrub ok 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 5.15 deep-scrub starts 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: 5.15 deep-scrub ok 2024-03-21T14:25:52.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:25:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: osdmap e489: 8 total, 8 up, 8 in 2024-03-21T14:25:52.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:51 smithi003 ceph-mon[40510]: pgmap v1100: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 987 B/s rd, 0 op/s; 181 B/s, 10 objects/s recovering 2024-03-21T14:25:53.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:52 smithi118 ceph-mon[33862]: 4.7 scrub starts 2024-03-21T14:25:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:52 smithi118 ceph-mon[33862]: 4.7 scrub ok 2024-03-21T14:25:53.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:52 smithi118 ceph-mon[33862]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-21T14:25:53.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:52 smithi003 ceph-mon[29576]: 4.7 scrub starts 2024-03-21T14:25:53.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:52 smithi003 ceph-mon[29576]: 4.7 scrub ok 2024-03-21T14:25:53.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:52 smithi003 ceph-mon[29576]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-21T14:25:53.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:52 smithi003 ceph-mon[40510]: 4.7 scrub starts 2024-03-21T14:25:53.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:52 smithi003 ceph-mon[40510]: 4.7 scrub ok 2024-03-21T14:25:53.251 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:52 smithi003 ceph-mon[40510]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2024-03-21T14:25:53.938 INFO:teuthology.orchestra.run.smithi003.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[29576]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 152 B/s, 9 objects/s recovering 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[29576]: 3.c scrub starts 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[29576]: 3.c scrub ok 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/747559653' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[40510]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 152 B/s, 9 objects/s recovering 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[40510]: 3.c scrub starts 2024-03-21T14:25:54.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[40510]: 3.c scrub ok 2024-03-21T14:25:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/747559653' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:54.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:53 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:53 smithi118 ceph-mon[33862]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s; 152 B/s, 9 objects/s recovering 2024-03-21T14:25:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:53 smithi118 ceph-mon[33862]: 3.c scrub starts 2024-03-21T14:25:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:53 smithi118 ceph-mon[33862]: 3.c scrub ok 2024-03-21T14:25:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:53 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/747559653' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:54.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:53 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-21T14:25:54.670 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-21T14:25:54.670 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-21T14:25:54.671 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-21T14:25:54.671 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-21T14:25:54.671 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-21T14:25:54.671 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T14:25:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:54 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:25:55.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:54 smithi003 ceph-mon[29576]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T14:25:55.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:54 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:25:55.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:54 smithi003 ceph-mon[40510]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T14:25:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:54 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-21T14:25:55.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:54 smithi118 ceph-mon[33862]: osdmap e490: 8 total, 8 up, 8 in 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[29576]: 6.1b scrub starts 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[29576]: 6.1b scrub ok 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[29576]: 6.3 deep-scrub starts 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[29576]: pgmap v1103: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 713 B/s rd, 0 op/s; 82 B/s, 4 objects/s recovering 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[40510]: 6.1b scrub starts 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[40510]: 6.1b scrub ok 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[40510]: 6.3 deep-scrub starts 2024-03-21T14:25:56.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:55 smithi003 ceph-mon[40510]: pgmap v1103: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 713 B/s rd, 0 op/s; 82 B/s, 4 objects/s recovering 2024-03-21T14:25:56.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:55 smithi118 ceph-mon[33862]: 6.1b scrub starts 2024-03-21T14:25:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:55 smithi118 ceph-mon[33862]: 6.1b scrub ok 2024-03-21T14:25:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:55 smithi118 ceph-mon[33862]: 6.3 deep-scrub starts 2024-03-21T14:25:56.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:55 smithi118 ceph-mon[33862]: pgmap v1103: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 713 B/s rd, 0 op/s; 82 B/s, 4 objects/s recovering 2024-03-21T14:25:56.754 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:25:56] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:25:56.934 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:25:56.935 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":490,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","created":"2024-03-21T13:56:54.530125+0000","modified":"2024-03-21T14:25:53.930924+0000","last_up_change":"2024-03-21T14:19:13.730963+0000","last_in_change":"2024-03-21T14:21:25.732577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T14:00:12.925535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"463","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"268","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T14:02:31.373659+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T14:02:31.794831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T14:02:33.750394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T14:02:35.711498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T14:02:37.763124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"daf9bc3c-4e73-4379-afb6-7ee181b686fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":930096433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":930096433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":930096433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":930096433}]},"public_addr":"172.21.15.3:6801/930096433","cluster_addr":"172.21.15.3:6802/930096433","heartbeat_back_addr":"172.21.15.3:6804/930096433","heartbeat_front_addr":"172.21.15.3:6803/930096433","state":["exists","up"]},{"osd":1,"uuid":"1652f24d-aa0e-471c-bf54-03814b1ee11d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1526485482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1526485482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1526485482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1526485482}]},"public_addr":"172.21.15.3:6805/1526485482","cluster_addr":"172.21.15.3:6806/1526485482","heartbeat_back_addr":"172.21.15.3:6808/1526485482","heartbeat_front_addr":"172.21.15.3:6807/1526485482","state":["exists","up"]},{"osd":2,"uuid":"5e67d907-f8d1-4ecc-85d3-df88da5c364c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":277,"up_from":358,"up_thru":431,"down_at":278,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3011075847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3011075847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3011075847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3011075847}]},"public_addr":"172.21.15.3:6809/3011075847","cluster_addr":"172.21.15.3:6810/3011075847","heartbeat_back_addr":"172.21.15.3:6812/3011075847","heartbeat_front_addr":"172.21.15.3:6811/3011075847","state":["exists","up"]},{"osd":3,"uuid":"ad8dabdf-fce5-4312-9e80-dbbb6ba2947c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":1918209225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1918209225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1918209225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":1918209225}]},"public_addr":"172.21.15.3:6813/1918209225","cluster_addr":"172.21.15.3:6814/1918209225","heartbeat_back_addr":"172.21.15.3:6816/1918209225","heartbeat_front_addr":"172.21.15.3:6815/1918209225","state":["exists","up"]},{"osd":4,"uuid":"b5d486b1-2cb6-46c7-8664-994af07ff7b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":296472230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":296472230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":296472230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":296472230}]},"public_addr":"172.21.15.118:6800/296472230","cluster_addr":"172.21.15.118:6801/296472230","heartbeat_back_addr":"172.21.15.118:6803/296472230","heartbeat_front_addr":"172.21.15.118:6802/296472230","state":["exists","up"]},{"osd":5,"uuid":"bff24605-38ce-47ce-9142-4ced3766476e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":326385780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":326385780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":326385780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":326385780}]},"public_addr":"172.21.15.118:6804/326385780","cluster_addr":"172.21.15.118:6805/326385780","heartbeat_back_addr":"172.21.15.118:6807/326385780","heartbeat_front_addr":"172.21.15.118:6806/326385780","state":["exists","up"]},{"osd":6,"uuid":"73a781e7-8660-4885-b976-2aeb8515109a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":332,"up_from":351,"up_thru":487,"down_at":333,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1863637416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":1863637416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":1863637416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1863637416}]},"public_addr":"172.21.15.118:6808/1863637416","cluster_addr":"172.21.15.118:6809/1863637416","heartbeat_back_addr":"172.21.15.118:6811/1863637416","heartbeat_front_addr":"172.21.15.118:6810/1863637416","state":["exists","up"]},{"osd":7,"uuid":"3e95559b-79f6-40ba-9d98-d350bcdd60e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2128678671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":2128678671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":2128678671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2128678671}]},"public_addr":"172.21.15.118:6812/2128678671","cluster_addr":"172.21.15.118:6813/2128678671","heartbeat_back_addr":"172.21.15.118:6815/2128678671","heartbeat_front_addr":"172.21.15.118:6814/2128678671","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:56.555072+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:59.033221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:55.928922+0000","dead_epoch":277},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:22.261834+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:49.923756+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:51.465357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:05.191736+0000","dead_epoch":332},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:22.482166+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.c","osds":[0,6,7]},{"pgid":"6.10","osds":[0,4,3]}],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":4,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":0}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":3,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"6.2","mappings":[{"from":2,"to":0}]},{"pgid":"6.4","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/45234419":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/384540368":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/3883474825":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/1598547836":"2024-03-22T14:03:15.227370+0000","172.21.15.3:6800/3687155565":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/3884352348":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2319227492":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2154949954":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/4162970769":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/2292819669":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/3411869253":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/520312891":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2267426691":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/3211426112":"2024-03-22T13:57:56.576834+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[40510]: 6.3 deep-scrub ok 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[40510]: 6.b scrub starts 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[40510]: 6.b scrub ok 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3342743893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[29576]: 6.3 deep-scrub ok 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[29576]: 6.b scrub starts 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[29576]: 6.b scrub ok 2024-03-21T14:25:57.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:56 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3342743893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:25:57.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:56 smithi118 ceph-mon[33862]: 6.3 deep-scrub ok 2024-03-21T14:25:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:56 smithi118 ceph-mon[33862]: 6.b scrub starts 2024-03-21T14:25:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:56 smithi118 ceph-mon[33862]: 6.b scrub ok 2024-03-21T14:25:57.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:56 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3342743893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:25:57.719 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-21T14:25:57.719 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-21T14:25:57.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:25:57 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:25:57.975 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:25:58.222 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[29576]: 3.1a scrub starts 2024-03-21T14:25:58.222 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[29576]: 3.1a scrub ok 2024-03-21T14:25:58.222 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[29576]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 46 B/s, 2 objects/s recovering 2024-03-21T14:25:58.222 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[29576]: 3.6 scrub starts 2024-03-21T14:25:58.222 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[29576]: 3.6 scrub ok 2024-03-21T14:25:58.223 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[40510]: 3.1a scrub starts 2024-03-21T14:25:58.223 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[40510]: 3.1a scrub ok 2024-03-21T14:25:58.223 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[40510]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 46 B/s, 2 objects/s recovering 2024-03-21T14:25:58.223 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[40510]: 3.6 scrub starts 2024-03-21T14:25:58.223 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:57 smithi003 ceph-mon[40510]: 3.6 scrub ok 2024-03-21T14:25:58.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:57 smithi118 ceph-mon[33862]: 3.1a scrub starts 2024-03-21T14:25:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:57 smithi118 ceph-mon[33862]: 3.1a scrub ok 2024-03-21T14:25:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:57 smithi118 ceph-mon[33862]: pgmap v1104: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 46 B/s, 2 objects/s recovering 2024-03-21T14:25:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:57 smithi118 ceph-mon[33862]: 3.6 scrub starts 2024-03-21T14:25:58.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:57 smithi118 ceph-mon[33862]: 3.6 scrub ok 2024-03-21T14:25:59.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: 6.5 scrub starts 2024-03-21T14:25:59.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: 6.5 scrub ok 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: 4.18 scrub starts 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: 4.18 scrub ok 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: 5.17 deep-scrub starts 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: 5.17 deep-scrub ok 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: 6.5 scrub starts 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: 6.5 scrub ok 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: 4.18 scrub starts 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: 4.18 scrub ok 2024-03-21T14:25:59.231 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: 5.17 deep-scrub starts 2024-03-21T14:25:59.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: 5.17 deep-scrub ok 2024-03-21T14:25:59.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:59.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:25:59.232 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:58 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:25:59.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: 6.5 scrub starts 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: 6.5 scrub ok 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: 4.18 scrub starts 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: 4.18 scrub ok 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: 5.17 deep-scrub starts 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: 5.17 deep-scrub ok 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:25:59.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:58 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:25:59.814 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:25:59.814 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":491,"fsid":"c60e082a-e78a-11ee-95cd-87774f69a715","created":"2024-03-21T13:56:54.530125+0000","modified":"2024-03-21T14:25:58.954879+0000","last_up_change":"2024-03-21T14:19:13.730963+0000","last_in_change":"2024-03-21T14:21:25.732577+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-21T14:00:12.925535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"463","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"268","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-21T14:02:31.373659+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-21T14:02:31.794831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-21T14:02:33.750394+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-21T14:02:35.711498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-21T14:02:37.763124+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"daf9bc3c-4e73-4379-afb6-7ee181b686fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6801","nonce":930096433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":930096433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":930096433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6803","nonce":930096433}]},"public_addr":"172.21.15.3:6801/930096433","cluster_addr":"172.21.15.3:6802/930096433","heartbeat_back_addr":"172.21.15.3:6804/930096433","heartbeat_front_addr":"172.21.15.3:6803/930096433","state":["exists","up"]},{"osd":1,"uuid":"1652f24d-aa0e-471c-bf54-03814b1ee11d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6805","nonce":1526485482}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":1526485482}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":1526485482}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6807","nonce":1526485482}]},"public_addr":"172.21.15.3:6805/1526485482","cluster_addr":"172.21.15.3:6806/1526485482","heartbeat_back_addr":"172.21.15.3:6808/1526485482","heartbeat_front_addr":"172.21.15.3:6807/1526485482","state":["exists","up"]},{"osd":2,"uuid":"5e67d907-f8d1-4ecc-85d3-df88da5c364c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":277,"up_from":358,"up_thru":431,"down_at":278,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6809","nonce":3011075847}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3011075847}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3011075847}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6811","nonce":3011075847}]},"public_addr":"172.21.15.3:6809/3011075847","cluster_addr":"172.21.15.3:6810/3011075847","heartbeat_back_addr":"172.21.15.3:6812/3011075847","heartbeat_front_addr":"172.21.15.3:6811/3011075847","state":["exists","up"]},{"osd":3,"uuid":"ad8dabdf-fce5-4312-9e80-dbbb6ba2947c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":486,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6813","nonce":1918209225}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":1918209225}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":1918209225}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6815","nonce":1918209225}]},"public_addr":"172.21.15.3:6813/1918209225","cluster_addr":"172.21.15.3:6814/1918209225","heartbeat_back_addr":"172.21.15.3:6816/1918209225","heartbeat_front_addr":"172.21.15.3:6815/1918209225","state":["exists","up"]},{"osd":4,"uuid":"b5d486b1-2cb6-46c7-8664-994af07ff7b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":296472230}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6801","nonce":296472230}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6803","nonce":296472230}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":296472230}]},"public_addr":"172.21.15.118:6800/296472230","cluster_addr":"172.21.15.118:6801/296472230","heartbeat_back_addr":"172.21.15.118:6803/296472230","heartbeat_front_addr":"172.21.15.118:6802/296472230","state":["exists","up"]},{"osd":5,"uuid":"bff24605-38ce-47ce-9142-4ced3766476e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":326385780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6805","nonce":326385780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6807","nonce":326385780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":326385780}]},"public_addr":"172.21.15.118:6804/326385780","cluster_addr":"172.21.15.118:6805/326385780","heartbeat_back_addr":"172.21.15.118:6807/326385780","heartbeat_front_addr":"172.21.15.118:6806/326385780","state":["exists","up"]},{"osd":6,"uuid":"73a781e7-8660-4885-b976-2aeb8515109a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":48,"last_clean_end":332,"up_from":351,"up_thru":487,"down_at":333,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":1863637416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6809","nonce":1863637416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6811","nonce":1863637416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":1863637416}]},"public_addr":"172.21.15.118:6808/1863637416","cluster_addr":"172.21.15.118:6809/1863637416","heartbeat_back_addr":"172.21.15.118:6811/1863637416","heartbeat_front_addr":"172.21.15.118:6810/1863637416","state":["exists","up"]},{"osd":7,"uuid":"3e95559b-79f6-40ba-9d98-d350bcdd60e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":58,"up_thru":487,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":2128678671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6813","nonce":2128678671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6815","nonce":2128678671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":2128678671}]},"public_addr":"172.21.15.118:6812/2128678671","cluster_addr":"172.21.15.118:6813/2128678671","heartbeat_back_addr":"172.21.15.118:6815/2128678671","heartbeat_front_addr":"172.21.15.118:6814/2128678671","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:56.555072+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:59.033221+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:55.928922+0000","dead_epoch":277},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:22.261834+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:57.985790+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:24:51.465357+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:05.191736+0000","dead_epoch":332},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-21T14:25:22.482166+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.c","osds":[0,6,7]},{"pgid":"6.10","osds":[0,4,3]}],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":4,"to":2}]},{"pgid":"3.13","mappings":[{"from":4,"to":0}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":3,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"6.2","mappings":[{"from":2,"to":0}]},{"pgid":"6.4","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:0/45234419":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/384540368":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/3883474825":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/1598547836":"2024-03-22T14:03:15.227370+0000","172.21.15.3:6800/3687155565":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/3884352348":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2319227492":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2154949954":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/4162970769":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/2292819669":"2024-03-22T13:57:29.327821+0000","172.21.15.3:0/3411869253":"2024-03-22T14:03:15.227370+0000","172.21.15.3:0/520312891":"2024-03-22T13:57:56.576834+0000","172.21.15.3:0/2267426691":"2024-03-22T13:57:29.327821+0000","172.21.15.3:6800/3211426112":"2024-03-22T13:57:56.576834+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-21T14:26:00.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:59 smithi003 ceph-mon[29576]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T14:26:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:59 smithi003 ceph-mon[29576]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:00.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:25:59 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3139486208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:26:00.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:59 smithi003 ceph-mon[40510]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T14:26:00.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:59 smithi003 ceph-mon[40510]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:00.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:25:59 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3139486208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:26:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:59 smithi118 ceph-mon[33862]: osdmap e491: 8 total, 8 up, 8 in 2024-03-21T14:26:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:59 smithi118 ceph-mon[33862]: pgmap v1106: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:00.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:25:59 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3139486208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-21T14:26:00.654 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-21T14:26:00.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-21T14:26:00.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-21T14:26:00.656 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-21T14:26:00.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-21T14:26:00.657 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-21T14:26:00.658 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-21T14:26:00.659 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-21T14:26:00.978 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[29576]: 4.17 scrub starts 2024-03-21T14:26:00.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[29576]: 4.17 scrub ok 2024-03-21T14:26:00.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[29576]: 5.c scrub starts 2024-03-21T14:26:00.979 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[29576]: 5.c scrub ok 2024-03-21T14:26:01.136 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:01.139 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:01.141 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:01.145 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:01.149 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:01.150 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:01.253 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[40510]: 4.17 scrub starts 2024-03-21T14:26:01.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[40510]: 4.17 scrub ok 2024-03-21T14:26:01.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[40510]: 5.c scrub starts 2024-03-21T14:26:01.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:00 smithi003 ceph-mon[40510]: 5.c scrub ok 2024-03-21T14:26:01.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:00 smithi118 ceph-mon[33862]: 4.17 scrub starts 2024-03-21T14:26:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:00 smithi118 ceph-mon[33862]: 4.17 scrub ok 2024-03-21T14:26:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:00 smithi118 ceph-mon[33862]: 5.c scrub starts 2024-03-21T14:26:01.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:00 smithi118 ceph-mon[33862]: 5.c scrub ok 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[29576]: 3.1e scrub starts 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[29576]: 3.1e scrub ok 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[29576]: 4.2 scrub starts 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[29576]: 4.2 scrub ok 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[29576]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[40510]: 3.1e scrub starts 2024-03-21T14:26:02.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[40510]: 3.1e scrub ok 2024-03-21T14:26:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[40510]: 4.2 scrub starts 2024-03-21T14:26:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[40510]: 4.2 scrub ok 2024-03-21T14:26:02.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:01 smithi003 ceph-mon[40510]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:02.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:01 smithi118 ceph-mon[33862]: 3.1e scrub starts 2024-03-21T14:26:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:01 smithi118 ceph-mon[33862]: 3.1e scrub ok 2024-03-21T14:26:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:01 smithi118 ceph-mon[33862]: 4.2 scrub starts 2024-03-21T14:26:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:01 smithi118 ceph-mon[33862]: 4.2 scrub ok 2024-03-21T14:26:02.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:01 smithi118 ceph-mon[33862]: pgmap v1107: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:02.734 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:02.734 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:04.157 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: 4.19 scrub starts 2024-03-21T14:26:04.157 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: 4.19 scrub ok 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: 5.f scrub starts 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: 5.f scrub ok 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: pgmap v1108: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: 5.5 deep-scrub starts 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[29576]: 5.5 deep-scrub ok 2024-03-21T14:26:04.158 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: 4.19 scrub starts 2024-03-21T14:26:04.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: 4.19 scrub ok 2024-03-21T14:26:04.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:04.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:04.159 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: 5.f scrub starts 2024-03-21T14:26:04.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: 5.f scrub ok 2024-03-21T14:26:04.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: pgmap v1108: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:04.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: 5.5 deep-scrub starts 2024-03-21T14:26:04.161 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:03 smithi003 ceph-mon[40510]: 5.5 deep-scrub ok 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: 4.19 scrub starts 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: 4.19 scrub ok 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: 5.f scrub starts 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: 5.f scrub ok 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: pgmap v1108: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: 5.5 deep-scrub starts 2024-03-21T14:26:04.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:03 smithi118 ceph-mon[33862]: 5.5 deep-scrub ok 2024-03-21T14:26:05.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:04 smithi118 ceph-mon[33862]: 1.5 scrub starts 2024-03-21T14:26:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:04 smithi118 ceph-mon[33862]: 1.5 scrub ok 2024-03-21T14:26:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:04 smithi118 ceph-mon[33862]: 5.14 scrub starts 2024-03-21T14:26:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:04 smithi118 ceph-mon[33862]: 5.14 scrub ok 2024-03-21T14:26:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:04 smithi118 ceph-mon[33862]: 5.1b scrub starts 2024-03-21T14:26:05.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:04 smithi118 ceph-mon[33862]: 5.1b scrub ok 2024-03-21T14:26:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[29576]: 1.5 scrub starts 2024-03-21T14:26:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[29576]: 1.5 scrub ok 2024-03-21T14:26:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[29576]: 5.14 scrub starts 2024-03-21T14:26:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[29576]: 5.14 scrub ok 2024-03-21T14:26:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[29576]: 5.1b scrub starts 2024-03-21T14:26:05.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[29576]: 5.1b scrub ok 2024-03-21T14:26:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[40510]: 1.5 scrub starts 2024-03-21T14:26:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[40510]: 1.5 scrub ok 2024-03-21T14:26:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[40510]: 5.14 scrub starts 2024-03-21T14:26:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[40510]: 5.14 scrub ok 2024-03-21T14:26:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[40510]: 5.1b scrub starts 2024-03-21T14:26:05.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:04 smithi003 ceph-mon[40510]: 5.1b scrub ok 2024-03-21T14:26:05.880 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[29576]: 6.11 scrub starts 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[29576]: 6.11 scrub ok 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[29576]: 6.14 scrub starts 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[29576]: 6.14 scrub ok 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[29576]: pgmap v1109: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[40510]: 6.11 scrub starts 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[40510]: 6.11 scrub ok 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[40510]: 6.14 scrub starts 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[40510]: 6.14 scrub ok 2024-03-21T14:26:05.881 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:05 smithi003 ceph-mon[40510]: pgmap v1109: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:06.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:05 smithi118 ceph-mon[33862]: 6.11 scrub starts 2024-03-21T14:26:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:05 smithi118 ceph-mon[33862]: 6.11 scrub ok 2024-03-21T14:26:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:05 smithi118 ceph-mon[33862]: 6.14 scrub starts 2024-03-21T14:26:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:05 smithi118 ceph-mon[33862]: 6.14 scrub ok 2024-03-21T14:26:06.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:05 smithi118 ceph-mon[33862]: pgmap v1109: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:06.753 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:06 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:26:06] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:26:07.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:06 smithi118 ceph-mon[33862]: 1.4 scrub starts 2024-03-21T14:26:07.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:06 smithi118 ceph-mon[33862]: 1.4 scrub ok 2024-03-21T14:26:07.198 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:06 smithi003 ceph-mon[29576]: 1.4 scrub starts 2024-03-21T14:26:07.199 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:06 smithi003 ceph-mon[29576]: 1.4 scrub ok 2024-03-21T14:26:07.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:06 smithi003 ceph-mon[40510]: 1.4 scrub starts 2024-03-21T14:26:07.199 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:06 smithi003 ceph-mon[40510]: 1.4 scrub ok 2024-03-21T14:26:07.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:26:07 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:26:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:07 smithi118 ceph-mon[33862]: 3.18 scrub starts 2024-03-21T14:26:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:07 smithi118 ceph-mon[33862]: 3.18 scrub ok 2024-03-21T14:26:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:07 smithi118 ceph-mon[33862]: 5.6 scrub starts 2024-03-21T14:26:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:07 smithi118 ceph-mon[33862]: 5.6 scrub ok 2024-03-21T14:26:07.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:07 smithi118 ceph-mon[33862]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[29576]: 3.18 scrub starts 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[29576]: 3.18 scrub ok 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[29576]: 5.6 scrub starts 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[29576]: 5.6 scrub ok 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[29576]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[40510]: 3.18 scrub starts 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[40510]: 3.18 scrub ok 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[40510]: 5.6 scrub starts 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[40510]: 5.6 scrub ok 2024-03-21T14:26:08.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:07 smithi003 ceph-mon[40510]: pgmap v1110: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:08.878 INFO:teuthology.orchestra.run.smithi003.stdout:171798692138 2024-03-21T14:26:08.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[29576]: 2.0 scrub starts 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[29576]: 2.0 scrub ok 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[29576]: 3.1c scrub starts 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[29576]: 3.1c scrub ok 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[40510]: 2.0 scrub starts 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[40510]: 2.0 scrub ok 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[40510]: 3.1c scrub starts 2024-03-21T14:26:09.099 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:08 smithi003 ceph-mon[40510]: 3.1c scrub ok 2024-03-21T14:26:09.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:08 smithi118 ceph-mon[33862]: 2.0 scrub starts 2024-03-21T14:26:09.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:08 smithi118 ceph-mon[33862]: 2.0 scrub ok 2024-03-21T14:26:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:08 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:08 smithi118 ceph-mon[33862]: 3.1c scrub starts 2024-03-21T14:26:09.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:08 smithi118 ceph-mon[33862]: 3.1c scrub ok 2024-03-21T14:26:09.545 INFO:teuthology.orchestra.run.smithi003.stdout:1507533520983 2024-03-21T14:26:09.546 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-21T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:09 smithi118 ceph-mon[33862]: 3.15 deep-scrub starts 2024-03-21T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:09 smithi118 ceph-mon[33862]: 3.15 deep-scrub ok 2024-03-21T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:09 smithi118 ceph-mon[33862]: 5.8 scrub starts 2024-03-21T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:09 smithi118 ceph-mon[33862]: 5.8 scrub ok 2024-03-21T14:26:10.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:09 smithi118 ceph-mon[33862]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[29576]: 3.15 deep-scrub starts 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[29576]: 3.15 deep-scrub ok 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[29576]: 5.8 scrub starts 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[29576]: 5.8 scrub ok 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[29576]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[40510]: 3.15 deep-scrub starts 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[40510]: 3.15 deep-scrub ok 2024-03-21T14:26:10.204 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[40510]: 5.8 scrub starts 2024-03-21T14:26:10.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[40510]: 5.8 scrub ok 2024-03-21T14:26:10.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:09 smithi003 ceph-mon[40510]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-21T14:26:11.113 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[29576]: 5.18 scrub starts 2024-03-21T14:26:11.113 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[29576]: 5.18 scrub ok 2024-03-21T14:26:11.113 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[29576]: 1.3 scrub starts 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[29576]: 1.3 scrub ok 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[29576]: 1.9 scrub starts 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[29576]: 1.9 scrub ok 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[40510]: 5.18 scrub starts 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[40510]: 5.18 scrub ok 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[40510]: 1.3 scrub starts 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[40510]: 1.3 scrub ok 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[40510]: 1.9 scrub starts 2024-03-21T14:26:11.114 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:10 smithi003 ceph-mon[40510]: 1.9 scrub ok 2024-03-21T14:26:11.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:10 smithi118 ceph-mon[33862]: 5.18 scrub starts 2024-03-21T14:26:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:10 smithi118 ceph-mon[33862]: 5.18 scrub ok 2024-03-21T14:26:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:10 smithi118 ceph-mon[33862]: 1.3 scrub starts 2024-03-21T14:26:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:10 smithi118 ceph-mon[33862]: 1.3 scrub ok 2024-03-21T14:26:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:10 smithi118 ceph-mon[33862]: 1.9 scrub starts 2024-03-21T14:26:11.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:10 smithi118 ceph-mon[33862]: 1.9 scrub ok 2024-03-21T14:26:12.051 INFO:teuthology.orchestra.run.smithi003.stdout:107374182708 2024-03-21T14:26:12.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: 4.13 scrub starts 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: 4.13 scrub ok 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: 5.1d deep-scrub starts 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: 5.1d deep-scrub ok 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: 3.16 scrub starts 2024-03-21T14:26:12.079 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[29576]: 3.16 scrub ok 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: 4.13 scrub starts 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: 4.13 scrub ok 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: 5.1d deep-scrub starts 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: 5.1d deep-scrub ok 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: 3.16 scrub starts 2024-03-21T14:26:12.080 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:11 smithi003 ceph-mon[40510]: 3.16 scrub ok 2024-03-21T14:26:12.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: 4.13 scrub starts 2024-03-21T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: 4.13 scrub ok 2024-03-21T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: 5.1d deep-scrub starts 2024-03-21T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: 5.1d deep-scrub ok 2024-03-21T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: pgmap v1112: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-21T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: 3.16 scrub starts 2024-03-21T14:26:12.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:11 smithi118 ceph-mon[33862]: 3.16 scrub ok 2024-03-21T14:26:12.648 INFO:teuthology.orchestra.run.smithi003.stdout:34359738693 2024-03-21T14:26:12.649 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-21T14:26:13.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[40510]: 6.17 scrub starts 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[40510]: 6.17 scrub ok 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[40510]: 5.1e scrub starts 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[40510]: 5.1e scrub ok 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[29576]: 6.17 scrub starts 2024-03-21T14:26:13.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[29576]: 6.17 scrub ok 2024-03-21T14:26:13.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[29576]: 5.1e scrub starts 2024-03-21T14:26:13.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[29576]: 5.1e scrub ok 2024-03-21T14:26:13.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:13.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:12 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:13.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:12 smithi118 ceph-mon[33862]: 6.17 scrub starts 2024-03-21T14:26:13.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:12 smithi118 ceph-mon[33862]: 6.17 scrub ok 2024-03-21T14:26:13.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:12 smithi118 ceph-mon[33862]: 5.1e scrub starts 2024-03-21T14:26:13.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:12 smithi118 ceph-mon[33862]: 5.1e scrub ok 2024-03-21T14:26:13.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:12 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:13.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:12 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:14.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: 4.1c scrub starts 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: 4.1c scrub ok 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: 3.e deep-scrub starts 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: 3.e deep-scrub ok 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: 4.1e scrub starts 2024-03-21T14:26:14.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:13 smithi118 ceph-mon[33862]: 4.1e scrub ok 2024-03-21T14:26:14.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:14.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:14.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: 4.1c scrub starts 2024-03-21T14:26:14.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: 4.1c scrub ok 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: 3.e deep-scrub starts 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: 3.e deep-scrub ok 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: 4.1e scrub starts 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[29576]: 4.1e scrub ok 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: 4.1c scrub starts 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: 4.1c scrub ok 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: osdmap e492: 8 total, 8 up, 8 in 2024-03-21T14:26:14.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: 3.e deep-scrub starts 2024-03-21T14:26:14.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: 3.e deep-scrub ok 2024-03-21T14:26:14.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:14.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: 4.1e scrub starts 2024-03-21T14:26:14.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:13 smithi003 ceph-mon[40510]: 4.1e scrub ok 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[29576]: 5.a scrub starts 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[29576]: 5.a scrub ok 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[29576]: 3.17 scrub starts 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[29576]: 3.17 scrub ok 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[40510]: 5.a scrub starts 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[40510]: 5.a scrub ok 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[40510]: 3.17 scrub starts 2024-03-21T14:26:15.091 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:14 smithi003 ceph-mon[40510]: 3.17 scrub ok 2024-03-21T14:26:15.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:14 smithi118 ceph-mon[33862]: 5.a scrub starts 2024-03-21T14:26:15.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:14 smithi118 ceph-mon[33862]: 5.a scrub ok 2024-03-21T14:26:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:14 smithi118 ceph-mon[33862]: 3.17 scrub starts 2024-03-21T14:26:15.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:14 smithi118 ceph-mon[33862]: 3.17 scrub ok 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: 3.0 deep-scrub starts 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: 3.0 deep-scrub ok 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: 6.1f deep-scrub starts 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: 6.1f deep-scrub ok 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: pgmap v1115: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: 4.d scrub starts 2024-03-21T14:26:16.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[29576]: 4.d scrub ok 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: 3.0 deep-scrub starts 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: 3.0 deep-scrub ok 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: 6.1f deep-scrub starts 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: 6.1f deep-scrub ok 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: pgmap v1115: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: 4.d scrub starts 2024-03-21T14:26:16.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:15 smithi003 ceph-mon[40510]: 4.d scrub ok 2024-03-21T14:26:16.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: 3.0 deep-scrub starts 2024-03-21T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: 3.0 deep-scrub ok 2024-03-21T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: 6.1f deep-scrub starts 2024-03-21T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: 6.1f deep-scrub ok 2024-03-21T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: pgmap v1115: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: 4.d scrub starts 2024-03-21T14:26:16.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:15 smithi118 ceph-mon[33862]: 4.d scrub ok 2024-03-21T14:26:16.666 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:26:16] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:26:17.003 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[40510]: 6.6 deep-scrub starts 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[40510]: 6.6 deep-scrub ok 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[40510]: 3.14 scrub starts 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[40510]: 3.14 scrub ok 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[40510]: 4.6 scrub starts 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[40510]: 4.6 scrub ok 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[29576]: 6.6 deep-scrub starts 2024-03-21T14:26:17.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[29576]: 6.6 deep-scrub ok 2024-03-21T14:26:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[29576]: 3.14 scrub starts 2024-03-21T14:26:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[29576]: 3.14 scrub ok 2024-03-21T14:26:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[29576]: 4.6 scrub starts 2024-03-21T14:26:17.005 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:16 smithi003 ceph-mon[29576]: 4.6 scrub ok 2024-03-21T14:26:17.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:16 smithi118 ceph-mon[33862]: 6.6 deep-scrub starts 2024-03-21T14:26:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:16 smithi118 ceph-mon[33862]: 6.6 deep-scrub ok 2024-03-21T14:26:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:16 smithi118 ceph-mon[33862]: 3.14 scrub starts 2024-03-21T14:26:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:16 smithi118 ceph-mon[33862]: 3.14 scrub ok 2024-03-21T14:26:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:16 smithi118 ceph-mon[33862]: 4.6 scrub starts 2024-03-21T14:26:17.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:16 smithi118 ceph-mon[33862]: 4.6 scrub ok 2024-03-21T14:26:17.805 INFO:teuthology.orchestra.run.smithi003.stdout:128849019185 2024-03-21T14:26:17.805 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-21T14:26:17.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:26:17 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:26:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:17 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:17 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:17 smithi118 ceph-mon[33862]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:26:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:26:17.931 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:17 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:18.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:18.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:18.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[29576]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:26:18.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:26:18.165 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:18.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:18.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:18.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[40510]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-21T14:26:18.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:26:18.166 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:17 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:18.602 INFO:teuthology.orchestra.run.smithi003.stdout:1537598292054 2024-03-21T14:26:18.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-21T14:26:18.847 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: 5.1f scrub starts 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: 5.1f scrub ok 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:26:18.859 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:19.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: 5.1f scrub starts 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: 5.1f scrub ok 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:18 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:19.181 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: 5.1f scrub starts 2024-03-21T14:26:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: 5.1f scrub ok 2024-03-21T14:26:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: osdmap e493: 8 total, 8 up, 8 in 2024-03-21T14:26:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:26:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:26:19.182 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:18 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:19.217 INFO:teuthology.orchestra.run.smithi003.stdout:249108103457 2024-03-21T14:26:19.218 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-21T14:26:19.228 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:19.229 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:19.234 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:20.172 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:19 smithi003 ceph-mon[29576]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:26:20.173 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:19 smithi003 ceph-mon[29576]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T14:26:20.173 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:19 smithi003 ceph-mon[40510]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:26:20.173 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:19 smithi003 ceph-mon[40510]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T14:26:20.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:19 smithi118 ceph-mon[33862]: pgmap v1118: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:26:20.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:19 smithi118 ceph-mon[33862]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-21T14:26:21.003 INFO:teuthology.orchestra.run.smithi003.stdout:55834575169 2024-03-21T14:26:21.004 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-21T14:26:21.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:20 smithi003 ceph-mon[29576]: 4.14 scrub starts 2024-03-21T14:26:21.119 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:20 smithi003 ceph-mon[29576]: 4.14 scrub ok 2024-03-21T14:26:21.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:20 smithi003 ceph-mon[40510]: 4.14 scrub starts 2024-03-21T14:26:21.120 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:20 smithi003 ceph-mon[40510]: 4.14 scrub ok 2024-03-21T14:26:21.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:20 smithi118 ceph-mon[33862]: 4.14 scrub starts 2024-03-21T14:26:21.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:20 smithi118 ceph-mon[33862]: 4.14 scrub ok 2024-03-21T14:26:21.717 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:22.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:21 smithi118 ceph-mon[33862]: 3.a scrub starts 2024-03-21T14:26:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:21 smithi118 ceph-mon[33862]: 3.a scrub ok 2024-03-21T14:26:22.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:21 smithi118 ceph-mon[33862]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-21T14:26:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:21 smithi003 ceph-mon[29576]: 3.a scrub starts 2024-03-21T14:26:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:21 smithi003 ceph-mon[29576]: 3.a scrub ok 2024-03-21T14:26:22.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:21 smithi003 ceph-mon[29576]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-21T14:26:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:21 smithi003 ceph-mon[40510]: 3.a scrub starts 2024-03-21T14:26:22.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:21 smithi003 ceph-mon[40510]: 3.a scrub ok 2024-03-21T14:26:22.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:21 smithi003 ceph-mon[40510]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-21T14:26:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:23.119 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:23.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:22 smithi118 ceph-mon[33862]: 1.6 deep-scrub starts 2024-03-21T14:26:23.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:22 smithi118 ceph-mon[33862]: 1.6 deep-scrub ok 2024-03-21T14:26:23.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:22 smithi003 ceph-mon[29576]: 1.6 deep-scrub starts 2024-03-21T14:26:23.205 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:22 smithi003 ceph-mon[29576]: 1.6 deep-scrub ok 2024-03-21T14:26:23.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:22 smithi003 ceph-mon[40510]: 1.6 deep-scrub starts 2024-03-21T14:26:23.205 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:22 smithi003 ceph-mon[40510]: 1.6 deep-scrub ok 2024-03-21T14:26:24.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:23 smithi118 ceph-mon[33862]: 4.12 scrub starts 2024-03-21T14:26:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:23 smithi118 ceph-mon[33862]: 4.12 scrub ok 2024-03-21T14:26:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:23 smithi118 ceph-mon[33862]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:26:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:23 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:24.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:23 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[29576]: 4.12 scrub starts 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[29576]: 4.12 scrub ok 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[29576]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[40510]: 4.12 scrub starts 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[40510]: 4.12 scrub ok 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[40510]: pgmap v1121: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:24.200 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:23 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:25.121 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: 6.13 scrub starts 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: 3.4 deep-scrub starts 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: 3.4 deep-scrub ok 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: 6.13 scrub ok 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: 1.1 scrub starts 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[29576]: 1.1 scrub ok 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: 6.13 scrub starts 2024-03-21T14:26:25.122 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: 3.4 deep-scrub starts 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: 3.4 deep-scrub ok 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: 6.13 scrub ok 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: 1.1 scrub starts 2024-03-21T14:26:25.123 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:24 smithi003 ceph-mon[40510]: 1.1 scrub ok 2024-03-21T14:26:25.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: 6.13 scrub starts 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: 3.4 deep-scrub starts 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: 3.4 deep-scrub ok 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: 6.13 scrub ok 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: osdmap e494: 8 total, 8 up, 8 in 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: 1.1 scrub starts 2024-03-21T14:26:25.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:24 smithi118 ceph-mon[33862]: 1.1 scrub ok 2024-03-21T14:26:25.753 INFO:teuthology.orchestra.run.smithi003.stdout:128849019187 2024-03-21T14:26:25.980 INFO:teuthology.orchestra.run.smithi003.stdout:1507533520987 2024-03-21T14:26:25.980 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[29576]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[29576]: 5.16 scrub starts 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[29576]: 5.16 scrub ok 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2892612223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[40510]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[40510]: 5.16 scrub starts 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[40510]: 5.16 scrub ok 2024-03-21T14:26:25.981 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:25 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2892612223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T14:26:26.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:25 smithi118 ceph-mon[33862]: pgmap v1123: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-03-21T14:26:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:25 smithi118 ceph-mon[33862]: 5.16 scrub starts 2024-03-21T14:26:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:25 smithi118 ceph-mon[33862]: 5.16 scrub ok 2024-03-21T14:26:26.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:25 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2892612223' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-21T14:26:26.753 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:26:26] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:26:26.839 INFO:teuthology.orchestra.run.smithi003.stdout:107374182711 2024-03-21T14:26:27.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: 5.10 scrub starts 2024-03-21T14:26:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: 5.10 scrub ok 2024-03-21T14:26:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T14:26:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/525190338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T14:26:27.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:26 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1368726281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: 5.10 scrub starts 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: 5.10 scrub ok 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/525190338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1368726281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: 5.10 scrub starts 2024-03-21T14:26:27.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: 5.10 scrub ok 2024-03-21T14:26:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: osdmap e495: 8 total, 8 up, 8 in 2024-03-21T14:26:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/525190338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-21T14:26:27.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:26 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1368726281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-21T14:26:27.524 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1507533520983 got 1507533520987 for osd.6 2024-03-21T14:26:27.524 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:27.930 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:27 smithi118 ceph-mon[33862]: pgmap v1125: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:27.931 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:26:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:26:27.931 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:27 smithi003 ceph-mon[29576]: pgmap v1125: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:27.931 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:27 smithi003 ceph-mon[40510]: pgmap v1125: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:28.926 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:28 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:28.926 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:28 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:29.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:28 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:29.849 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849019185 got 128849019187 for osd.4 2024-03-21T14:26:29.849 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:30.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:29 smithi118 ceph-mon[33862]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:30.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:29 smithi118 ceph-mon[33862]: 6.7 scrub starts 2024-03-21T14:26:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:29 smithi003 ceph-mon[29576]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:30.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:29 smithi003 ceph-mon[29576]: 6.7 scrub starts 2024-03-21T14:26:30.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:29 smithi003 ceph-mon[40510]: pgmap v1126: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:30.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:29 smithi003 ceph-mon[40510]: 6.7 scrub starts 2024-03-21T14:26:30.902 INFO:teuthology.orchestra.run.smithi003.stdout:55834575172 2024-03-21T14:26:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:30 smithi118 ceph-mon[33862]: 4.1d scrub starts 2024-03-21T14:26:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:30 smithi118 ceph-mon[33862]: 4.1d scrub ok 2024-03-21T14:26:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:30 smithi118 ceph-mon[33862]: 6.7 scrub ok 2024-03-21T14:26:31.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:30 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2727105872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[29576]: 4.1d scrub starts 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[29576]: 4.1d scrub ok 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[29576]: 6.7 scrub ok 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2727105872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[40510]: 4.1d scrub starts 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[40510]: 4.1d scrub ok 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[40510]: 6.7 scrub ok 2024-03-21T14:26:31.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:30 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2727105872' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-21T14:26:31.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182708 got 107374182711 for osd.3 2024-03-21T14:26:31.407 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:31.842 INFO:teuthology.orchestra.run.smithi003.stdout:1537598292057 2024-03-21T14:26:32.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:31 smithi118 ceph-mon[33862]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:32.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:31 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2465662265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T14:26:32.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:31 smithi003 ceph-mon[29576]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:32.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:31 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2465662265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T14:26:32.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:31 smithi003 ceph-mon[40510]: pgmap v1127: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-21T14:26:32.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:31 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2465662265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-21T14:26:32.270 INFO:teuthology.orchestra.run.smithi003.stdout:249108103460 2024-03-21T14:26:32.896 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1537598292054 got 1537598292057 for osd.2 2024-03-21T14:26:32.896 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:33.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: 6.12 deep-scrub starts 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: 6.12 deep-scrub ok 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: 4.16 scrub starts 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: 4.16 scrub ok 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/1471843690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: 2.1 scrub starts 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: 2.1 scrub ok 2024-03-21T14:26:33.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:32 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: 6.12 deep-scrub starts 2024-03-21T14:26:33.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: 6.12 deep-scrub ok 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: 4.16 scrub starts 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: 4.16 scrub ok 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/1471843690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: 2.1 scrub starts 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: 2.1 scrub ok 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: 6.12 deep-scrub starts 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: 6.12 deep-scrub ok 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: 4.16 scrub starts 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: 4.16 scrub ok 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/1471843690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: 2.1 scrub starts 2024-03-21T14:26:33.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: 2.1 scrub ok 2024-03-21T14:26:33.256 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:32 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-21T14:26:33.957 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575169 got 55834575172 for osd.1 2024-03-21T14:26:33.957 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:34.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:33 smithi118 ceph-mon[33862]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:33 smithi118 ceph-mon[33862]: 3.1 scrub starts 2024-03-21T14:26:34.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:33 smithi118 ceph-mon[33862]: 3.1 scrub ok 2024-03-21T14:26:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:33 smithi003 ceph-mon[29576]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:33 smithi003 ceph-mon[29576]: 3.1 scrub starts 2024-03-21T14:26:34.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:33 smithi003 ceph-mon[29576]: 3.1 scrub ok 2024-03-21T14:26:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:33 smithi003 ceph-mon[40510]: pgmap v1128: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:33 smithi003 ceph-mon[40510]: 3.1 scrub starts 2024-03-21T14:26:34.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:33 smithi003 ceph-mon[40510]: 3.1 scrub ok 2024-03-21T14:26:35.084 INFO:teuthology.orchestra.run.smithi003.stdout:34359738698 2024-03-21T14:26:35.096 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[29576]: 4.1f scrub starts 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[29576]: 4.1f scrub ok 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[29576]: 6.9 deep-scrub starts 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[29576]: 6.9 deep-scrub ok 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[29576]: 4.15 scrub starts 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[29576]: 4.15 scrub ok 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[40510]: 4.1f scrub starts 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[40510]: 4.1f scrub ok 2024-03-21T14:26:35.097 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[40510]: 6.9 deep-scrub starts 2024-03-21T14:26:35.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[40510]: 6.9 deep-scrub ok 2024-03-21T14:26:35.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[40510]: 4.15 scrub starts 2024-03-21T14:26:35.098 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:34 smithi003 ceph-mon[40510]: 4.15 scrub ok 2024-03-21T14:26:35.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:34 smithi118 ceph-mon[33862]: 4.1f scrub starts 2024-03-21T14:26:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:34 smithi118 ceph-mon[33862]: 4.1f scrub ok 2024-03-21T14:26:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:34 smithi118 ceph-mon[33862]: 6.9 deep-scrub starts 2024-03-21T14:26:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:34 smithi118 ceph-mon[33862]: 6.9 deep-scrub ok 2024-03-21T14:26:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:34 smithi118 ceph-mon[33862]: 4.15 scrub starts 2024-03-21T14:26:35.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:34 smithi118 ceph-mon[33862]: 4.15 scrub ok 2024-03-21T14:26:35.398 INFO:tasks.cephadm.ceph_manager.ceph:need seq 249108103457 got 249108103460 for osd.7 2024-03-21T14:26:35.399 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:36.180 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:35 smithi118 ceph-mon[33862]: 1.2 scrub starts 2024-03-21T14:26:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:35 smithi118 ceph-mon[33862]: 1.2 scrub ok 2024-03-21T14:26:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:35 smithi118 ceph-mon[33862]: pgmap v1129: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:36.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:35 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2067641396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T14:26:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[29576]: 1.2 scrub starts 2024-03-21T14:26:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[29576]: 1.2 scrub ok 2024-03-21T14:26:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[29576]: pgmap v1129: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:36.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2067641396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T14:26:36.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[40510]: 1.2 scrub starts 2024-03-21T14:26:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[40510]: 1.2 scrub ok 2024-03-21T14:26:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[40510]: pgmap v1129: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:36.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:35 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2067641396' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-21T14:26:36.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738693 got 34359738698 for osd.0 2024-03-21T14:26:36.500 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:36.724 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:36 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:26:36] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:26:37.174 INFO:teuthology.orchestra.run.smithi003.stdout:171798692144 2024-03-21T14:26:37.767 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692138 got 171798692144 for osd.5 2024-03-21T14:26:37.767 DEBUG:teuthology.parallel:result is None 2024-03-21T14:26:37.767 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-21T14:26:37.767 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:26:38.003 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 3.f scrub starts 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 3.f scrub ok 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 3.3 scrub starts 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 3.3 scrub ok 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/3050915790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 5.d deep-scrub starts 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 5.d deep-scrub ok 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 4.4 deep-scrub starts 2024-03-21T14:26:38.004 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[29576]: 4.4 deep-scrub ok 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 3.f scrub starts 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 3.f scrub ok 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 3.3 scrub starts 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 3.3 scrub ok 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/3050915790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 5.d deep-scrub starts 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 5.d deep-scrub ok 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 4.4 deep-scrub starts 2024-03-21T14:26:38.005 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:37 smithi003 ceph-mon[40510]: 4.4 deep-scrub ok 2024-03-21T14:26:38.017 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:38.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 3.f scrub starts 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 3.f scrub ok 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 3.3 scrub starts 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 3.3 scrub ok 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: pgmap v1130: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 935 B/s rd, 0 op/s 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/3050915790' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-21T14:26:38.181 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 5.d deep-scrub starts 2024-03-21T14:26:38.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 5.d deep-scrub ok 2024-03-21T14:26:38.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 4.4 deep-scrub starts 2024-03-21T14:26:38.182 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:37 smithi118 ceph-mon[33862]: 4.4 deep-scrub ok 2024-03-21T14:26:39.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: 5.12 scrub starts 2024-03-21T14:26:39.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: 5.12 scrub ok 2024-03-21T14:26:39.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:39.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: 4.a scrub starts 2024-03-21T14:26:39.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: 4.a scrub ok 2024-03-21T14:26:39.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: 5.b scrub starts 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[29576]: 5.b scrub ok 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: 5.12 scrub starts 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: 5.12 scrub ok 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: 4.a scrub starts 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: 4.a scrub ok 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: 5.b scrub starts 2024-03-21T14:26:39.229 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:38 smithi003 ceph-mon[40510]: 5.b scrub ok 2024-03-21T14:26:39.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: 5.12 scrub starts 2024-03-21T14:26:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: 5.12 scrub ok 2024-03-21T14:26:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-21T14:26:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: 4.a scrub starts 2024-03-21T14:26:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: 4.a scrub ok 2024-03-21T14:26:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: 5.b scrub starts 2024-03-21T14:26:39.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:38 smithi118 ceph-mon[33862]: 5.b scrub ok 2024-03-21T14:26:40.021 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:26:40.024 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:26:40.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[29576]: purged_snaps scrub starts 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[29576]: purged_snaps scrub ok 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[29576]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[29576]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[29576]: 6.10 scrub starts 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[29576]: 6.10 scrub ok 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[40510]: purged_snaps scrub starts 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[40510]: purged_snaps scrub ok 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[40510]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[40510]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[40510]: 6.10 scrub starts 2024-03-21T14:26:40.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:39 smithi003 ceph-mon[40510]: 6.10 scrub ok 2024-03-21T14:26:40.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:39 smithi118 ceph-mon[33862]: purged_snaps scrub starts 2024-03-21T14:26:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:39 smithi118 ceph-mon[33862]: purged_snaps scrub ok 2024-03-21T14:26:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:39 smithi118 ceph-mon[33862]: pgmap v1131: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-21T14:26:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:39 smithi118 ceph-mon[33862]: osdmap e496: 8 total, 8 up, 8 in 2024-03-21T14:26:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:39 smithi118 ceph-mon[33862]: 6.10 scrub starts 2024-03-21T14:26:40.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:39 smithi118 ceph-mon[33862]: 6.10 scrub ok 2024-03-21T14:26:40.656 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1132,"stamp":"2024-03-21T14:26:38.936515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2444,"num_read_kb":2196,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":5004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2627,"ondisk_log_size":2627,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5837380,"kb_used_data":7868,"kb_used_omap":21,"kb_used_meta":5829418,"kb_avail":744123836,"statfs":{"total":767960285184,"available":761982808064,"internally_reserved":0,"allocated":8056832,"data_stored":4925464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22067,"internal_metadata":5969324493},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.096526"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":930,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895972+0000","last_change":"2024-03-21T14:25:32.652858+0000","last_active":"2024-03-21T14:26:25.895972+0000","last_peered":"2024-03-21T14:26:25.895972+0000","last_clean":"2024-03-21T14:26:25.895972+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:26:25.895972+0000","last_undegraded":"2024-03-21T14:26:25.895972+0000","last_fullsized":"2024-03-21T14:26:25.895972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:32.652753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:32.652753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:37.108829+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":959,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895309+0000","last_change":"2024-03-21T14:25:30.704554+0000","last_active":"2024-03-21T14:26:25.895309+0000","last_peered":"2024-03-21T14:26:25.895309+0000","last_clean":"2024-03-21T14:26:25.895309+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:26:25.895309+0000","last_undegraded":"2024-03-21T14:26:25.895309+0000","last_fullsized":"2024-03-21T14:26:25.895309+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:30.704477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:25:30.704477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:43.641722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":281,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896894+0000","last_change":"2024-03-21T14:25:40.566340+0000","last_active":"2024-03-21T14:26:25.896894+0000","last_peered":"2024-03-21T14:26:25.896894+0000","last_clean":"2024-03-21T14:26:25.896894+0000","last_became_active":"2024-03-21T14:25:06.187486+0000","last_became_peered":"2024-03-21T14:25:06.187486+0000","last_unstale":"2024-03-21T14:26:25.896894+0000","last_undegraded":"2024-03-21T14:26:25.896894+0000","last_fullsized":"2024-03-21T14:26:25.896894+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:40.566239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:40.566239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:42.823444+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":1021,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895596+0000","last_change":"2024-03-21T14:25:57.458594+0000","last_active":"2024-03-21T14:26:25.895596+0000","last_peered":"2024-03-21T14:26:25.895596+0000","last_clean":"2024-03-21T14:26:25.895596+0000","last_became_active":"2024-03-21T14:25:44.829149+0000","last_became_peered":"2024-03-21T14:25:44.829149+0000","last_unstale":"2024-03-21T14:26:25.895596+0000","last_undegraded":"2024-03-21T14:26:25.895596+0000","last_fullsized":"2024-03-21T14:26:25.895596+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:25:57.458462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.458462+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.731738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":241,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003170+0000","last_change":"2024-03-21T14:25:54.676827+0000","last_active":"2024-03-21T14:26:26.003170+0000","last_peered":"2024-03-21T14:26:26.003170+0000","last_clean":"2024-03-21T14:26:26.003170+0000","last_became_active":"2024-03-21T14:25:44.830791+0000","last_became_peered":"2024-03-21T14:25:44.830791+0000","last_unstale":"2024-03-21T14:26:26.003170+0000","last_undegraded":"2024-03-21T14:26:26.003170+0000","last_fullsized":"2024-03-21T14:26:26.003170+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:54.676712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:54.676712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:21.173467+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":955,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895425+0000","last_change":"2024-03-21T14:26:09.371706+0000","last_active":"2024-03-21T14:26:25.895425+0000","last_peered":"2024-03-21T14:26:25.895425+0000","last_clean":"2024-03-21T14:26:25.895425+0000","last_became_active":"2024-03-21T14:25:44.828628+0000","last_became_peered":"2024-03-21T14:25:44.828628+0000","last_unstale":"2024-03-21T14:26:25.895425+0000","last_undegraded":"2024-03-21T14:26:25.895425+0000","last_fullsized":"2024-03-21T14:26:25.895425+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:09.371608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:26:09.371608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:19.630543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":967,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003531+0000","last_change":"2024-03-21T14:26:00.448519+0000","last_active":"2024-03-21T14:26:26.003531+0000","last_peered":"2024-03-21T14:26:26.003531+0000","last_clean":"2024-03-21T14:26:26.003531+0000","last_became_active":"2024-03-21T14:25:44.829775+0000","last_became_peered":"2024-03-21T14:25:44.829775+0000","last_unstale":"2024-03-21T14:26:26.003531+0000","last_undegraded":"2024-03-21T14:26:26.003531+0000","last_fullsized":"2024-03-21T14:26:26.003531+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:00.448423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:00.448423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:00.848779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'46","reported_seq":1016,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002249+0000","last_change":"2024-03-21T14:26:02.534332+0000","last_active":"2024-03-21T14:26:26.002249+0000","last_peered":"2024-03-21T14:26:26.002249+0000","last_clean":"2024-03-21T14:26:26.002249+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:26:26.002249+0000","last_undegraded":"2024-03-21T14:26:26.002249+0000","last_fullsized":"2024-03-21T14:26:26.002249+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-21T14:26:02.534201+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:26:02.534201+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.683916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":594,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895895+0000","last_change":"2024-03-21T14:25:35.794434+0000","last_active":"2024-03-21T14:26:25.895895+0000","last_peered":"2024-03-21T14:26:25.895895+0000","last_clean":"2024-03-21T14:26:25.895895+0000","last_became_active":"2024-03-21T14:25:06.180542+0000","last_became_peered":"2024-03-21T14:25:06.180542+0000","last_unstale":"2024-03-21T14:26:25.895895+0000","last_undegraded":"2024-03-21T14:26:25.895895+0000","last_fullsized":"2024-03-21T14:26:25.895895+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:35.794333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:35.794333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:02.359190+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":966,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002456+0000","last_change":"2024-03-21T14:26:03.852118+0000","last_active":"2024-03-21T14:26:26.002456+0000","last_peered":"2024-03-21T14:26:26.002456+0000","last_clean":"2024-03-21T14:26:26.002456+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:26:26.002456+0000","last_undegraded":"2024-03-21T14:26:26.002456+0000","last_fullsized":"2024-03-21T14:26:26.002456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:03.852004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.852004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:11.861692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":594,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004258+0000","last_change":"2024-03-21T14:25:44.824475+0000","last_active":"2024-03-21T14:26:26.004258+0000","last_peered":"2024-03-21T14:26:26.004258+0000","last_clean":"2024-03-21T14:26:26.004258+0000","last_became_active":"2024-03-21T14:25:44.823480+0000","last_became_peered":"2024-03-21T14:25:44.823480+0000","last_unstale":"2024-03-21T14:26:26.004258+0000","last_undegraded":"2024-03-21T14:26:26.004258+0000","last_fullsized":"2024-03-21T14:26:26.004258+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:29.587727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:25:29.587727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:56.792309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'21","reported_seq":964,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896176+0000","last_change":"2024-03-21T14:25:28.565057+0000","last_active":"2024-03-21T14:26:25.896176+0000","last_peered":"2024-03-21T14:26:25.896176+0000","last_clean":"2024-03-21T14:26:25.896176+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:26:25.896176+0000","last_undegraded":"2024-03-21T14:26:25.896176+0000","last_fullsized":"2024-03-21T14:26:25.896176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:25:28.564942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:28.564942+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:45.461197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":709,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001971+0000","last_change":"2024-03-21T14:25:33.854317+0000","last_active":"2024-03-21T14:26:26.001971+0000","last_peered":"2024-03-21T14:26:26.001971+0000","last_clean":"2024-03-21T14:26:26.001971+0000","last_became_active":"2024-03-21T14:25:33.853824+0000","last_became_peered":"2024-03-21T14:25:33.853824+0000","last_unstale":"2024-03-21T14:26:26.001971+0000","last_undegraded":"2024-03-21T14:26:26.001971+0000","last_fullsized":"2024-03-21T14:26:26.001971+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:11.787910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:11.787910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:35.359492+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":959,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002356+0000","last_change":"2024-03-21T14:25:49.999479+0000","last_active":"2024-03-21T14:26:26.002356+0000","last_peered":"2024-03-21T14:26:26.002356+0000","last_clean":"2024-03-21T14:26:26.002356+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:26:26.002356+0000","last_undegraded":"2024-03-21T14:26:26.002356+0000","last_fullsized":"2024-03-21T14:26:26.002356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:49.999361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:25:49.999361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:10.807521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":590,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003508+0000","last_change":"2024-03-21T14:26:07.821639+0000","last_active":"2024-03-21T14:26:26.003508+0000","last_peered":"2024-03-21T14:26:26.003508+0000","last_clean":"2024-03-21T14:26:26.003508+0000","last_became_active":"2024-03-21T14:25:33.849959+0000","last_became_peered":"2024-03-21T14:25:33.849959+0000","last_unstale":"2024-03-21T14:26:26.003508+0000","last_undegraded":"2024-03-21T14:26:26.003508+0000","last_fullsized":"2024-03-21T14:26:26.003508+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:07.821538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_clean_scrub_stamp":"2024-03-21T14:26:07.821538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:22.857051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"445'10","reported_seq":955,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896079+0000","last_change":"2024-03-21T14:25:38.571193+0000","last_active":"2024-03-21T14:26:25.896079+0000","last_peered":"2024-03-21T14:26:25.896079+0000","last_clean":"2024-03-21T14:26:25.896079+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:26:25.896079+0000","last_undegraded":"2024-03-21T14:26:25.896079+0000","last_fullsized":"2024-03-21T14:26:25.896079+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'10","last_scrub_stamp":"2024-03-21T14:25:38.571104+0000","last_deep_scrub":"445'10","last_deep_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.571104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:52.055693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":936,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896643+0000","last_change":"2024-03-21T14:25:44.821878+0000","last_active":"2024-03-21T14:26:25.896643+0000","last_peered":"2024-03-21T14:26:25.896643+0000","last_clean":"2024-03-21T14:26:25.896643+0000","last_became_active":"2024-03-21T14:25:44.821097+0000","last_became_peered":"2024-03-21T14:25:44.821097+0000","last_unstale":"2024-03-21T14:26:25.896643+0000","last_undegraded":"2024-03-21T14:26:25.896643+0000","last_fullsized":"2024-03-21T14:26:25.896643+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:03.866108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:03.866108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:31.753805+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":951,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897040+0000","last_change":"2024-03-21T14:26:10.507780+0000","last_active":"2024-03-21T14:26:25.897040+0000","last_peered":"2024-03-21T14:26:25.897040+0000","last_clean":"2024-03-21T14:26:25.897040+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:26:25.897040+0000","last_undegraded":"2024-03-21T14:26:25.897040+0000","last_fullsized":"2024-03-21T14:26:25.897040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:10.507698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:10.507698+0000","last_clean_scrub_stamp":"2024-03-21T14:26:10.507698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:12.801218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":584,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897289+0000","last_change":"2024-03-21T14:25:41.585099+0000","last_active":"2024-03-21T14:26:25.897289+0000","last_peered":"2024-03-21T14:26:25.897289+0000","last_clean":"2024-03-21T14:26:25.897289+0000","last_became_active":"2024-03-21T14:25:06.185211+0000","last_became_peered":"2024-03-21T14:25:06.185211+0000","last_unstale":"2024-03-21T14:26:25.897289+0000","last_undegraded":"2024-03-21T14:26:25.897289+0000","last_fullsized":"2024-03-21T14:26:25.897289+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:41.584987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:25:41.584987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.619880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":882,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002265+0000","last_change":"2024-03-21T14:26:12.607821+0000","last_active":"2024-03-21T14:26:26.002265+0000","last_peered":"2024-03-21T14:26:26.002265+0000","last_clean":"2024-03-21T14:26:26.002265+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:26:26.002265+0000","last_undegraded":"2024-03-21T14:26:26.002265+0000","last_fullsized":"2024-03-21T14:26:26.002265+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-21T14:26:12.607692+0000","last_deep_scrub":"432'34","last_deep_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_clean_scrub_stamp":"2024-03-21T14:26:12.607692+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:39.617259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":961,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003662+0000","last_change":"2024-03-21T14:26:14.782547+0000","last_active":"2024-03-21T14:26:26.003662+0000","last_peered":"2024-03-21T14:26:26.003662+0000","last_clean":"2024-03-21T14:26:26.003662+0000","last_became_active":"2024-03-21T14:25:44.829912+0000","last_became_peered":"2024-03-21T14:25:44.829912+0000","last_unstale":"2024-03-21T14:26:26.003662+0000","last_undegraded":"2024-03-21T14:26:26.003662+0000","last_fullsized":"2024-03-21T14:26:26.003662+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:14.782388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:14.782388+0000","last_clean_scrub_stamp":"2024-03-21T14:26:14.782388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.923263+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":935,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896186+0000","last_change":"2024-03-21T14:25:15.656176+0000","last_active":"2024-03-21T14:26:25.896186+0000","last_peered":"2024-03-21T14:26:25.896186+0000","last_clean":"2024-03-21T14:26:25.896186+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:26:25.896186+0000","last_undegraded":"2024-03-21T14:26:25.896186+0000","last_fullsized":"2024-03-21T14:26:25.896186+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:15.656080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:15.656080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:39.924790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":943,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896368+0000","last_change":"2024-03-21T14:25:56.492197+0000","last_active":"2024-03-21T14:26:25.896368+0000","last_peered":"2024-03-21T14:26:25.896368+0000","last_clean":"2024-03-21T14:26:25.896368+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:26:25.896368+0000","last_undegraded":"2024-03-21T14:26:25.896368+0000","last_fullsized":"2024-03-21T14:26:25.896368+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:56.492049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:25:56.492049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:08.701378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":996,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:29.494525+0000","last_change":"2024-03-21T14:26:29.494564+0000","last_active":"2024-03-21T14:26:29.494525+0000","last_peered":"2024-03-21T14:26:29.494525+0000","last_clean":"2024-03-21T14:26:29.494525+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:26:29.494525+0000","last_undegraded":"2024-03-21T14:26:29.494525+0000","last_fullsized":"2024-03-21T14:26:29.494525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-21T14:26:29.494442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:26:29.494442+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:37.258263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":676,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003051+0000","last_change":"2024-03-21T14:25:40.881675+0000","last_active":"2024-03-21T14:26:26.003051+0000","last_peered":"2024-03-21T14:26:26.003051+0000","last_clean":"2024-03-21T14:26:26.003051+0000","last_became_active":"2024-03-21T14:25:40.881203+0000","last_became_peered":"2024-03-21T14:25:40.881203+0000","last_unstale":"2024-03-21T14:26:26.003051+0000","last_undegraded":"2024-03-21T14:26:26.003051+0000","last_fullsized":"2024-03-21T14:26:26.003051+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:25:38.140789+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.140789+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:42.969048+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"425'15","reported_seq":261,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776936+0000","last_change":"2024-03-21T14:26:17.632016+0000","last_active":"2024-03-21T14:26:28.776936+0000","last_peered":"2024-03-21T14:26:28.776936+0000","last_clean":"2024-03-21T14:26:28.776936+0000","last_became_active":"2024-03-21T14:21:26.767116+0000","last_became_peered":"2024-03-21T14:21:26.767116+0000","last_unstale":"2024-03-21T14:26:28.776936+0000","last_undegraded":"2024-03-21T14:26:28.776936+0000","last_fullsized":"2024-03-21T14:26:28.776936+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'15","last_scrub_stamp":"2024-03-21T14:26:17.631885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:17.631885+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.670224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":968,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895199+0000","last_change":"2024-03-21T14:25:18.715391+0000","last_active":"2024-03-21T14:26:25.895199+0000","last_peered":"2024-03-21T14:26:25.895199+0000","last_clean":"2024-03-21T14:26:25.895199+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:26:25.895199+0000","last_undegraded":"2024-03-21T14:26:25.895199+0000","last_fullsized":"2024-03-21T14:26:25.895199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:25:18.715307+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:25:18.715307+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.314016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":629,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897601+0000","last_change":"2024-03-21T14:26:13.474687+0000","last_active":"2024-03-21T14:26:25.897601+0000","last_peered":"2024-03-21T14:26:25.897601+0000","last_clean":"2024-03-21T14:26:25.897601+0000","last_became_active":"2024-03-21T14:25:06.188176+0000","last_became_peered":"2024-03-21T14:25:06.188176+0000","last_unstale":"2024-03-21T14:26:25.897601+0000","last_undegraded":"2024-03-21T14:26:25.897601+0000","last_fullsized":"2024-03-21T14:26:25.897601+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-21T14:26:13.474554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:26:13.474554+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.509099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":947,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896755+0000","last_change":"2024-03-21T14:25:28.823011+0000","last_active":"2024-03-21T14:26:25.896755+0000","last_peered":"2024-03-21T14:26:25.896755+0000","last_clean":"2024-03-21T14:26:25.896755+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:26:25.896755+0000","last_undegraded":"2024-03-21T14:26:25.896755+0000","last_fullsized":"2024-03-21T14:26:25.896755+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:28.822882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:06.928522+0000","last_clean_scrub_stamp":"2024-03-21T14:25:28.822882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:55.344105+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":422,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897738+0000","last_change":"2024-03-21T14:26:12.450780+0000","last_active":"2024-03-21T14:26:25.897738+0000","last_peered":"2024-03-21T14:26:25.897738+0000","last_clean":"2024-03-21T14:26:25.897738+0000","last_became_active":"2024-03-21T14:25:06.187798+0000","last_became_peered":"2024-03-21T14:25:06.187798+0000","last_unstale":"2024-03-21T14:26:25.897738+0000","last_undegraded":"2024-03-21T14:26:25.897738+0000","last_fullsized":"2024-03-21T14:26:25.897738+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:12.450693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:26:12.450693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.727757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":972,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002703+0000","last_change":"2024-03-21T14:26:06.458507+0000","last_active":"2024-03-21T14:26:26.002703+0000","last_peered":"2024-03-21T14:26:26.002703+0000","last_clean":"2024-03-21T14:26:26.002703+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:26:26.002703+0000","last_undegraded":"2024-03-21T14:26:26.002703+0000","last_fullsized":"2024-03-21T14:26:26.002703+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:06.458427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:26:06.458427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:16.698768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'32","reported_seq":51,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:33.697732+0000","last_change":"2024-03-21T14:26:33.697778+0000","last_active":"2024-03-21T14:26:33.697732+0000","last_peered":"2024-03-21T14:26:33.697732+0000","last_clean":"2024-03-21T14:26:33.697732+0000","last_became_active":"2024-03-21T14:25:44.830640+0000","last_became_peered":"2024-03-21T14:25:44.830640+0000","last_unstale":"2024-03-21T14:26:33.697732+0000","last_undegraded":"2024-03-21T14:26:33.697732+0000","last_fullsized":"2024-03-21T14:26:33.697732+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:26:33.697657+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-21T14:25:29.702878+0000","last_clean_scrub_stamp":"2024-03-21T14:26:33.697657+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:34.251898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":588,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:31.226029+0000","last_change":"2024-03-21T14:26:31.226056+0000","last_active":"2024-03-21T14:26:31.226029+0000","last_peered":"2024-03-21T14:26:31.226029+0000","last_clean":"2024-03-21T14:26:31.226029+0000","last_became_active":"2024-03-21T14:25:40.881790+0000","last_became_peered":"2024-03-21T14:25:40.881790+0000","last_unstale":"2024-03-21T14:26:31.226029+0000","last_undegraded":"2024-03-21T14:26:31.226029+0000","last_fullsized":"2024-03-21T14:26:31.226029+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:31.225936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:31.225936+0000","last_clean_scrub_stamp":"2024-03-21T14:26:31.225936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:55.860753+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":31,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002647+0000","last_change":"2024-03-21T14:25:44.829116+0000","last_active":"2024-03-21T14:26:26.002647+0000","last_peered":"2024-03-21T14:26:26.002647+0000","last_clean":"2024-03-21T14:26:26.002647+0000","last_became_active":"2024-03-21T14:25:44.827267+0000","last_became_peered":"2024-03-21T14:25:44.827267+0000","last_unstale":"2024-03-21T14:26:26.002647+0000","last_undegraded":"2024-03-21T14:26:26.002647+0000","last_fullsized":"2024-03-21T14:26:26.002647+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:43.509640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:25:43.509640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:10.670416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":446,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897159+0000","last_change":"2024-03-21T14:26:14.489099+0000","last_active":"2024-03-21T14:26:25.897159+0000","last_peered":"2024-03-21T14:26:25.897159+0000","last_clean":"2024-03-21T14:26:25.897159+0000","last_became_active":"2024-03-21T14:25:06.187626+0000","last_became_peered":"2024-03-21T14:25:06.187626+0000","last_unstale":"2024-03-21T14:26:25.897159+0000","last_undegraded":"2024-03-21T14:26:25.897159+0000","last_fullsized":"2024-03-21T14:26:25.897159+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:14.489015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:14.489015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:27.251845+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"441'14","reported_seq":1000,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002800+0000","last_change":"2024-03-21T14:25:44.855919+0000","last_active":"2024-03-21T14:26:26.002800+0000","last_peered":"2024-03-21T14:26:26.002800+0000","last_clean":"2024-03-21T14:26:26.002800+0000","last_became_active":"2024-03-21T14:25:44.830268+0000","last_became_peered":"2024-03-21T14:25:44.830268+0000","last_unstale":"2024-03-21T14:26:26.002800+0000","last_undegraded":"2024-03-21T14:26:26.002800+0000","last_fullsized":"2024-03-21T14:26:26.002800+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'14","last_scrub_stamp":"2024-03-21T14:25:42.334314+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:25:42.334314+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:56.758732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":978,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003532+0000","last_change":"2024-03-21T14:26:23.650255+0000","last_active":"2024-03-21T14:26:26.003532+0000","last_peered":"2024-03-21T14:26:26.003532+0000","last_clean":"2024-03-21T14:26:26.003532+0000","last_became_active":"2024-03-21T14:25:44.835043+0000","last_became_peered":"2024-03-21T14:25:44.835043+0000","last_unstale":"2024-03-21T14:26:26.003532+0000","last_undegraded":"2024-03-21T14:26:26.003532+0000","last_fullsized":"2024-03-21T14:26:26.003532+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:23.650090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:04.400332+0000","last_clean_scrub_stamp":"2024-03-21T14:26:23.650090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:49.232270+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":996,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002864+0000","last_change":"2024-03-21T14:26:25.090014+0000","last_active":"2024-03-21T14:26:26.002864+0000","last_peered":"2024-03-21T14:26:26.002864+0000","last_clean":"2024-03-21T14:26:26.002864+0000","last_became_active":"2024-03-21T14:25:44.824474+0000","last_became_peered":"2024-03-21T14:25:44.824474+0000","last_unstale":"2024-03-21T14:26:26.002864+0000","last_undegraded":"2024-03-21T14:26:26.002864+0000","last_fullsized":"2024-03-21T14:26:26.002864+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:25.089884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:25.089884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:38.286023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":596,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004247+0000","last_change":"2024-03-21T14:26:11.714981+0000","last_active":"2024-03-21T14:26:26.004247+0000","last_peered":"2024-03-21T14:26:26.004247+0000","last_clean":"2024-03-21T14:26:26.004247+0000","last_became_active":"2024-03-21T14:25:33.855990+0000","last_became_peered":"2024-03-21T14:25:33.855990+0000","last_unstale":"2024-03-21T14:26:26.004247+0000","last_undegraded":"2024-03-21T14:26:26.004247+0000","last_fullsized":"2024-03-21T14:26:26.004247+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:26:11.714855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:11.714855+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.516720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'32","reported_seq":1040,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002137+0000","last_change":"2024-03-21T14:25:50.432842+0000","last_active":"2024-03-21T14:26:26.002137+0000","last_peered":"2024-03-21T14:26:26.002137+0000","last_clean":"2024-03-21T14:26:26.002137+0000","last_became_active":"2024-03-21T14:25:44.829719+0000","last_became_peered":"2024-03-21T14:25:44.829719+0000","last_unstale":"2024-03-21T14:26:26.002137+0000","last_undegraded":"2024-03-21T14:26:26.002137+0000","last_fullsized":"2024-03-21T14:26:26.002137+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:25:50.432680+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:25:50.432680+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:54.065801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":426,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895689+0000","last_change":"2024-03-21T14:25:29.852606+0000","last_active":"2024-03-21T14:26:25.895689+0000","last_peered":"2024-03-21T14:26:25.895689+0000","last_clean":"2024-03-21T14:26:25.895689+0000","last_became_active":"2024-03-21T14:25:06.181970+0000","last_became_peered":"2024-03-21T14:25:06.181970+0000","last_unstale":"2024-03-21T14:26:25.895689+0000","last_undegraded":"2024-03-21T14:26:25.895689+0000","last_fullsized":"2024-03-21T14:26:25.895689+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:29.852502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:29.852502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:37.300731+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":944,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003469+0000","last_change":"2024-03-21T14:25:14.222159+0000","last_active":"2024-03-21T14:26:26.003469+0000","last_peered":"2024-03-21T14:26:26.003469+0000","last_clean":"2024-03-21T14:26:26.003469+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:26:26.003469+0000","last_undegraded":"2024-03-21T14:26:26.003469+0000","last_fullsized":"2024-03-21T14:26:26.003469+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:14.222043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:25:14.222043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.188015+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":975,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003461+0000","last_change":"2024-03-21T14:26:08.041589+0000","last_active":"2024-03-21T14:26:26.003461+0000","last_peered":"2024-03-21T14:26:26.003461+0000","last_clean":"2024-03-21T14:26:26.003461+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:26:26.003461+0000","last_undegraded":"2024-03-21T14:26:26.003461+0000","last_fullsized":"2024-03-21T14:26:26.003461+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:08.041435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:08.041435+0000","last_clean_scrub_stamp":"2024-03-21T14:26:08.041435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:12.644782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"472'25","reported_seq":1035,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896111+0000","last_change":"2024-03-21T14:26:22.497135+0000","last_active":"2024-03-21T14:26:25.896111+0000","last_peered":"2024-03-21T14:26:25.896111+0000","last_clean":"2024-03-21T14:26:25.896111+0000","last_became_active":"2024-03-21T14:25:44.820409+0000","last_became_peered":"2024-03-21T14:25:44.820409+0000","last_unstale":"2024-03-21T14:26:25.896111+0000","last_undegraded":"2024-03-21T14:26:25.896111+0000","last_fullsized":"2024-03-21T14:26:25.896111+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'25","last_scrub_stamp":"2024-03-21T14:26:22.497041+0000","last_deep_scrub":"472'25","last_deep_scrub_stamp":"2024-03-21T14:25:10.723028+0000","last_clean_scrub_stamp":"2024-03-21T14:26:22.497041+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.134295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":243,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001763+0000","last_change":"2024-03-21T14:26:04.523815+0000","last_active":"2024-03-21T14:26:26.001763+0000","last_peered":"2024-03-21T14:26:26.001763+0000","last_clean":"2024-03-21T14:26:26.001763+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:26:26.001763+0000","last_undegraded":"2024-03-21T14:26:26.001763+0000","last_fullsized":"2024-03-21T14:26:26.001763+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:04.523701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:04.523701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.784394+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":980,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:37.646717+0000","last_change":"2024-03-21T14:26:37.646760+0000","last_active":"2024-03-21T14:26:37.646717+0000","last_peered":"2024-03-21T14:26:37.646717+0000","last_clean":"2024-03-21T14:26:37.646717+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:26:37.646717+0000","last_undegraded":"2024-03-21T14:26:37.646717+0000","last_fullsized":"2024-03-21T14:26:37.646717+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:37.646641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:31.695568+0000","last_clean_scrub_stamp":"2024-03-21T14:26:37.646641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:56.755984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":972,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895984+0000","last_change":"2024-03-21T14:26:16.289389+0000","last_active":"2024-03-21T14:26:25.895984+0000","last_peered":"2024-03-21T14:26:25.895984+0000","last_clean":"2024-03-21T14:26:25.895984+0000","last_became_active":"2024-03-21T14:25:44.819688+0000","last_became_peered":"2024-03-21T14:25:44.819688+0000","last_unstale":"2024-03-21T14:26:25.895984+0000","last_undegraded":"2024-03-21T14:26:25.895984+0000","last_fullsized":"2024-03-21T14:26:25.895984+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:16.289291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:16.289291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.260254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":1057,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896155+0000","last_change":"2024-03-21T14:26:10.357525+0000","last_active":"2024-03-21T14:26:25.896155+0000","last_peered":"2024-03-21T14:26:25.896155+0000","last_clean":"2024-03-21T14:26:25.896155+0000","last_became_active":"2024-03-21T14:21:26.767846+0000","last_became_peered":"2024-03-21T14:21:26.767846+0000","last_unstale":"2024-03-21T14:26:25.896155+0000","last_undegraded":"2024-03-21T14:26:25.896155+0000","last_fullsized":"2024-03-21T14:26:25.896155+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:26:10.357408+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:26:10.357408+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:34.385462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":660,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897435+0000","last_change":"2024-03-21T14:25:42.619333+0000","last_active":"2024-03-21T14:26:25.897435+0000","last_peered":"2024-03-21T14:26:25.897435+0000","last_clean":"2024-03-21T14:26:25.897435+0000","last_became_active":"2024-03-21T14:25:06.187467+0000","last_became_peered":"2024-03-21T14:25:06.187467+0000","last_unstale":"2024-03-21T14:26:25.897435+0000","last_undegraded":"2024-03-21T14:26:25.897435+0000","last_fullsized":"2024-03-21T14:26:25.897435+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:42.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:42.619208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.155327+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"480'11","reported_seq":778,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776593+0000","last_change":"2024-03-21T14:25:50.890555+0000","last_active":"2024-03-21T14:26:28.776593+0000","last_peered":"2024-03-21T14:26:28.776593+0000","last_clean":"2024-03-21T14:26:28.776593+0000","last_became_active":"2024-03-21T14:25:33.858686+0000","last_became_peered":"2024-03-21T14:25:33.858686+0000","last_unstale":"2024-03-21T14:26:28.776593+0000","last_undegraded":"2024-03-21T14:26:28.776593+0000","last_fullsized":"2024-03-21T14:26:28.776593+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'11","last_scrub_stamp":"2024-03-21T14:25:50.890406+0000","last_deep_scrub":"480'11","last_deep_scrub_stamp":"2024-03-21T14:25:50.890406+0000","last_clean_scrub_stamp":"2024-03-21T14:25:50.890406+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:19.903307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002533+0000","last_change":"2024-03-21T14:25:15.956821+0000","last_active":"2024-03-21T14:26:26.002533+0000","last_peered":"2024-03-21T14:26:26.002533+0000","last_clean":"2024-03-21T14:26:26.002533+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:26:26.002533+0000","last_undegraded":"2024-03-21T14:26:26.002533+0000","last_fullsized":"2024-03-21T14:26:26.002533+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:15.956682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:15.956682+0000","last_clean_scrub_stamp":"2024-03-21T14:25:15.956682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:35.812116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'35","reported_seq":998,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002821+0000","last_change":"2024-03-21T14:26:19.540382+0000","last_active":"2024-03-21T14:26:26.002821+0000","last_peered":"2024-03-21T14:26:26.002821+0000","last_clean":"2024-03-21T14:26:26.002821+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:26:26.002821+0000","last_undegraded":"2024-03-21T14:26:26.002821+0000","last_fullsized":"2024-03-21T14:26:26.002821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:26:19.540295+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:26:19.540295+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:22.816245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":931,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896707+0000","last_change":"2024-03-21T14:26:11.504481+0000","last_active":"2024-03-21T14:26:25.896707+0000","last_peered":"2024-03-21T14:26:25.896707+0000","last_clean":"2024-03-21T14:26:25.896707+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:26:25.896707+0000","last_undegraded":"2024-03-21T14:26:25.896707+0000","last_fullsized":"2024-03-21T14:26:25.896707+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:11.504383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:26:11.504383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.680629+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":322,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776535+0000","last_change":"2024-03-21T14:26:03.501335+0000","last_active":"2024-03-21T14:26:28.776535+0000","last_peered":"2024-03-21T14:26:28.776535+0000","last_clean":"2024-03-21T14:26:28.776535+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:26:28.776535+0000","last_undegraded":"2024-03-21T14:26:28.776535+0000","last_fullsized":"2024-03-21T14:26:28.776535+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:26:03.501200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.501200+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:24.623319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":444,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896184+0000","last_change":"2024-03-21T14:25:24.874294+0000","last_active":"2024-03-21T14:26:25.896184+0000","last_peered":"2024-03-21T14:26:25.896184+0000","last_clean":"2024-03-21T14:26:25.896184+0000","last_became_active":"2024-03-21T14:25:06.179429+0000","last_became_peered":"2024-03-21T14:25:06.179429+0000","last_unstale":"2024-03-21T14:26:25.896184+0000","last_undegraded":"2024-03-21T14:26:25.896184+0000","last_fullsized":"2024-03-21T14:26:25.896184+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:24.874215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:24.874215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.610200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"441'25","reported_seq":775,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:34.619675+0000","last_change":"2024-03-21T14:26:34.619719+0000","last_active":"2024-03-21T14:26:34.619675+0000","last_peered":"2024-03-21T14:26:34.619675+0000","last_clean":"2024-03-21T14:26:34.619675+0000","last_became_active":"2024-03-21T14:25:33.857978+0000","last_became_peered":"2024-03-21T14:25:33.857978+0000","last_unstale":"2024-03-21T14:26:34.619675+0000","last_undegraded":"2024-03-21T14:26:34.619675+0000","last_fullsized":"2024-03-21T14:26:34.619675+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-21T14:26:34.619601+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:26:34.619601+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:44.889422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":789,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002365+0000","last_change":"2024-03-21T14:26:04.629299+0000","last_active":"2024-03-21T14:26:26.002365+0000","last_peered":"2024-03-21T14:26:26.002365+0000","last_clean":"2024-03-21T14:26:26.002365+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:26:26.002365+0000","last_undegraded":"2024-03-21T14:26:26.002365+0000","last_fullsized":"2024-03-21T14:26:26.002365+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:04.629212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:04.629212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.685194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":235,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003795+0000","last_change":"2024-03-21T14:25:57.481292+0000","last_active":"2024-03-21T14:26:26.003795+0000","last_peered":"2024-03-21T14:26:26.003795+0000","last_clean":"2024-03-21T14:26:26.003795+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:26:26.003795+0000","last_undegraded":"2024-03-21T14:26:26.003795+0000","last_fullsized":"2024-03-21T14:26:26.003795+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:57.481185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:57.481185+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.481185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.296347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":811,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002112+0000","last_change":"2024-03-21T14:25:40.954500+0000","last_active":"2024-03-21T14:26:26.002112+0000","last_peered":"2024-03-21T14:26:26.002112+0000","last_clean":"2024-03-21T14:26:26.002112+0000","last_became_active":"2024-03-21T14:25:40.881176+0000","last_became_peered":"2024-03-21T14:25:40.881176+0000","last_unstale":"2024-03-21T14:26:26.002112+0000","last_undegraded":"2024-03-21T14:26:26.002112+0000","last_fullsized":"2024-03-21T14:26:26.002112+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:40.954378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:40.954378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.553550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'25","reported_seq":610,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:32.270496+0000","last_change":"2024-03-21T14:26:32.270529+0000","last_active":"2024-03-21T14:26:32.270496+0000","last_peered":"2024-03-21T14:26:32.270496+0000","last_clean":"2024-03-21T14:26:32.270496+0000","last_became_active":"2024-03-21T14:25:06.191249+0000","last_became_peered":"2024-03-21T14:25:06.191249+0000","last_unstale":"2024-03-21T14:26:32.270496+0000","last_undegraded":"2024-03-21T14:26:32.270496+0000","last_fullsized":"2024-03-21T14:26:32.270496+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-21T14:26:32.270430+0000","last_deep_scrub":"432'19","last_deep_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_clean_scrub_stamp":"2024-03-21T14:26:32.270430+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:39.435787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":975,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002590+0000","last_change":"2024-03-21T14:25:44.822031+0000","last_active":"2024-03-21T14:26:26.002590+0000","last_peered":"2024-03-21T14:26:26.002590+0000","last_clean":"2024-03-21T14:26:26.002590+0000","last_became_active":"2024-03-21T14:25:44.821221+0000","last_became_peered":"2024-03-21T14:25:44.821221+0000","last_unstale":"2024-03-21T14:26:26.002590+0000","last_undegraded":"2024-03-21T14:26:26.002590+0000","last_fullsized":"2024-03-21T14:26:26.002590+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.065344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.065344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:59.177723+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":582,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004195+0000","last_change":"2024-03-21T14:26:25.639560+0000","last_active":"2024-03-21T14:26:26.004195+0000","last_peered":"2024-03-21T14:26:26.004195+0000","last_clean":"2024-03-21T14:26:26.004195+0000","last_became_active":"2024-03-21T14:25:33.853341+0000","last_became_peered":"2024-03-21T14:25:33.853341+0000","last_unstale":"2024-03-21T14:26:26.004195+0000","last_undegraded":"2024-03-21T14:26:26.004195+0000","last_fullsized":"2024-03-21T14:26:26.004195+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:25.639429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:07.301711+0000","last_clean_scrub_stamp":"2024-03-21T14:26:25.639429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:46.063037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":187,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002345+0000","last_change":"2024-03-21T14:25:21.880898+0000","last_active":"2024-03-21T14:26:26.002345+0000","last_peered":"2024-03-21T14:26:26.002345+0000","last_clean":"2024-03-21T14:26:26.002345+0000","last_became_active":"2024-03-21T14:21:26.765914+0000","last_became_peered":"2024-03-21T14:21:26.765914+0000","last_unstale":"2024-03-21T14:26:26.002345+0000","last_undegraded":"2024-03-21T14:26:26.002345+0000","last_fullsized":"2024-03-21T14:26:26.002345+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:21.880770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:21.880770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:33.461677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'18","reported_seq":252,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003545+0000","last_change":"2024-03-21T14:25:59.486475+0000","last_active":"2024-03-21T14:26:26.003545+0000","last_peered":"2024-03-21T14:26:26.003545+0000","last_clean":"2024-03-21T14:26:26.003545+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:26:26.003545+0000","last_undegraded":"2024-03-21T14:26:26.003545+0000","last_fullsized":"2024-03-21T14:26:26.003545+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'18","last_scrub_stamp":"2024-03-21T14:25:59.486351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:59.486351+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.064493+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":589,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002433+0000","last_change":"2024-03-21T14:25:44.823215+0000","last_active":"2024-03-21T14:26:26.002433+0000","last_peered":"2024-03-21T14:26:26.002433+0000","last_clean":"2024-03-21T14:26:26.002433+0000","last_became_active":"2024-03-21T14:25:44.822423+0000","last_became_peered":"2024-03-21T14:25:44.822423+0000","last_unstale":"2024-03-21T14:26:26.002433+0000","last_undegraded":"2024-03-21T14:26:26.002433+0000","last_fullsized":"2024-03-21T14:26:26.002433+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:28.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_clean_scrub_stamp":"2024-03-21T14:25:28.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:37.329389+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":1014,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:36.163317+0000","last_change":"2024-03-21T14:26:36.163345+0000","last_active":"2024-03-21T14:26:36.163317+0000","last_peered":"2024-03-21T14:26:36.163317+0000","last_clean":"2024-03-21T14:26:36.163317+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:26:36.163317+0000","last_undegraded":"2024-03-21T14:26:36.163317+0000","last_fullsized":"2024-03-21T14:26:36.163317+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:26:36.163244+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:25:14.914516+0000","last_clean_scrub_stamp":"2024-03-21T14:26:36.163244+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:41.864856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"486'46","reported_seq":857,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001766+0000","last_change":"2024-03-21T14:25:46.098239+0000","last_active":"2024-03-21T14:26:26.001766+0000","last_peered":"2024-03-21T14:26:26.001766+0000","last_clean":"2024-03-21T14:26:26.001766+0000","last_became_active":"2024-03-21T14:25:44.829654+0000","last_became_peered":"2024-03-21T14:25:44.829654+0000","last_unstale":"2024-03-21T14:26:26.001766+0000","last_undegraded":"2024-03-21T14:26:26.001766+0000","last_fullsized":"2024-03-21T14:26:26.001766+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'46","last_scrub_stamp":"2024-03-21T14:25:46.098114+0000","last_deep_scrub":"486'46","last_deep_scrub_stamp":"2024-03-21T14:25:46.098114+0000","last_clean_scrub_stamp":"2024-03-21T14:25:46.098114+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:58.383137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"488'11","reported_seq":1279,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:32.157571+0000","last_change":"2024-03-21T14:25:44.842237+0000","last_active":"2024-03-21T14:26:32.157571+0000","last_peered":"2024-03-21T14:26:32.157571+0000","last_clean":"2024-03-21T14:26:32.157571+0000","last_became_active":"2024-03-21T14:25:44.824707+0000","last_became_peered":"2024-03-21T14:25:44.824707+0000","last_unstale":"2024-03-21T14:26:32.157571+0000","last_undegraded":"2024-03-21T14:26:32.157571+0000","last_fullsized":"2024-03-21T14:26:32.157571+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:25:30.037228+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:25:30.037228+0000","last_clean_scrub_stamp":"2024-03-21T14:25:30.037228+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:56.824245+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":693,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003268+0000","last_change":"2024-03-21T14:25:55.625669+0000","last_active":"2024-03-21T14:26:26.003268+0000","last_peered":"2024-03-21T14:26:26.003268+0000","last_clean":"2024-03-21T14:26:26.003268+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:26:26.003268+0000","last_undegraded":"2024-03-21T14:26:26.003268+0000","last_fullsized":"2024-03-21T14:26:26.003268+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:55.625575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:55.625575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:24.833630+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":992,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003753+0000","last_change":"2024-03-21T14:26:12.891340+0000","last_active":"2024-03-21T14:26:26.003753+0000","last_peered":"2024-03-21T14:26:26.003753+0000","last_clean":"2024-03-21T14:26:26.003753+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:26:26.003753+0000","last_undegraded":"2024-03-21T14:26:26.003753+0000","last_fullsized":"2024-03-21T14:26:26.003753+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:12.891238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:12.891238+0000","last_clean_scrub_stamp":"2024-03-21T14:26:12.891238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:18.616398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"488'29","reported_seq":1040,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895964+0000","last_change":"2024-03-21T14:25:50.493067+0000","last_active":"2024-03-21T14:26:25.895964+0000","last_peered":"2024-03-21T14:26:25.895964+0000","last_clean":"2024-03-21T14:26:25.895964+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:26:25.895964+0000","last_undegraded":"2024-03-21T14:26:25.895964+0000","last_fullsized":"2024-03-21T14:26:25.895964+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'29","last_scrub_stamp":"2024-03-21T14:25:50.492941+0000","last_deep_scrub":"488'29","last_deep_scrub_stamp":"2024-03-21T14:25:50.492941+0000","last_clean_scrub_stamp":"2024-03-21T14:25:50.492941+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:59.184709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":801,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001408+0000","last_change":"2024-03-21T14:26:08.620478+0000","last_active":"2024-03-21T14:26:26.001408+0000","last_peered":"2024-03-21T14:26:26.001408+0000","last_clean":"2024-03-21T14:26:26.001408+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:26:26.001408+0000","last_undegraded":"2024-03-21T14:26:26.001408+0000","last_fullsized":"2024-03-21T14:26:26.001408+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:08.620360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:26:08.620360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:13.936929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":573,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001671+0000","last_change":"2024-03-21T14:25:41.133611+0000","last_active":"2024-03-21T14:26:26.001671+0000","last_peered":"2024-03-21T14:26:26.001671+0000","last_clean":"2024-03-21T14:26:26.001671+0000","last_became_active":"2024-03-21T14:25:41.133266+0000","last_became_peered":"2024-03-21T14:25:41.133266+0000","last_unstale":"2024-03-21T14:26:26.001671+0000","last_undegraded":"2024-03-21T14:26:26.001671+0000","last_fullsized":"2024-03-21T14:26:26.001671+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:15.472742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:15.472742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:38.272345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":754,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001976+0000","last_change":"2024-03-21T14:25:24.473276+0000","last_active":"2024-03-21T14:26:26.001976+0000","last_peered":"2024-03-21T14:26:26.001976+0000","last_clean":"2024-03-21T14:26:26.001976+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:26:26.001976+0000","last_undegraded":"2024-03-21T14:26:26.001976+0000","last_fullsized":"2024-03-21T14:26:26.001976+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:24.473195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:24.473195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:38.561497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":604,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003693+0000","last_change":"2024-03-21T14:25:45.950388+0000","last_active":"2024-03-21T14:26:26.003693+0000","last_peered":"2024-03-21T14:26:26.003693+0000","last_clean":"2024-03-21T14:26:26.003693+0000","last_became_active":"2024-03-21T14:25:40.881274+0000","last_became_peered":"2024-03-21T14:25:40.881274+0000","last_unstale":"2024-03-21T14:26:26.003693+0000","last_undegraded":"2024-03-21T14:26:26.003693+0000","last_fullsized":"2024-03-21T14:26:26.003693+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:45.950289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:45.950289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:05.031215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'60","reported_seq":51,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003181+0000","last_change":"2024-03-21T14:25:44.850897+0000","last_active":"2024-03-21T14:26:26.003181+0000","last_peered":"2024-03-21T14:26:26.003181+0000","last_clean":"2024-03-21T14:26:26.003181+0000","last_became_active":"2024-03-21T14:25:44.828035+0000","last_became_peered":"2024-03-21T14:25:44.828035+0000","last_unstale":"2024-03-21T14:26:26.003181+0000","last_undegraded":"2024-03-21T14:26:26.003181+0000","last_fullsized":"2024-03-21T14:26:26.003181+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'60","last_scrub_stamp":"2024-03-21T14:25:24.692200+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:25:24.692200+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:33.024726+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":95,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:34.416087+0000","last_change":"2024-03-21T14:26:34.416111+0000","last_active":"2024-03-21T14:26:34.416087+0000","last_peered":"2024-03-21T14:26:34.416087+0000","last_clean":"2024-03-21T14:26:34.416087+0000","last_became_active":"2024-03-21T14:25:06.182766+0000","last_became_peered":"2024-03-21T14:25:06.182766+0000","last_unstale":"2024-03-21T14:26:34.416087+0000","last_undegraded":"2024-03-21T14:26:34.416087+0000","last_fullsized":"2024-03-21T14:26:34.416087+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:34.416008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:34.416008+0000","last_clean_scrub_stamp":"2024-03-21T14:26:34.416008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:50.636577+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":801,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001808+0000","last_change":"2024-03-21T14:26:13.569734+0000","last_active":"2024-03-21T14:26:26.001808+0000","last_peered":"2024-03-21T14:26:26.001808+0000","last_clean":"2024-03-21T14:26:26.001808+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:26:26.001808+0000","last_undegraded":"2024-03-21T14:26:26.001808+0000","last_fullsized":"2024-03-21T14:26:26.001808+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:13.569654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:26:13.569654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:43.442676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":70,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896406+0000","last_change":"2024-03-21T14:25:53.649028+0000","last_active":"2024-03-21T14:26:25.896406+0000","last_peered":"2024-03-21T14:26:25.896406+0000","last_clean":"2024-03-21T14:26:25.896406+0000","last_became_active":"2024-03-21T14:25:06.179804+0000","last_became_peered":"2024-03-21T14:25:06.179804+0000","last_unstale":"2024-03-21T14:26:25.896406+0000","last_undegraded":"2024-03-21T14:26:25.896406+0000","last_fullsized":"2024-03-21T14:26:25.896406+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:53.648939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:53.648939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:04.581288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"441'21","reported_seq":405,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897789+0000","last_change":"2024-03-21T14:25:47.531439+0000","last_active":"2024-03-21T14:26:25.897789+0000","last_peered":"2024-03-21T14:26:25.897789+0000","last_clean":"2024-03-21T14:26:25.897789+0000","last_became_active":"2024-03-21T14:25:06.185191+0000","last_became_peered":"2024-03-21T14:25:06.185191+0000","last_unstale":"2024-03-21T14:26:25.897789+0000","last_undegraded":"2024-03-21T14:26:25.897789+0000","last_fullsized":"2024-03-21T14:26:25.897789+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:25:47.531335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:47.531335+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:17.385156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":756,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001746+0000","last_change":"2024-03-21T14:26:24.687482+0000","last_active":"2024-03-21T14:26:26.001746+0000","last_peered":"2024-03-21T14:26:26.001746+0000","last_clean":"2024-03-21T14:26:26.001746+0000","last_became_active":"2024-03-21T14:25:33.853161+0000","last_became_peered":"2024-03-21T14:25:33.853161+0000","last_unstale":"2024-03-21T14:26:26.001746+0000","last_undegraded":"2024-03-21T14:26:26.001746+0000","last_fullsized":"2024-03-21T14:26:26.001746+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:24.687341+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:24.687341+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:48.649683+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":941,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003646+0000","last_change":"2024-03-21T14:26:15.627527+0000","last_active":"2024-03-21T14:26:26.003646+0000","last_peered":"2024-03-21T14:26:26.003646+0000","last_clean":"2024-03-21T14:26:26.003646+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:26:26.003646+0000","last_undegraded":"2024-03-21T14:26:26.003646+0000","last_fullsized":"2024-03-21T14:26:26.003646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:15.627438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:15.627438+0000","last_clean_scrub_stamp":"2024-03-21T14:26:15.627438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:45.401433+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":614,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002238+0000","last_change":"2024-03-21T14:25:44.842873+0000","last_active":"2024-03-21T14:26:26.002238+0000","last_peered":"2024-03-21T14:26:26.002238+0000","last_clean":"2024-03-21T14:26:26.002238+0000","last_became_active":"2024-03-21T14:25:44.822898+0000","last_became_peered":"2024-03-21T14:25:44.822898+0000","last_unstale":"2024-03-21T14:26:26.002238+0000","last_undegraded":"2024-03-21T14:26:26.002238+0000","last_fullsized":"2024-03-21T14:26:26.002238+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:25:06.655436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:25:06.655436+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:34.775941+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":956,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:36.345745+0000","last_change":"2024-03-21T14:26:36.345770+0000","last_active":"2024-03-21T14:26:36.345745+0000","last_peered":"2024-03-21T14:26:36.345745+0000","last_clean":"2024-03-21T14:26:36.345745+0000","last_became_active":"2024-03-21T14:25:44.819631+0000","last_became_peered":"2024-03-21T14:25:44.819631+0000","last_unstale":"2024-03-21T14:26:36.345745+0000","last_undegraded":"2024-03-21T14:26:36.345745+0000","last_fullsized":"2024-03-21T14:26:36.345745+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:36.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:14.635743+0000","last_clean_scrub_stamp":"2024-03-21T14:26:36.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:43.092044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"480'67","reported_seq":833,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776611+0000","last_change":"2024-03-21T14:25:33.855721+0000","last_active":"2024-03-21T14:26:28.776611+0000","last_peered":"2024-03-21T14:26:28.776611+0000","last_clean":"2024-03-21T14:26:28.776611+0000","last_became_active":"2024-03-21T14:25:33.853554+0000","last_became_peered":"2024-03-21T14:25:33.853554+0000","last_unstale":"2024-03-21T14:26:28.776611+0000","last_undegraded":"2024-03-21T14:26:28.776611+0000","last_fullsized":"2024-03-21T14:26:28.776611+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'66","last_scrub_stamp":"2024-03-21T14:25:09.456754+0000","last_deep_scrub":"128'39","last_deep_scrub_stamp":"2024-03-21T14:10:03.405289+0000","last_clean_scrub_stamp":"2024-03-21T14:25:09.456754+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":600,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896781+0000","last_change":"2024-03-21T14:26:03.590862+0000","last_active":"2024-03-21T14:26:25.896781+0000","last_peered":"2024-03-21T14:26:25.896781+0000","last_clean":"2024-03-21T14:26:25.896781+0000","last_became_active":"2024-03-21T14:25:06.179299+0000","last_became_peered":"2024-03-21T14:25:06.179299+0000","last_unstale":"2024-03-21T14:26:25.896781+0000","last_undegraded":"2024-03-21T14:26:25.896781+0000","last_fullsized":"2024-03-21T14:26:25.896781+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:03.590757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:03.590757+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.590757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:28.253491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":901,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:34.172741+0000","last_change":"2024-03-21T14:26:34.172768+0000","last_active":"2024-03-21T14:26:34.172741+0000","last_peered":"2024-03-21T14:26:34.172741+0000","last_clean":"2024-03-21T14:26:34.172741+0000","last_became_active":"2024-03-21T14:25:40.881368+0000","last_became_peered":"2024-03-21T14:25:40.881368+0000","last_unstale":"2024-03-21T14:26:34.172741+0000","last_undegraded":"2024-03-21T14:26:34.172741+0000","last_fullsized":"2024-03-21T14:26:34.172741+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:34.172654+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:34.172654+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:49.615631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003873+0000","last_change":"2024-03-21T14:25:57.129592+0000","last_active":"2024-03-21T14:26:26.003873+0000","last_peered":"2024-03-21T14:26:26.003873+0000","last_clean":"2024-03-21T14:26:26.003873+0000","last_became_active":"2024-03-21T14:25:44.822674+0000","last_became_peered":"2024-03-21T14:25:44.822674+0000","last_unstale":"2024-03-21T14:26:26.003873+0000","last_undegraded":"2024-03-21T14:26:26.003873+0000","last_fullsized":"2024-03-21T14:26:26.003873+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:57.129466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.129466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:57.817089+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":975,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895222+0000","last_change":"2024-03-21T14:26:14.513221+0000","last_active":"2024-03-21T14:26:25.895222+0000","last_peered":"2024-03-21T14:26:25.895222+0000","last_clean":"2024-03-21T14:26:25.895222+0000","last_became_active":"2024-03-21T14:25:44.823980+0000","last_became_peered":"2024-03-21T14:25:44.823980+0000","last_unstale":"2024-03-21T14:26:25.895222+0000","last_undegraded":"2024-03-21T14:26:25.895222+0000","last_fullsized":"2024-03-21T14:26:25.895222+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:14.513144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:14.513144+0000","last_clean_scrub_stamp":"2024-03-21T14:26:14.513144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.322255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":804,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:32.630490+0000","last_change":"2024-03-21T14:26:32.630513+0000","last_active":"2024-03-21T14:26:32.630490+0000","last_peered":"2024-03-21T14:26:32.630490+0000","last_clean":"2024-03-21T14:26:32.630490+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:26:32.630490+0000","last_undegraded":"2024-03-21T14:26:32.630490+0000","last_fullsized":"2024-03-21T14:26:32.630490+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:32.630425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:29.414864+0000","last_clean_scrub_stamp":"2024-03-21T14:26:32.630425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:42.132480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":1053,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897156+0000","last_change":"2024-03-21T14:25:51.672227+0000","last_active":"2024-03-21T14:26:25.897156+0000","last_peered":"2024-03-21T14:26:25.897156+0000","last_clean":"2024-03-21T14:26:25.897156+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:26:25.897156+0000","last_undegraded":"2024-03-21T14:26:25.897156+0000","last_fullsized":"2024-03-21T14:26:25.897156+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-21T14:25:51.672111+0000","last_deep_scrub":"236'22","last_deep_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_clean_scrub_stamp":"2024-03-21T14:25:51.672111+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:18.811171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":779,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002344+0000","last_change":"2024-03-21T14:26:06.660489+0000","last_active":"2024-03-21T14:26:26.002344+0000","last_peered":"2024-03-21T14:26:26.002344+0000","last_clean":"2024-03-21T14:26:26.002344+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:26:26.002344+0000","last_undegraded":"2024-03-21T14:26:26.002344+0000","last_fullsized":"2024-03-21T14:26:26.002344+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:06.660395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:49.614290+0000","last_clean_scrub_stamp":"2024-03-21T14:26:06.660395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:15.205994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":571,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:29.779088+0000","last_change":"2024-03-21T14:26:29.779112+0000","last_active":"2024-03-21T14:26:29.779088+0000","last_peered":"2024-03-21T14:26:29.779088+0000","last_clean":"2024-03-21T14:26:29.779088+0000","last_became_active":"2024-03-21T14:25:33.853793+0000","last_became_peered":"2024-03-21T14:25:33.853793+0000","last_unstale":"2024-03-21T14:26:29.779088+0000","last_undegraded":"2024-03-21T14:26:29.779088+0000","last_fullsized":"2024-03-21T14:26:29.779088+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:29.778995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:29.778995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:42.549764+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1285,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001852+0000","last_change":"2024-03-21T14:25:47.821305+0000","last_active":"2024-03-21T14:26:26.001852+0000","last_peered":"2024-03-21T14:26:26.001852+0000","last_clean":"2024-03-21T14:26:26.001852+0000","last_became_active":"2024-03-21T14:25:47.820896+0000","last_became_peered":"2024-03-21T14:25:47.820896+0000","last_unstale":"2024-03-21T14:26:26.001852+0000","last_undegraded":"2024-03-21T14:26:26.001852+0000","last_fullsized":"2024-03-21T14:26:26.001852+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:22.065873+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:22.065873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.877713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":973,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002462+0000","last_change":"2024-03-21T14:25:44.831295+0000","last_active":"2024-03-21T14:26:26.002462+0000","last_peered":"2024-03-21T14:26:26.002462+0000","last_clean":"2024-03-21T14:26:26.002462+0000","last_became_active":"2024-03-21T14:25:44.830942+0000","last_became_peered":"2024-03-21T14:25:44.830942+0000","last_unstale":"2024-03-21T14:26:26.002462+0000","last_undegraded":"2024-03-21T14:26:26.002462+0000","last_fullsized":"2024-03-21T14:26:26.002462+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.320710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.320710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:55.779155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"488'42","reported_seq":59,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.777011+0000","last_change":"2024-03-21T14:25:44.850008+0000","last_active":"2024-03-21T14:26:28.777011+0000","last_peered":"2024-03-21T14:26:28.777011+0000","last_clean":"2024-03-21T14:26:28.777011+0000","last_became_active":"2024-03-21T14:25:44.829253+0000","last_became_peered":"2024-03-21T14:25:44.829253+0000","last_unstale":"2024-03-21T14:26:28.777011+0000","last_undegraded":"2024-03-21T14:26:28.777011+0000","last_fullsized":"2024-03-21T14:26:28.777011+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'41","last_scrub_stamp":"2024-03-21T14:25:33.551480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:33.551480+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:53.122097+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":671,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003171+0000","last_change":"2024-03-21T14:25:40.882008+0000","last_active":"2024-03-21T14:26:26.003171+0000","last_peered":"2024-03-21T14:26:26.003171+0000","last_clean":"2024-03-21T14:26:26.003171+0000","last_became_active":"2024-03-21T14:25:40.881519+0000","last_became_peered":"2024-03-21T14:25:40.881519+0000","last_unstale":"2024-03-21T14:26:26.003171+0000","last_undegraded":"2024-03-21T14:26:26.003171+0000","last_fullsized":"2024-03-21T14:26:26.003171+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:19.476852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:19.476852+0000","last_clean_scrub_stamp":"2024-03-21T14:25:19.476852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:40.161115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":1040,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895876+0000","last_change":"2024-03-21T14:26:09.521260+0000","last_active":"2024-03-21T14:26:25.895876+0000","last_peered":"2024-03-21T14:26:25.895876+0000","last_clean":"2024-03-21T14:26:25.895876+0000","last_became_active":"2024-03-21T14:25:47.816308+0000","last_became_peered":"2024-03-21T14:25:47.816308+0000","last_unstale":"2024-03-21T14:26:25.895876+0000","last_undegraded":"2024-03-21T14:26:25.895876+0000","last_fullsized":"2024-03-21T14:26:25.895876+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:09.521162+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:09.521162+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:33.929416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":967,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896246+0000","last_change":"2024-03-21T14:25:35.587679+0000","last_active":"2024-03-21T14:26:25.896246+0000","last_peered":"2024-03-21T14:26:25.896246+0000","last_clean":"2024-03-21T14:26:25.896246+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:26:25.896246+0000","last_undegraded":"2024-03-21T14:26:25.896246+0000","last_fullsized":"2024-03-21T14:26:25.896246+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:35.587577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:25:35.587577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:52.412718+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":594,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:33.235606+0000","last_change":"2024-03-21T14:26:33.235630+0000","last_active":"2024-03-21T14:26:33.235606+0000","last_peered":"2024-03-21T14:26:33.235606+0000","last_clean":"2024-03-21T14:26:33.235606+0000","last_became_active":"2024-03-21T14:25:06.179126+0000","last_became_peered":"2024-03-21T14:25:06.179126+0000","last_unstale":"2024-03-21T14:26:33.235606+0000","last_undegraded":"2024-03-21T14:26:33.235606+0000","last_fullsized":"2024-03-21T14:26:33.235606+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:33.235531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:33.235531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:43.277062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1501,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:35.967431+0000","last_change":"2024-03-21T14:26:07.020323+0000","last_active":"2024-03-21T14:26:35.967431+0000","last_peered":"2024-03-21T14:26:35.967431+0000","last_clean":"2024-03-21T14:26:35.967431+0000","last_became_active":"2024-03-21T14:25:40.879713+0000","last_became_peered":"2024-03-21T14:25:40.879713+0000","last_unstale":"2024-03-21T14:26:35.967431+0000","last_undegraded":"2024-03-21T14:26:35.967431+0000","last_fullsized":"2024-03-21T14:26:35.967431+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:26:07.020164+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:07.020164+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:25.549813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":888,"num_read_kb":883,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":854,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897315+0000","last_change":"2024-03-21T14:26:16.500849+0000","last_active":"2024-03-21T14:26:25.897315+0000","last_peered":"2024-03-21T14:26:25.897315+0000","last_clean":"2024-03-21T14:26:25.897315+0000","last_became_active":"2024-03-21T14:25:06.185735+0000","last_became_peered":"2024-03-21T14:25:06.185735+0000","last_unstale":"2024-03-21T14:26:25.897315+0000","last_undegraded":"2024-03-21T14:26:25.897315+0000","last_fullsized":"2024-03-21T14:26:25.897315+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-21T14:26:16.500753+0000","last_deep_scrub":"441'36","last_deep_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_clean_scrub_stamp":"2024-03-21T14:26:16.500753+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.770255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":583,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004764+0000","last_change":"2024-03-21T14:25:40.151087+0000","last_active":"2024-03-21T14:26:26.004764+0000","last_peered":"2024-03-21T14:26:26.004764+0000","last_clean":"2024-03-21T14:26:26.004764+0000","last_became_active":"2024-03-21T14:25:33.852775+0000","last_became_peered":"2024-03-21T14:25:33.852775+0000","last_unstale":"2024-03-21T14:26:26.004764+0000","last_undegraded":"2024-03-21T14:26:26.004764+0000","last_fullsized":"2024-03-21T14:26:26.004764+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:40.150980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:25:40.150980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:47.587288+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":604,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003935+0000","last_change":"2024-03-21T14:26:21.630400+0000","last_active":"2024-03-21T14:26:26.003935+0000","last_peered":"2024-03-21T14:26:26.003935+0000","last_clean":"2024-03-21T14:26:26.003935+0000","last_became_active":"2024-03-21T14:25:47.819834+0000","last_became_peered":"2024-03-21T14:25:47.819834+0000","last_unstale":"2024-03-21T14:26:26.003935+0000","last_undegraded":"2024-03-21T14:26:26.003935+0000","last_fullsized":"2024-03-21T14:26:26.003935+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:21.630293+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:26:21.630293+0000","last_clean_scrub_stamp":"2024-03-21T14:26:21.630293+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.490949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895520+0000","last_change":"2024-03-21T14:25:44.820606+0000","last_active":"2024-03-21T14:26:25.895520+0000","last_peered":"2024-03-21T14:26:25.895520+0000","last_clean":"2024-03-21T14:26:25.895520+0000","last_became_active":"2024-03-21T14:25:44.820252+0000","last_became_peered":"2024-03-21T14:25:44.820252+0000","last_unstale":"2024-03-21T14:26:25.895520+0000","last_undegraded":"2024-03-21T14:26:25.895520+0000","last_fullsized":"2024-03-21T14:26:25.895520+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.610372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:05.738523+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.610372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:51.054393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896470+0000","last_change":"2024-03-21T14:26:23.540870+0000","last_active":"2024-03-21T14:26:25.896470+0000","last_peered":"2024-03-21T14:26:25.896470+0000","last_clean":"2024-03-21T14:26:25.896470+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:26:25.896470+0000","last_undegraded":"2024-03-21T14:26:25.896470+0000","last_fullsized":"2024-03-21T14:26:25.896470+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:23.540724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:23.540724+0000","last_clean_scrub_stamp":"2024-03-21T14:26:23.540724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.249226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"485'47","reported_seq":654,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897314+0000","last_change":"2024-03-21T14:25:45.611105+0000","last_active":"2024-03-21T14:26:25.897314+0000","last_peered":"2024-03-21T14:26:25.897314+0000","last_clean":"2024-03-21T14:26:25.897314+0000","last_became_active":"2024-03-21T14:25:06.190350+0000","last_became_peered":"2024-03-21T14:25:06.190350+0000","last_unstale":"2024-03-21T14:26:25.897314+0000","last_undegraded":"2024-03-21T14:26:25.897314+0000","last_fullsized":"2024-03-21T14:26:25.897314+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'47","last_scrub_stamp":"2024-03-21T14:25:45.611005+0000","last_deep_scrub":"485'47","last_deep_scrub_stamp":"2024-03-21T14:25:45.611005+0000","last_clean_scrub_stamp":"2024-03-21T14:25:45.611005+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:58.728839+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":31,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003076+0000","last_change":"2024-03-21T14:25:44.833182+0000","last_active":"2024-03-21T14:26:26.003076+0000","last_peered":"2024-03-21T14:26:26.003076+0000","last_clean":"2024-03-21T14:26:26.003076+0000","last_became_active":"2024-03-21T14:25:44.832497+0000","last_became_peered":"2024-03-21T14:25:44.832497+0000","last_unstale":"2024-03-21T14:26:26.003076+0000","last_undegraded":"2024-03-21T14:26:26.003076+0000","last_fullsized":"2024-03-21T14:26:26.003076+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:34.759171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:25:34.759171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.617355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":936,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003837+0000","last_change":"2024-03-21T14:25:49.680443+0000","last_active":"2024-03-21T14:26:26.003837+0000","last_peered":"2024-03-21T14:26:26.003837+0000","last_clean":"2024-03-21T14:26:26.003837+0000","last_became_active":"2024-03-21T14:25:47.819406+0000","last_became_peered":"2024-03-21T14:25:47.819406+0000","last_unstale":"2024-03-21T14:26:26.003837+0000","last_undegraded":"2024-03-21T14:26:26.003837+0000","last_fullsized":"2024-03-21T14:26:26.003837+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:49.680292+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:49.680292+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:08.330411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":432,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895800+0000","last_change":"2024-03-21T14:25:38.717994+0000","last_active":"2024-03-21T14:26:25.895800+0000","last_peered":"2024-03-21T14:26:25.895800+0000","last_clean":"2024-03-21T14:26:25.895800+0000","last_became_active":"2024-03-21T14:25:06.181080+0000","last_became_peered":"2024-03-21T14:25:06.181080+0000","last_unstale":"2024-03-21T14:26:25.895800+0000","last_undegraded":"2024-03-21T14:26:25.895800+0000","last_fullsized":"2024-03-21T14:26:25.895800+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.717867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.717867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:08.476649+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":634,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004494+0000","last_change":"2024-03-21T14:25:43.156304+0000","last_active":"2024-03-21T14:26:26.004494+0000","last_peered":"2024-03-21T14:26:26.004494+0000","last_clean":"2024-03-21T14:26:26.004494+0000","last_became_active":"2024-03-21T14:25:33.853651+0000","last_became_peered":"2024-03-21T14:25:33.853651+0000","last_unstale":"2024-03-21T14:26:26.004494+0000","last_undegraded":"2024-03-21T14:26:26.004494+0000","last_fullsized":"2024-03-21T14:26:26.004494+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:43.156192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:43.156192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:07.984041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":1015,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896011+0000","last_change":"2024-03-21T14:26:00.607520+0000","last_active":"2024-03-21T14:26:25.896011+0000","last_peered":"2024-03-21T14:26:25.896011+0000","last_clean":"2024-03-21T14:26:25.896011+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:26:25.896011+0000","last_undegraded":"2024-03-21T14:26:25.896011+0000","last_fullsized":"2024-03-21T14:26:25.896011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:26:00.607405+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:26:00.607405+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:25.304382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"488'14","reported_seq":595,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776450+0000","last_change":"2024-03-21T14:25:44.845004+0000","last_active":"2024-03-21T14:26:28.776450+0000","last_peered":"2024-03-21T14:26:28.776450+0000","last_clean":"2024-03-21T14:26:28.776450+0000","last_became_active":"2024-03-21T14:25:44.828780+0000","last_became_peered":"2024-03-21T14:25:44.828780+0000","last_unstale":"2024-03-21T14:26:28.776450+0000","last_undegraded":"2024-03-21T14:26:28.776450+0000","last_fullsized":"2024-03-21T14:26:28.776450+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'13","last_scrub_stamp":"2024-03-21T14:25:33.790676+0000","last_deep_scrub":"479'13","last_deep_scrub_stamp":"2024-03-21T14:25:33.790676+0000","last_clean_scrub_stamp":"2024-03-21T14:25:33.790676+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.957223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":1015,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002622+0000","last_change":"2024-03-21T14:26:05.476597+0000","last_active":"2024-03-21T14:26:26.002622+0000","last_peered":"2024-03-21T14:26:26.002622+0000","last_clean":"2024-03-21T14:26:26.002622+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:26:26.002622+0000","last_undegraded":"2024-03-21T14:26:26.002622+0000","last_fullsized":"2024-03-21T14:26:26.002622+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:05.476458+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:05.476458+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.292381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003002+0000","last_change":"2024-03-21T14:25:55.109936+0000","last_active":"2024-03-21T14:26:26.003002+0000","last_peered":"2024-03-21T14:26:26.003002+0000","last_clean":"2024-03-21T14:26:26.003002+0000","last_became_active":"2024-03-21T14:25:44.820219+0000","last_became_peered":"2024-03-21T14:25:44.820219+0000","last_unstale":"2024-03-21T14:26:26.003002+0000","last_undegraded":"2024-03-21T14:26:26.003002+0000","last_fullsized":"2024-03-21T14:26:26.003002+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:55.109799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:55.109799+0000","last_clean_scrub_stamp":"2024-03-21T14:25:55.109799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:15.368434+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":598,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897292+0000","last_change":"2024-03-21T14:25:57.609618+0000","last_active":"2024-03-21T14:26:25.897292+0000","last_peered":"2024-03-21T14:26:25.897292+0000","last_clean":"2024-03-21T14:26:25.897292+0000","last_became_active":"2024-03-21T14:25:06.185624+0000","last_became_peered":"2024-03-21T14:25:06.185624+0000","last_unstale":"2024-03-21T14:26:25.897292+0000","last_undegraded":"2024-03-21T14:26:25.897292+0000","last_fullsized":"2024-03-21T14:26:25.897292+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:25:57.609485+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.609485+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:27.358450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'42","reported_seq":1047,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895773+0000","last_change":"2024-03-21T14:25:18.618565+0000","last_active":"2024-03-21T14:26:25.895773+0000","last_peered":"2024-03-21T14:26:25.895773+0000","last_clean":"2024-03-21T14:26:25.895773+0000","last_became_active":"2024-03-21T14:21:26.768622+0000","last_became_peered":"2024-03-21T14:21:26.768622+0000","last_unstale":"2024-03-21T14:26:25.895773+0000","last_undegraded":"2024-03-21T14:26:25.895773+0000","last_fullsized":"2024-03-21T14:26:25.895773+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-21T14:25:18.618422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:18.618422+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:36.938948+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":1242,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776786+0000","last_change":"2024-03-21T14:25:32.205688+0000","last_active":"2024-03-21T14:26:28.776786+0000","last_peered":"2024-03-21T14:26:28.776786+0000","last_clean":"2024-03-21T14:26:28.776786+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:26:28.776786+0000","last_undegraded":"2024-03-21T14:26:28.776786+0000","last_fullsized":"2024-03-21T14:26:28.776786+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:25:32.205594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:32.205594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:45.179180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":1026,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896444+0000","last_change":"2024-03-21T14:26:03.442579+0000","last_active":"2024-03-21T14:26:25.896444+0000","last_peered":"2024-03-21T14:26:25.896444+0000","last_clean":"2024-03-21T14:26:25.896444+0000","last_became_active":"2024-03-21T14:25:47.815743+0000","last_became_peered":"2024-03-21T14:25:47.815743+0000","last_unstale":"2024-03-21T14:26:25.896444+0000","last_undegraded":"2024-03-21T14:26:25.896444+0000","last_fullsized":"2024-03-21T14:26:25.896444+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:03.442461+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.442461+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.094457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":956,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895883+0000","last_change":"2024-03-21T14:25:10.802241+0000","last_active":"2024-03-21T14:26:25.895883+0000","last_peered":"2024-03-21T14:26:25.895883+0000","last_clean":"2024-03-21T14:26:25.895883+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:26:25.895883+0000","last_undegraded":"2024-03-21T14:26:25.895883+0000","last_fullsized":"2024-03-21T14:26:25.895883+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:10.802145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:25:10.802145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:38.924494+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":953,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003734+0000","last_change":"2024-03-21T14:25:17.236560+0000","last_active":"2024-03-21T14:26:26.003734+0000","last_peered":"2024-03-21T14:26:26.003734+0000","last_clean":"2024-03-21T14:26:26.003734+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:26:26.003734+0000","last_undegraded":"2024-03-21T14:26:26.003734+0000","last_fullsized":"2024-03-21T14:26:26.003734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:17.236474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:25:17.236474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:44.448612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'50","reported_seq":272,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001996+0000","last_change":"2024-03-21T14:25:44.320136+0000","last_active":"2024-03-21T14:26:26.001996+0000","last_peered":"2024-03-21T14:26:26.001996+0000","last_clean":"2024-03-21T14:26:26.001996+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:26:26.001996+0000","last_undegraded":"2024-03-21T14:26:26.001996+0000","last_fullsized":"2024-03-21T14:26:26.001996+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'50","last_scrub_stamp":"2024-03-21T14:25:44.320004+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:25:44.320004+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:46.047233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":928,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895617+0000","last_change":"2024-03-21T14:25:27.525688+0000","last_active":"2024-03-21T14:26:25.895617+0000","last_peered":"2024-03-21T14:26:25.895617+0000","last_clean":"2024-03-21T14:26:25.895617+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:26:25.895617+0000","last_undegraded":"2024-03-21T14:26:25.895617+0000","last_fullsized":"2024-03-21T14:26:25.895617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:27.525551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:27.525551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:54.393229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":650,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001752+0000","last_change":"2024-03-21T14:25:47.816918+0000","last_active":"2024-03-21T14:26:26.001752+0000","last_peered":"2024-03-21T14:26:26.001752+0000","last_clean":"2024-03-21T14:26:26.001752+0000","last_became_active":"2024-03-21T14:25:47.815317+0000","last_became_peered":"2024-03-21T14:25:47.815317+0000","last_unstale":"2024-03-21T14:26:26.001752+0000","last_undegraded":"2024-03-21T14:26:26.001752+0000","last_fullsized":"2024-03-21T14:26:26.001752+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:32.486240+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:32.486240+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:55.152806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":717,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004195+0000","last_change":"2024-03-21T14:25:33.856569+0000","last_active":"2024-03-21T14:26:26.004195+0000","last_peered":"2024-03-21T14:26:26.004195+0000","last_clean":"2024-03-21T14:26:26.004195+0000","last_became_active":"2024-03-21T14:25:33.854431+0000","last_became_peered":"2024-03-21T14:25:33.854431+0000","last_unstale":"2024-03-21T14:26:26.004195+0000","last_undegraded":"2024-03-21T14:26:26.004195+0000","last_fullsized":"2024-03-21T14:26:26.004195+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:27.839318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:25:27.839318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:47.601257+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1232,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:28.776618+0000","last_change":"2024-03-21T14:25:12.729483+0000","last_active":"2024-03-21T14:26:28.776618+0000","last_peered":"2024-03-21T14:26:28.776618+0000","last_clean":"2024-03-21T14:26:28.776618+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:26:28.776618+0000","last_undegraded":"2024-03-21T14:26:28.776618+0000","last_fullsized":"2024-03-21T14:26:28.776618+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:25:12.729381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:12.729381+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:38.562359+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":235,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002465+0000","last_change":"2024-03-21T14:25:25.214748+0000","last_active":"2024-03-21T14:26:26.002465+0000","last_peered":"2024-03-21T14:26:26.002465+0000","last_clean":"2024-03-21T14:26:26.002465+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:26:26.002465+0000","last_undegraded":"2024-03-21T14:26:26.002465+0000","last_fullsized":"2024-03-21T14:26:26.002465+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:25.214650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:25.214650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:48.181660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'39","reported_seq":1039,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897217+0000","last_change":"2024-03-21T14:25:20.682193+0000","last_active":"2024-03-21T14:26:25.897217+0000","last_peered":"2024-03-21T14:26:25.897217+0000","last_clean":"2024-03-21T14:26:25.897217+0000","last_became_active":"2024-03-21T14:21:19.713483+0000","last_became_peered":"2024-03-21T14:21:19.713483+0000","last_unstale":"2024-03-21T14:26:25.897217+0000","last_undegraded":"2024-03-21T14:26:25.897217+0000","last_fullsized":"2024-03-21T14:26:25.897217+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-21T14:25:20.682101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:20.682101+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:39.338545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":952,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002725+0000","last_change":"2024-03-21T14:25:48.350718+0000","last_active":"2024-03-21T14:26:26.002725+0000","last_peered":"2024-03-21T14:26:26.002725+0000","last_clean":"2024-03-21T14:26:26.002725+0000","last_became_active":"2024-03-21T14:25:44.835088+0000","last_became_peered":"2024-03-21T14:25:44.835088+0000","last_unstale":"2024-03-21T14:26:26.002725+0000","last_undegraded":"2024-03-21T14:26:26.002725+0000","last_fullsized":"2024-03-21T14:26:26.002725+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:48.350617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:48.350617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.456467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":817,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.004681+0000","last_change":"2024-03-21T14:26:02.856790+0000","last_active":"2024-03-21T14:26:26.004681+0000","last_peered":"2024-03-21T14:26:26.004681+0000","last_clean":"2024-03-21T14:26:26.004681+0000","last_became_active":"2024-03-21T14:25:33.849755+0000","last_became_peered":"2024-03-21T14:25:33.849755+0000","last_unstale":"2024-03-21T14:26:26.004681+0000","last_undegraded":"2024-03-21T14:26:26.004681+0000","last_fullsized":"2024-03-21T14:26:26.004681+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:02.856647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:02.856647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.498375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":953,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895705+0000","last_change":"2024-03-21T14:25:05.783808+0000","last_active":"2024-03-21T14:26:25.895705+0000","last_peered":"2024-03-21T14:26:25.895705+0000","last_clean":"2024-03-21T14:26:25.895705+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:26:25.895705+0000","last_undegraded":"2024-03-21T14:26:25.895705+0000","last_fullsized":"2024-03-21T14:26:25.895705+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:05.783708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:05.783708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:31.873266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'27","reported_seq":1018,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896514+0000","last_change":"2024-03-21T14:25:44.840396+0000","last_active":"2024-03-21T14:26:25.896514+0000","last_peered":"2024-03-21T14:26:25.896514+0000","last_clean":"2024-03-21T14:26:25.896514+0000","last_became_active":"2024-03-21T14:25:44.820435+0000","last_became_peered":"2024-03-21T14:25:44.820435+0000","last_unstale":"2024-03-21T14:26:25.896514+0000","last_undegraded":"2024-03-21T14:26:25.896514+0000","last_fullsized":"2024-03-21T14:26:25.896514+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-21T14:25:41.505523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:41.505523+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:57.713745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":641,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001711+0000","last_change":"2024-03-21T14:25:33.853837+0000","last_active":"2024-03-21T14:26:26.001711+0000","last_peered":"2024-03-21T14:26:26.001711+0000","last_clean":"2024-03-21T14:26:26.001711+0000","last_became_active":"2024-03-21T14:25:33.852988+0000","last_became_peered":"2024-03-21T14:25:33.852988+0000","last_unstale":"2024-03-21T14:26:26.001711+0000","last_undegraded":"2024-03-21T14:26:26.001711+0000","last_fullsized":"2024-03-21T14:26:26.001711+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:21.473145+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:21.473145+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.653602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":748,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002115+0000","last_change":"2024-03-21T14:25:30.604982+0000","last_active":"2024-03-21T14:26:26.002115+0000","last_peered":"2024-03-21T14:26:26.002115+0000","last_clean":"2024-03-21T14:26:26.002115+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:26:26.002115+0000","last_undegraded":"2024-03-21T14:26:26.002115+0000","last_fullsized":"2024-03-21T14:26:26.002115+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:25:30.604798+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:25:30.604798+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:43.361050+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":960,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896627+0000","last_change":"2024-03-21T14:25:59.574316+0000","last_active":"2024-03-21T14:26:25.896627+0000","last_peered":"2024-03-21T14:26:25.896627+0000","last_clean":"2024-03-21T14:26:25.896627+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:26:25.896627+0000","last_undegraded":"2024-03-21T14:26:25.896627+0000","last_fullsized":"2024-03-21T14:26:25.896627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:59.574237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_clean_scrub_stamp":"2024-03-21T14:25:59.574237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.411374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002865+0000","last_change":"2024-03-21T14:26:20.645093+0000","last_active":"2024-03-21T14:26:26.002865+0000","last_peered":"2024-03-21T14:26:26.002865+0000","last_clean":"2024-03-21T14:26:26.002865+0000","last_became_active":"2024-03-21T14:25:44.824548+0000","last_became_peered":"2024-03-21T14:25:44.824548+0000","last_unstale":"2024-03-21T14:26:26.002865+0000","last_undegraded":"2024-03-21T14:26:26.002865+0000","last_fullsized":"2024-03-21T14:26:26.002865+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:20.644966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:20.644966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.247950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":1010,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896812+0000","last_change":"2024-03-21T14:26:15.296122+0000","last_active":"2024-03-21T14:26:25.896812+0000","last_peered":"2024-03-21T14:26:25.896812+0000","last_clean":"2024-03-21T14:26:25.896812+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:26:25.896812+0000","last_undegraded":"2024-03-21T14:26:25.896812+0000","last_fullsized":"2024-03-21T14:26:25.896812+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-21T14:26:15.295959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:26:15.295959+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.430574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1282,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001936+0000","last_change":"2024-03-21T14:26:09.982844+0000","last_active":"2024-03-21T14:26:26.001936+0000","last_peered":"2024-03-21T14:26:26.001936+0000","last_clean":"2024-03-21T14:26:26.001936+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:26:26.001936+0000","last_undegraded":"2024-03-21T14:26:26.001936+0000","last_fullsized":"2024-03-21T14:26:26.001936+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:09.982734+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:09.982734+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:19.580279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":956,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896772+0000","last_change":"2024-03-21T14:25:48.526680+0000","last_active":"2024-03-21T14:26:25.896772+0000","last_peered":"2024-03-21T14:26:25.896772+0000","last_clean":"2024-03-21T14:26:25.896772+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:26:25.896772+0000","last_undegraded":"2024-03-21T14:26:25.896772+0000","last_fullsized":"2024-03-21T14:26:25.896772+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:48.526574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:48.526574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:05.170583+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":1015,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:37.566263+0000","last_change":"2024-03-21T14:26:37.566307+0000","last_active":"2024-03-21T14:26:37.566263+0000","last_peered":"2024-03-21T14:26:37.566263+0000","last_clean":"2024-03-21T14:26:37.566263+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:26:37.566263+0000","last_undegraded":"2024-03-21T14:26:37.566263+0000","last_fullsized":"2024-03-21T14:26:37.566263+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:26:37.566179+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:26:37.566179+0000","last_clean_scrub_stamp":"2024-03-21T14:26:37.566179+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:28:03.908575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":964,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003365+0000","last_change":"2024-03-21T14:25:21.216238+0000","last_active":"2024-03-21T14:26:26.003365+0000","last_peered":"2024-03-21T14:26:26.003365+0000","last_clean":"2024-03-21T14:26:26.003365+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:26:26.003365+0000","last_undegraded":"2024-03-21T14:26:26.003365+0000","last_fullsized":"2024-03-21T14:26:26.003365+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:21.216147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:21.216147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:48.704283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":1025,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896241+0000","last_change":"2024-03-21T14:25:44.839269+0000","last_active":"2024-03-21T14:26:25.896241+0000","last_peered":"2024-03-21T14:26:25.896241+0000","last_clean":"2024-03-21T14:26:25.896241+0000","last_became_active":"2024-03-21T14:25:44.820487+0000","last_became_peered":"2024-03-21T14:25:44.820487+0000","last_unstale":"2024-03-21T14:26:25.896241+0000","last_undegraded":"2024-03-21T14:26:25.896241+0000","last_fullsized":"2024-03-21T14:26:25.896241+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:25:39.590137+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:25:39.590137+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.801270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":888,"num_read_kb":883,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":4736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103462,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898564,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846588,"statfs":{"total":95995035648,"available":95074906112,"internally_reserved":0,"allocated":1110016,"data_stored":722732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":351,"seq":1507533520989,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83828,"kb_used_data":740,"kb_used_omap":4,"kb_used_meta":83067,"kb_avail":93661324,"statfs":{"total":95995035648,"available":95909195776,"internally_reserved":0,"allocated":757760,"data_stored":393089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":85061284},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916316,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828836,"statfs":{"total":95995035648,"available":95056728064,"internally_reserved":0,"allocated":806912,"data_stored":395643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738699,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880196,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864956,"statfs":{"total":95995035648,"available":95093714944,"internally_reserved":0,"allocated":782336,"data_stored":393883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598292059,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":404220,"kb_used_data":1012,"kb_used_omap":7,"kb_used_meta":403192,"kb_avail":93340932,"statfs":{"total":95995035648,"available":95581114368,"internally_reserved":0,"allocated":1036288,"data_stored":656235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":412868740},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182714,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822488,"kb_used_data":1360,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92922664,"statfs":{"total":95995035648,"available":95152807936,"internally_reserved":0,"allocated":1392640,"data_stored":984687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":840825287},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019190,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924624,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820528,"statfs":{"total":95995035648,"available":95048220672,"internally_reserved":0,"allocated":794624,"data_stored":394143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692145,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907144,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92838008,"statfs":{"total":95995035648,"available":95066120192,"internally_reserved":0,"allocated":1376256,"data_stored":985052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:26:40.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-21T14:26:40.912 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:41.210 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:40 smithi003 ceph-mon[29576]: from='client.16395 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:26:41.210 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:40 smithi003 ceph-mon[40510]: from='client.16395 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:26:41.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:40 smithi118 ceph-mon[33862]: from='client.16395 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:26:42.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[29576]: 3.10 deep-scrub starts 2024-03-21T14:26:42.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[29576]: 3.10 deep-scrub ok 2024-03-21T14:26:42.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[29576]: pgmap v1133: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:42.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[29576]: 2.2 scrub starts 2024-03-21T14:26:42.211 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[29576]: 2.2 scrub ok 2024-03-21T14:26:42.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[40510]: 3.10 deep-scrub starts 2024-03-21T14:26:42.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[40510]: 3.10 deep-scrub ok 2024-03-21T14:26:42.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[40510]: pgmap v1133: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:42.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[40510]: 2.2 scrub starts 2024-03-21T14:26:42.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:41 smithi003 ceph-mon[40510]: 2.2 scrub ok 2024-03-21T14:26:42.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:41 smithi118 ceph-mon[33862]: 3.10 deep-scrub starts 2024-03-21T14:26:42.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:41 smithi118 ceph-mon[33862]: 3.10 deep-scrub ok 2024-03-21T14:26:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:41 smithi118 ceph-mon[33862]: pgmap v1133: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:41 smithi118 ceph-mon[33862]: 2.2 scrub starts 2024-03-21T14:26:42.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:41 smithi118 ceph-mon[33862]: 2.2 scrub ok 2024-03-21T14:26:43.000 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-03-21T14:26:43.003 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-03-21T14:26:43.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[29576]: 3.13 scrub starts 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[29576]: 3.13 scrub ok 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[29576]: 5.13 scrub starts 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[29576]: 5.13 scrub ok 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[29576]: 5.2 scrub starts 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[29576]: 5.2 scrub ok 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[40510]: 3.13 scrub starts 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[40510]: 3.13 scrub ok 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[40510]: 5.13 scrub starts 2024-03-21T14:26:43.254 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[40510]: 5.13 scrub ok 2024-03-21T14:26:43.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[40510]: 5.2 scrub starts 2024-03-21T14:26:43.255 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:42 smithi003 ceph-mon[40510]: 5.2 scrub ok 2024-03-21T14:26:43.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:42 smithi118 ceph-mon[33862]: 3.13 scrub starts 2024-03-21T14:26:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:42 smithi118 ceph-mon[33862]: 3.13 scrub ok 2024-03-21T14:26:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:42 smithi118 ceph-mon[33862]: 5.13 scrub starts 2024-03-21T14:26:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:42 smithi118 ceph-mon[33862]: 5.13 scrub ok 2024-03-21T14:26:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:42 smithi118 ceph-mon[33862]: 5.2 scrub starts 2024-03-21T14:26:43.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:42 smithi118 ceph-mon[33862]: 5.2 scrub ok 2024-03-21T14:26:43.692 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":1134,"stamp":"2024-03-21T14:26:42.843922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2449,"num_read_kb":2201,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":5004,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2627,"ondisk_log_size":2627,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":5837444,"kb_used_data":7928,"kb_used_omap":21,"kb_used_meta":5829418,"kb_avail":744123772,"statfs":{"total":767960285184,"available":761982742528,"internally_reserved":0,"allocated":8118272,"data_stored":4959224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":22067,"internal_metadata":5969324493},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002841"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":932,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937494+0000","last_change":"2024-03-21T14:25:32.652858+0000","last_active":"2024-03-21T14:26:38.937494+0000","last_peered":"2024-03-21T14:26:38.937494+0000","last_clean":"2024-03-21T14:26:38.937494+0000","last_became_active":"2024-03-21T14:02:39.462814+0000","last_became_peered":"2024-03-21T14:02:39.462814+0000","last_unstale":"2024-03-21T14:26:38.937494+0000","last_undegraded":"2024-03-21T14:26:38.937494+0000","last_fullsized":"2024-03-21T14:26:38.937494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:32.652753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:32.652753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":959,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895309+0000","last_change":"2024-03-21T14:25:30.704554+0000","last_active":"2024-03-21T14:26:25.895309+0000","last_peered":"2024-03-21T14:26:25.895309+0000","last_clean":"2024-03-21T14:26:25.895309+0000","last_became_active":"2024-03-21T14:02:37.454363+0000","last_became_peered":"2024-03-21T14:02:37.454363+0000","last_unstale":"2024-03-21T14:26:25.895309+0000","last_undegraded":"2024-03-21T14:26:25.895309+0000","last_fullsized":"2024-03-21T14:26:25.895309+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:30.704477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:14:03.429627+0000","last_clean_scrub_stamp":"2024-03-21T14:25:30.704477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:43.641722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":283,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938863+0000","last_change":"2024-03-21T14:25:40.566340+0000","last_active":"2024-03-21T14:26:38.938863+0000","last_peered":"2024-03-21T14:26:38.938863+0000","last_clean":"2024-03-21T14:26:38.938863+0000","last_became_active":"2024-03-21T14:25:06.187486+0000","last_became_peered":"2024-03-21T14:25:06.187486+0000","last_unstale":"2024-03-21T14:26:38.938863+0000","last_undegraded":"2024-03-21T14:26:38.938863+0000","last_fullsized":"2024-03-21T14:26:38.938863+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:40.566239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:40.566239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:42.823444+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"441'21","reported_seq":1023,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937312+0000","last_change":"2024-03-21T14:25:57.458594+0000","last_active":"2024-03-21T14:26:38.937312+0000","last_peered":"2024-03-21T14:26:38.937312+0000","last_clean":"2024-03-21T14:26:38.937312+0000","last_became_active":"2024-03-21T14:25:44.829149+0000","last_became_peered":"2024-03-21T14:25:44.829149+0000","last_unstale":"2024-03-21T14:26:38.937312+0000","last_undegraded":"2024-03-21T14:26:38.937312+0000","last_fullsized":"2024-03-21T14:26:38.937312+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:25:57.458462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.458462+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.731738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":243,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322630+0000","last_change":"2024-03-21T14:25:54.676827+0000","last_active":"2024-03-21T14:26:39.322630+0000","last_peered":"2024-03-21T14:26:39.322630+0000","last_clean":"2024-03-21T14:26:39.322630+0000","last_became_active":"2024-03-21T14:25:44.830791+0000","last_became_peered":"2024-03-21T14:25:44.830791+0000","last_unstale":"2024-03-21T14:26:39.322630+0000","last_undegraded":"2024-03-21T14:26:39.322630+0000","last_fullsized":"2024-03-21T14:26:39.322630+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:54.676712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:54.676712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:21.173467+0000","scrub_duration":142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":957,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937183+0000","last_change":"2024-03-21T14:26:09.371706+0000","last_active":"2024-03-21T14:26:38.937183+0000","last_peered":"2024-03-21T14:26:38.937183+0000","last_clean":"2024-03-21T14:26:38.937183+0000","last_became_active":"2024-03-21T14:25:44.828628+0000","last_became_peered":"2024-03-21T14:25:44.828628+0000","last_unstale":"2024-03-21T14:26:38.937183+0000","last_undegraded":"2024-03-21T14:26:38.937183+0000","last_fullsized":"2024-03-21T14:26:38.937183+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:09.371608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:03.258168+0000","last_clean_scrub_stamp":"2024-03-21T14:26:09.371608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:19.630543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":969,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323367+0000","last_change":"2024-03-21T14:26:00.448519+0000","last_active":"2024-03-21T14:26:39.323367+0000","last_peered":"2024-03-21T14:26:39.323367+0000","last_clean":"2024-03-21T14:26:39.323367+0000","last_became_active":"2024-03-21T14:25:44.829775+0000","last_became_peered":"2024-03-21T14:25:44.829775+0000","last_unstale":"2024-03-21T14:26:39.323367+0000","last_undegraded":"2024-03-21T14:26:39.323367+0000","last_fullsized":"2024-03-21T14:26:39.323367+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:00.448423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:00.448423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:00.848779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"441'46","reported_seq":1018,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322965+0000","last_change":"2024-03-21T14:26:02.534332+0000","last_active":"2024-03-21T14:26:39.322965+0000","last_peered":"2024-03-21T14:26:39.322965+0000","last_clean":"2024-03-21T14:26:39.322965+0000","last_became_active":"2024-03-21T14:19:14.696177+0000","last_became_peered":"2024-03-21T14:19:14.696177+0000","last_unstale":"2024-03-21T14:26:39.322965+0000","last_undegraded":"2024-03-21T14:26:39.322965+0000","last_fullsized":"2024-03-21T14:26:39.322965+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-21T14:26:02.534201+0000","last_deep_scrub":"83'22","last_deep_scrub_stamp":"2024-03-21T14:03:35.571531+0000","last_clean_scrub_stamp":"2024-03-21T14:26:02.534201+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.683916+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":596,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938502+0000","last_change":"2024-03-21T14:25:35.794434+0000","last_active":"2024-03-21T14:26:38.938502+0000","last_peered":"2024-03-21T14:26:38.938502+0000","last_clean":"2024-03-21T14:26:38.938502+0000","last_became_active":"2024-03-21T14:25:06.180542+0000","last_became_peered":"2024-03-21T14:25:06.180542+0000","last_unstale":"2024-03-21T14:26:38.938502+0000","last_undegraded":"2024-03-21T14:26:38.938502+0000","last_fullsized":"2024-03-21T14:26:38.938502+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:35.794333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:35.794333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:02.359190+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":968,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649429+0000","last_change":"2024-03-21T14:26:03.852118+0000","last_active":"2024-03-21T14:26:39.649429+0000","last_peered":"2024-03-21T14:26:39.649429+0000","last_clean":"2024-03-21T14:26:39.649429+0000","last_became_active":"2024-03-21T14:02:37.455264+0000","last_became_peered":"2024-03-21T14:02:37.455264+0000","last_unstale":"2024-03-21T14:26:39.649429+0000","last_undegraded":"2024-03-21T14:26:39.649429+0000","last_fullsized":"2024-03-21T14:26:39.649429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:03.852004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:04:14.328050+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.852004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:11.861692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":596,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649770+0000","last_change":"2024-03-21T14:25:44.824475+0000","last_active":"2024-03-21T14:26:39.649770+0000","last_peered":"2024-03-21T14:26:39.649770+0000","last_clean":"2024-03-21T14:26:39.649770+0000","last_became_active":"2024-03-21T14:25:44.823480+0000","last_became_peered":"2024-03-21T14:25:44.823480+0000","last_unstale":"2024-03-21T14:26:39.649770+0000","last_undegraded":"2024-03-21T14:26:39.649770+0000","last_fullsized":"2024-03-21T14:26:39.649770+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:29.587727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:10.251587+0000","last_clean_scrub_stamp":"2024-03-21T14:25:29.587727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:56.792309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"441'21","reported_seq":966,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937662+0000","last_change":"2024-03-21T14:25:28.565057+0000","last_active":"2024-03-21T14:26:38.937662+0000","last_peered":"2024-03-21T14:26:38.937662+0000","last_clean":"2024-03-21T14:26:38.937662+0000","last_became_active":"2024-03-21T14:02:35.445519+0000","last_became_peered":"2024-03-21T14:02:35.445519+0000","last_unstale":"2024-03-21T14:26:38.937662+0000","last_undegraded":"2024-03-21T14:26:38.937662+0000","last_fullsized":"2024-03-21T14:26:38.937662+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:25:28.564942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:28.564942+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:45.461197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":711,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649341+0000","last_change":"2024-03-21T14:25:33.854317+0000","last_active":"2024-03-21T14:26:39.649341+0000","last_peered":"2024-03-21T14:26:39.649341+0000","last_clean":"2024-03-21T14:26:39.649341+0000","last_became_active":"2024-03-21T14:25:33.853824+0000","last_became_peered":"2024-03-21T14:25:33.853824+0000","last_unstale":"2024-03-21T14:26:39.649341+0000","last_undegraded":"2024-03-21T14:26:39.649341+0000","last_fullsized":"2024-03-21T14:26:39.649341+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:11.787910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:11.787910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":961,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.547757+0000","last_change":"2024-03-21T14:25:49.999479+0000","last_active":"2024-03-21T14:26:39.547757+0000","last_peered":"2024-03-21T14:26:39.547757+0000","last_clean":"2024-03-21T14:26:39.547757+0000","last_became_active":"2024-03-21T14:02:37.456945+0000","last_became_peered":"2024-03-21T14:02:37.456945+0000","last_unstale":"2024-03-21T14:26:39.547757+0000","last_undegraded":"2024-03-21T14:26:39.547757+0000","last_fullsized":"2024-03-21T14:26:39.547757+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:49.999361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:01.560288+0000","last_clean_scrub_stamp":"2024-03-21T14:25:49.999361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:10.807521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":592,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649774+0000","last_change":"2024-03-21T14:26:07.821639+0000","last_active":"2024-03-21T14:26:39.649774+0000","last_peered":"2024-03-21T14:26:39.649774+0000","last_clean":"2024-03-21T14:26:39.649774+0000","last_became_active":"2024-03-21T14:25:33.849959+0000","last_became_peered":"2024-03-21T14:25:33.849959+0000","last_unstale":"2024-03-21T14:26:39.649774+0000","last_undegraded":"2024-03-21T14:26:39.649774+0000","last_fullsized":"2024-03-21T14:26:39.649774+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:07.821538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:53.335395+0000","last_clean_scrub_stamp":"2024-03-21T14:26:07.821538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:22.857051+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"445'10","reported_seq":957,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937772+0000","last_change":"2024-03-21T14:25:38.571193+0000","last_active":"2024-03-21T14:26:38.937772+0000","last_peered":"2024-03-21T14:26:38.937772+0000","last_clean":"2024-03-21T14:26:38.937772+0000","last_became_active":"2024-03-21T14:02:35.453608+0000","last_became_peered":"2024-03-21T14:02:35.453608+0000","last_unstale":"2024-03-21T14:26:38.937772+0000","last_undegraded":"2024-03-21T14:26:38.937772+0000","last_fullsized":"2024-03-21T14:26:38.937772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'10","last_scrub_stamp":"2024-03-21T14:25:38.571104+0000","last_deep_scrub":"445'10","last_deep_scrub_stamp":"2024-03-21T14:22:57.289894+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.571104+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:52.055693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":939,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938381+0000","last_change":"2024-03-21T14:25:44.821878+0000","last_active":"2024-03-21T14:26:38.938381+0000","last_peered":"2024-03-21T14:26:38.938381+0000","last_clean":"2024-03-21T14:26:38.938381+0000","last_became_active":"2024-03-21T14:25:44.821097+0000","last_became_peered":"2024-03-21T14:25:44.821097+0000","last_unstale":"2024-03-21T14:26:38.938381+0000","last_undegraded":"2024-03-21T14:26:38.938381+0000","last_fullsized":"2024-03-21T14:26:38.938381+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:03.866108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:03.866108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":951,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897040+0000","last_change":"2024-03-21T14:26:10.507780+0000","last_active":"2024-03-21T14:26:25.897040+0000","last_peered":"2024-03-21T14:26:25.897040+0000","last_clean":"2024-03-21T14:26:25.897040+0000","last_became_active":"2024-03-21T14:02:37.456958+0000","last_became_peered":"2024-03-21T14:02:37.456958+0000","last_unstale":"2024-03-21T14:26:25.897040+0000","last_undegraded":"2024-03-21T14:26:25.897040+0000","last_fullsized":"2024-03-21T14:26:25.897040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:10.507698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:10.507698+0000","last_clean_scrub_stamp":"2024-03-21T14:26:10.507698+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:12.801218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":586,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.939078+0000","last_change":"2024-03-21T14:25:41.585099+0000","last_active":"2024-03-21T14:26:38.939078+0000","last_peered":"2024-03-21T14:26:38.939078+0000","last_clean":"2024-03-21T14:26:38.939078+0000","last_became_active":"2024-03-21T14:25:06.185211+0000","last_became_peered":"2024-03-21T14:25:06.185211+0000","last_unstale":"2024-03-21T14:26:38.939078+0000","last_undegraded":"2024-03-21T14:26:38.939078+0000","last_fullsized":"2024-03-21T14:26:38.939078+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:41.584987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:37.108651+0000","last_clean_scrub_stamp":"2024-03-21T14:25:41.584987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.619880+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"441'46","reported_seq":882,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002265+0000","last_change":"2024-03-21T14:26:12.607821+0000","last_active":"2024-03-21T14:26:26.002265+0000","last_peered":"2024-03-21T14:26:26.002265+0000","last_clean":"2024-03-21T14:26:26.002265+0000","last_became_active":"2024-03-21T14:19:14.700298+0000","last_became_peered":"2024-03-21T14:19:14.700298+0000","last_unstale":"2024-03-21T14:26:26.002265+0000","last_undegraded":"2024-03-21T14:26:26.002265+0000","last_fullsized":"2024-03-21T14:26:26.002265+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'46","last_scrub_stamp":"2024-03-21T14:26:12.607692+0000","last_deep_scrub":"432'34","last_deep_scrub_stamp":"2024-03-21T14:21:50.322580+0000","last_clean_scrub_stamp":"2024-03-21T14:26:12.607692+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:39.617259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":963,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323521+0000","last_change":"2024-03-21T14:26:14.782547+0000","last_active":"2024-03-21T14:26:39.323521+0000","last_peered":"2024-03-21T14:26:39.323521+0000","last_clean":"2024-03-21T14:26:39.323521+0000","last_became_active":"2024-03-21T14:25:44.829912+0000","last_became_peered":"2024-03-21T14:25:44.829912+0000","last_unstale":"2024-03-21T14:26:39.323521+0000","last_undegraded":"2024-03-21T14:26:39.323521+0000","last_fullsized":"2024-03-21T14:26:39.323521+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:14.782388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:14.782388+0000","last_clean_scrub_stamp":"2024-03-21T14:26:14.782388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.923263+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":937,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937963+0000","last_change":"2024-03-21T14:25:15.656176+0000","last_active":"2024-03-21T14:26:38.937963+0000","last_peered":"2024-03-21T14:26:38.937963+0000","last_clean":"2024-03-21T14:26:38.937963+0000","last_became_active":"2024-03-21T14:19:14.690504+0000","last_became_peered":"2024-03-21T14:19:14.690504+0000","last_unstale":"2024-03-21T14:26:38.937963+0000","last_undegraded":"2024-03-21T14:26:38.937963+0000","last_fullsized":"2024-03-21T14:26:38.937963+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:15.656080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:15.656080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:39.924790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":945,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937780+0000","last_change":"2024-03-21T14:25:56.492197+0000","last_active":"2024-03-21T14:26:38.937780+0000","last_peered":"2024-03-21T14:26:38.937780+0000","last_clean":"2024-03-21T14:26:38.937780+0000","last_became_active":"2024-03-21T14:19:14.681923+0000","last_became_peered":"2024-03-21T14:19:14.681923+0000","last_unstale":"2024-03-21T14:26:38.937780+0000","last_undegraded":"2024-03-21T14:26:38.937780+0000","last_fullsized":"2024-03-21T14:26:38.937780+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:56.492049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:47.949388+0000","last_clean_scrub_stamp":"2024-03-21T14:25:56.492049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:08.701378+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"441'24","reported_seq":998,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323109+0000","last_change":"2024-03-21T14:26:29.494564+0000","last_active":"2024-03-21T14:26:39.323109+0000","last_peered":"2024-03-21T14:26:39.323109+0000","last_clean":"2024-03-21T14:26:39.323109+0000","last_became_active":"2024-03-21T14:02:35.451336+0000","last_became_peered":"2024-03-21T14:02:35.451336+0000","last_unstale":"2024-03-21T14:26:39.323109+0000","last_undegraded":"2024-03-21T14:26:39.323109+0000","last_fullsized":"2024-03-21T14:26:39.323109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'24","last_scrub_stamp":"2024-03-21T14:26:29.494442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:26:29.494442+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:37.258263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"128'2","reported_seq":678,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548162+0000","last_change":"2024-03-21T14:25:40.881675+0000","last_active":"2024-03-21T14:26:39.548162+0000","last_peered":"2024-03-21T14:26:39.548162+0000","last_clean":"2024-03-21T14:26:39.548162+0000","last_became_active":"2024-03-21T14:25:40.881203+0000","last_became_peered":"2024-03-21T14:25:40.881203+0000","last_unstale":"2024-03-21T14:26:39.548162+0000","last_undegraded":"2024-03-21T14:26:39.548162+0000","last_fullsized":"2024-03-21T14:26:39.548162+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"128'2","last_scrub_stamp":"2024-03-21T14:25:38.140789+0000","last_deep_scrub":"128'2","last_deep_scrub_stamp":"2024-03-21T14:14:00.547775+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.140789+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:42.969048+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"425'15","reported_seq":262,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:33.777010+0000","last_change":"2024-03-21T14:26:17.632016+0000","last_active":"2024-03-21T14:26:33.777010+0000","last_peered":"2024-03-21T14:26:33.777010+0000","last_clean":"2024-03-21T14:26:33.777010+0000","last_became_active":"2024-03-21T14:21:26.767116+0000","last_became_peered":"2024-03-21T14:21:26.767116+0000","last_unstale":"2024-03-21T14:26:33.777010+0000","last_undegraded":"2024-03-21T14:26:33.777010+0000","last_fullsized":"2024-03-21T14:26:33.777010+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"425'15","last_scrub_stamp":"2024-03-21T14:26:17.631885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:17.631885+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.670224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"85'4","reported_seq":968,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895199+0000","last_change":"2024-03-21T14:25:18.715391+0000","last_active":"2024-03-21T14:26:25.895199+0000","last_peered":"2024-03-21T14:26:25.895199+0000","last_clean":"2024-03-21T14:26:25.895199+0000","last_became_active":"2024-03-21T14:02:33.441183+0000","last_became_peered":"2024-03-21T14:02:33.441183+0000","last_unstale":"2024-03-21T14:26:25.895199+0000","last_undegraded":"2024-03-21T14:26:25.895199+0000","last_fullsized":"2024-03-21T14:26:25.895199+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'4","last_scrub_stamp":"2024-03-21T14:25:18.715307+0000","last_deep_scrub":"85'4","last_deep_scrub_stamp":"2024-03-21T14:03:54.710220+0000","last_clean_scrub_stamp":"2024-03-21T14:25:18.715307+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.314016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"441'30","reported_seq":631,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938554+0000","last_change":"2024-03-21T14:26:13.474687+0000","last_active":"2024-03-21T14:26:38.938554+0000","last_peered":"2024-03-21T14:26:38.938554+0000","last_clean":"2024-03-21T14:26:38.938554+0000","last_became_active":"2024-03-21T14:25:06.188176+0000","last_became_peered":"2024-03-21T14:25:06.188176+0000","last_unstale":"2024-03-21T14:26:38.938554+0000","last_undegraded":"2024-03-21T14:26:38.938554+0000","last_fullsized":"2024-03-21T14:26:38.938554+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'30","last_scrub_stamp":"2024-03-21T14:26:13.474554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:26:13.474554+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.509099+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":947,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896755+0000","last_change":"2024-03-21T14:25:28.823011+0000","last_active":"2024-03-21T14:26:25.896755+0000","last_peered":"2024-03-21T14:26:25.896755+0000","last_clean":"2024-03-21T14:26:25.896755+0000","last_became_active":"2024-03-21T14:02:39.465534+0000","last_became_peered":"2024-03-21T14:02:39.465534+0000","last_unstale":"2024-03-21T14:26:25.896755+0000","last_undegraded":"2024-03-21T14:26:25.896755+0000","last_fullsized":"2024-03-21T14:26:25.896755+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:28.822882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:06.928522+0000","last_clean_scrub_stamp":"2024-03-21T14:25:28.822882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:55.344105+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":424,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938968+0000","last_change":"2024-03-21T14:26:12.450780+0000","last_active":"2024-03-21T14:26:38.938968+0000","last_peered":"2024-03-21T14:26:38.938968+0000","last_clean":"2024-03-21T14:26:38.938968+0000","last_became_active":"2024-03-21T14:25:06.187798+0000","last_became_peered":"2024-03-21T14:25:06.187798+0000","last_unstale":"2024-03-21T14:26:38.938968+0000","last_undegraded":"2024-03-21T14:26:38.938968+0000","last_fullsized":"2024-03-21T14:26:38.938968+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:12.450693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:20.617078+0000","last_clean_scrub_stamp":"2024-03-21T14:26:12.450693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.727757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":974,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322351+0000","last_change":"2024-03-21T14:26:06.458507+0000","last_active":"2024-03-21T14:26:39.322351+0000","last_peered":"2024-03-21T14:26:39.322351+0000","last_clean":"2024-03-21T14:26:39.322351+0000","last_became_active":"2024-03-21T14:19:34.690281+0000","last_became_peered":"2024-03-21T14:19:34.690281+0000","last_unstale":"2024-03-21T14:26:39.322351+0000","last_undegraded":"2024-03-21T14:26:39.322351+0000","last_fullsized":"2024-03-21T14:26:39.322351+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:06.458427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:39.522309+0000","last_clean_scrub_stamp":"2024-03-21T14:26:06.458427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:16.698768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"441'32","reported_seq":51,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:33.697732+0000","last_change":"2024-03-21T14:26:33.697778+0000","last_active":"2024-03-21T14:26:33.697732+0000","last_peered":"2024-03-21T14:26:33.697732+0000","last_clean":"2024-03-21T14:26:33.697732+0000","last_became_active":"2024-03-21T14:25:44.830640+0000","last_became_peered":"2024-03-21T14:25:44.830640+0000","last_unstale":"2024-03-21T14:26:33.697732+0000","last_undegraded":"2024-03-21T14:26:33.697732+0000","last_fullsized":"2024-03-21T14:26:33.697732+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:26:33.697657+0000","last_deep_scrub":"441'32","last_deep_scrub_stamp":"2024-03-21T14:25:29.702878+0000","last_clean_scrub_stamp":"2024-03-21T14:26:33.697657+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:34.251898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":590,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548024+0000","last_change":"2024-03-21T14:26:31.226056+0000","last_active":"2024-03-21T14:26:39.548024+0000","last_peered":"2024-03-21T14:26:39.548024+0000","last_clean":"2024-03-21T14:26:39.548024+0000","last_became_active":"2024-03-21T14:25:40.881790+0000","last_became_peered":"2024-03-21T14:25:40.881790+0000","last_unstale":"2024-03-21T14:26:39.548024+0000","last_undegraded":"2024-03-21T14:26:39.548024+0000","last_fullsized":"2024-03-21T14:26:39.548024+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:31.225936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:31.225936+0000","last_clean_scrub_stamp":"2024-03-21T14:26:31.225936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:55.860753+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":31,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002647+0000","last_change":"2024-03-21T14:25:44.829116+0000","last_active":"2024-03-21T14:26:26.002647+0000","last_peered":"2024-03-21T14:26:26.002647+0000","last_clean":"2024-03-21T14:26:26.002647+0000","last_became_active":"2024-03-21T14:25:44.827267+0000","last_became_peered":"2024-03-21T14:25:44.827267+0000","last_unstale":"2024-03-21T14:26:26.002647+0000","last_undegraded":"2024-03-21T14:26:26.002647+0000","last_fullsized":"2024-03-21T14:26:26.002647+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:43.509640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:18:42.513167+0000","last_clean_scrub_stamp":"2024-03-21T14:25:43.509640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:10.670416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":448,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.939458+0000","last_change":"2024-03-21T14:26:14.489099+0000","last_active":"2024-03-21T14:26:38.939458+0000","last_peered":"2024-03-21T14:26:38.939458+0000","last_clean":"2024-03-21T14:26:38.939458+0000","last_became_active":"2024-03-21T14:25:06.187626+0000","last_became_peered":"2024-03-21T14:25:06.187626+0000","last_unstale":"2024-03-21T14:26:38.939458+0000","last_undegraded":"2024-03-21T14:26:38.939458+0000","last_fullsized":"2024-03-21T14:26:38.939458+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:14.489015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:14.489015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:27.251845+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"441'14","reported_seq":1002,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322791+0000","last_change":"2024-03-21T14:25:44.855919+0000","last_active":"2024-03-21T14:26:39.322791+0000","last_peered":"2024-03-21T14:26:39.322791+0000","last_clean":"2024-03-21T14:26:39.322791+0000","last_became_active":"2024-03-21T14:25:44.830268+0000","last_became_peered":"2024-03-21T14:25:44.830268+0000","last_unstale":"2024-03-21T14:26:39.322791+0000","last_undegraded":"2024-03-21T14:26:39.322791+0000","last_fullsized":"2024-03-21T14:26:39.322791+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'14","last_scrub_stamp":"2024-03-21T14:25:42.334314+0000","last_deep_scrub":"275'10","last_deep_scrub_stamp":"2024-03-21T14:14:47.712773+0000","last_clean_scrub_stamp":"2024-03-21T14:25:42.334314+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:56.758732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":980,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323133+0000","last_change":"2024-03-21T14:26:23.650255+0000","last_active":"2024-03-21T14:26:39.323133+0000","last_peered":"2024-03-21T14:26:39.323133+0000","last_clean":"2024-03-21T14:26:39.323133+0000","last_became_active":"2024-03-21T14:25:44.835043+0000","last_became_peered":"2024-03-21T14:25:44.835043+0000","last_unstale":"2024-03-21T14:26:39.323133+0000","last_undegraded":"2024-03-21T14:26:39.323133+0000","last_fullsized":"2024-03-21T14:26:39.323133+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:23.650090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:04.400332+0000","last_clean_scrub_stamp":"2024-03-21T14:26:23.650090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:49.232270+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":998,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548028+0000","last_change":"2024-03-21T14:26:25.090014+0000","last_active":"2024-03-21T14:26:39.548028+0000","last_peered":"2024-03-21T14:26:39.548028+0000","last_clean":"2024-03-21T14:26:39.548028+0000","last_became_active":"2024-03-21T14:25:44.824474+0000","last_became_peered":"2024-03-21T14:25:44.824474+0000","last_unstale":"2024-03-21T14:26:39.548028+0000","last_undegraded":"2024-03-21T14:26:39.548028+0000","last_fullsized":"2024-03-21T14:26:39.548028+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:25.089884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:25.089884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:38.286023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"65'1","reported_seq":598,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649850+0000","last_change":"2024-03-21T14:26:11.714981+0000","last_active":"2024-03-21T14:26:39.649850+0000","last_peered":"2024-03-21T14:26:39.649850+0000","last_clean":"2024-03-21T14:26:39.649850+0000","last_became_active":"2024-03-21T14:25:33.855990+0000","last_became_peered":"2024-03-21T14:25:33.855990+0000","last_unstale":"2024-03-21T14:26:39.649850+0000","last_undegraded":"2024-03-21T14:26:39.649850+0000","last_fullsized":"2024-03-21T14:26:39.649850+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'1","last_scrub_stamp":"2024-03-21T14:26:11.714855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:11.714855+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.516720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":911,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"441'32","reported_seq":1042,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322858+0000","last_change":"2024-03-21T14:25:50.432842+0000","last_active":"2024-03-21T14:26:39.322858+0000","last_peered":"2024-03-21T14:26:39.322858+0000","last_clean":"2024-03-21T14:26:39.322858+0000","last_became_active":"2024-03-21T14:25:44.829719+0000","last_became_peered":"2024-03-21T14:25:44.829719+0000","last_unstale":"2024-03-21T14:26:39.322858+0000","last_undegraded":"2024-03-21T14:26:39.322858+0000","last_fullsized":"2024-03-21T14:26:39.322858+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:25:50.432680+0000","last_deep_scrub":"261'24","last_deep_scrub_stamp":"2024-03-21T14:13:55.514646+0000","last_clean_scrub_stamp":"2024-03-21T14:25:50.432680+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:54.065801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":436,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.388917+0000","last_change":"2024-03-21T14:26:39.388950+0000","last_active":"2024-03-21T14:26:39.388917+0000","last_peered":"2024-03-21T14:26:39.388917+0000","last_clean":"2024-03-21T14:26:39.388917+0000","last_became_active":"2024-03-21T14:25:06.181970+0000","last_became_peered":"2024-03-21T14:25:06.181970+0000","last_unstale":"2024-03-21T14:26:39.388917+0000","last_undegraded":"2024-03-21T14:26:39.388917+0000","last_fullsized":"2024-03-21T14:26:39.388917+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:39.388832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:39.388832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:28:03.236829+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":954,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:41.486511+0000","last_change":"2024-03-21T14:26:41.486532+0000","last_active":"2024-03-21T14:26:41.486511+0000","last_peered":"2024-03-21T14:26:41.486511+0000","last_clean":"2024-03-21T14:26:41.486511+0000","last_became_active":"2024-03-21T14:02:37.456600+0000","last_became_peered":"2024-03-21T14:02:37.456600+0000","last_unstale":"2024-03-21T14:26:41.486511+0000","last_undegraded":"2024-03-21T14:26:41.486511+0000","last_fullsized":"2024-03-21T14:26:41.486511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:41.486453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:06:29.146695+0000","last_clean_scrub_stamp":"2024-03-21T14:26:41.486453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:50.202817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":978,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548376+0000","last_change":"2024-03-21T14:26:08.041589+0000","last_active":"2024-03-21T14:26:39.548376+0000","last_peered":"2024-03-21T14:26:39.548376+0000","last_clean":"2024-03-21T14:26:39.548376+0000","last_became_active":"2024-03-21T14:02:33.448892+0000","last_became_peered":"2024-03-21T14:02:33.448892+0000","last_unstale":"2024-03-21T14:26:39.548376+0000","last_undegraded":"2024-03-21T14:26:39.548376+0000","last_fullsized":"2024-03-21T14:26:39.548376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:08.041435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:08.041435+0000","last_clean_scrub_stamp":"2024-03-21T14:26:08.041435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:12.644782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"472'25","reported_seq":1035,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896111+0000","last_change":"2024-03-21T14:26:22.497135+0000","last_active":"2024-03-21T14:26:25.896111+0000","last_peered":"2024-03-21T14:26:25.896111+0000","last_clean":"2024-03-21T14:26:25.896111+0000","last_became_active":"2024-03-21T14:25:44.820409+0000","last_became_peered":"2024-03-21T14:25:44.820409+0000","last_unstale":"2024-03-21T14:26:25.896111+0000","last_undegraded":"2024-03-21T14:26:25.896111+0000","last_fullsized":"2024-03-21T14:26:25.896111+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'25","last_scrub_stamp":"2024-03-21T14:26:22.497041+0000","last_deep_scrub":"472'25","last_deep_scrub_stamp":"2024-03-21T14:25:10.723028+0000","last_clean_scrub_stamp":"2024-03-21T14:26:22.497041+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.134295+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":245,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322541+0000","last_change":"2024-03-21T14:26:04.523815+0000","last_active":"2024-03-21T14:26:39.322541+0000","last_peered":"2024-03-21T14:26:39.322541+0000","last_clean":"2024-03-21T14:26:39.322541+0000","last_became_active":"2024-03-21T14:20:21.783618+0000","last_became_peered":"2024-03-21T14:20:21.783618+0000","last_unstale":"2024-03-21T14:26:39.322541+0000","last_undegraded":"2024-03-21T14:26:39.322541+0000","last_fullsized":"2024-03-21T14:26:39.322541+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:04.523701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:04.523701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.784394+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":980,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:37.646717+0000","last_change":"2024-03-21T14:26:37.646760+0000","last_active":"2024-03-21T14:26:37.646717+0000","last_peered":"2024-03-21T14:26:37.646717+0000","last_clean":"2024-03-21T14:26:37.646717+0000","last_became_active":"2024-03-21T14:20:21.778058+0000","last_became_peered":"2024-03-21T14:20:21.778058+0000","last_unstale":"2024-03-21T14:26:37.646717+0000","last_undegraded":"2024-03-21T14:26:37.646717+0000","last_fullsized":"2024-03-21T14:26:37.646717+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:37.646641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:31.695568+0000","last_clean_scrub_stamp":"2024-03-21T14:26:37.646641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:56.755984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":974,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937763+0000","last_change":"2024-03-21T14:26:16.289389+0000","last_active":"2024-03-21T14:26:38.937763+0000","last_peered":"2024-03-21T14:26:38.937763+0000","last_clean":"2024-03-21T14:26:38.937763+0000","last_became_active":"2024-03-21T14:25:44.819688+0000","last_became_peered":"2024-03-21T14:25:44.819688+0000","last_unstale":"2024-03-21T14:26:38.937763+0000","last_undegraded":"2024-03-21T14:26:38.937763+0000","last_fullsized":"2024-03-21T14:26:38.937763+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:16.289291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:16.289291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.260254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"441'32","reported_seq":1059,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937781+0000","last_change":"2024-03-21T14:26:10.357525+0000","last_active":"2024-03-21T14:26:38.937781+0000","last_peered":"2024-03-21T14:26:38.937781+0000","last_clean":"2024-03-21T14:26:38.937781+0000","last_became_active":"2024-03-21T14:21:26.767846+0000","last_became_peered":"2024-03-21T14:21:26.767846+0000","last_unstale":"2024-03-21T14:26:38.937781+0000","last_undegraded":"2024-03-21T14:26:38.937781+0000","last_fullsized":"2024-03-21T14:26:38.937781+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'32","last_scrub_stamp":"2024-03-21T14:26:10.357408+0000","last_deep_scrub":"153'16","last_deep_scrub_stamp":"2024-03-21T14:08:30.871984+0000","last_clean_scrub_stamp":"2024-03-21T14:26:10.357408+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:34.385462+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":662,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938372+0000","last_change":"2024-03-21T14:25:42.619333+0000","last_active":"2024-03-21T14:26:38.938372+0000","last_peered":"2024-03-21T14:26:38.938372+0000","last_clean":"2024-03-21T14:26:38.938372+0000","last_became_active":"2024-03-21T14:25:06.187467+0000","last_became_peered":"2024-03-21T14:25:06.187467+0000","last_unstale":"2024-03-21T14:26:38.938372+0000","last_undegraded":"2024-03-21T14:26:38.938372+0000","last_fullsized":"2024-03-21T14:26:38.938372+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:42.619208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:42.619208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.155327+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"480'11","reported_seq":781,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649776+0000","last_change":"2024-03-21T14:25:50.890555+0000","last_active":"2024-03-21T14:26:39.649776+0000","last_peered":"2024-03-21T14:26:39.649776+0000","last_clean":"2024-03-21T14:26:39.649776+0000","last_became_active":"2024-03-21T14:25:33.858686+0000","last_became_peered":"2024-03-21T14:25:33.858686+0000","last_unstale":"2024-03-21T14:26:39.649776+0000","last_undegraded":"2024-03-21T14:26:39.649776+0000","last_fullsized":"2024-03-21T14:26:39.649776+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'11","last_scrub_stamp":"2024-03-21T14:25:50.890406+0000","last_deep_scrub":"480'11","last_deep_scrub_stamp":"2024-03-21T14:25:50.890406+0000","last_clean_scrub_stamp":"2024-03-21T14:25:50.890406+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:19.903307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":980,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:41.121946+0000","last_change":"2024-03-21T14:26:41.121970+0000","last_active":"2024-03-21T14:26:41.121946+0000","last_peered":"2024-03-21T14:26:41.121946+0000","last_clean":"2024-03-21T14:26:41.121946+0000","last_became_active":"2024-03-21T14:19:14.684321+0000","last_became_peered":"2024-03-21T14:19:14.684321+0000","last_unstale":"2024-03-21T14:26:41.121946+0000","last_undegraded":"2024-03-21T14:26:41.121946+0000","last_fullsized":"2024-03-21T14:26:41.121946+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:41.121862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:15.956682+0000","last_clean_scrub_stamp":"2024-03-21T14:26:41.121862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:47.511096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"441'35","reported_seq":1000,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322955+0000","last_change":"2024-03-21T14:26:19.540382+0000","last_active":"2024-03-21T14:26:39.322955+0000","last_peered":"2024-03-21T14:26:39.322955+0000","last_clean":"2024-03-21T14:26:39.322955+0000","last_became_active":"2024-03-21T14:02:35.774846+0000","last_became_peered":"2024-03-21T14:02:35.774846+0000","last_unstale":"2024-03-21T14:26:39.322955+0000","last_undegraded":"2024-03-21T14:26:39.322955+0000","last_fullsized":"2024-03-21T14:26:39.322955+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:26:19.540295+0000","last_deep_scrub":"180'15","last_deep_scrub_stamp":"2024-03-21T14:10:09.765058+0000","last_clean_scrub_stamp":"2024-03-21T14:26:19.540295+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:22.816245+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":933,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938501+0000","last_change":"2024-03-21T14:26:11.504481+0000","last_active":"2024-03-21T14:26:38.938501+0000","last_peered":"2024-03-21T14:26:38.938501+0000","last_clean":"2024-03-21T14:26:38.938501+0000","last_became_active":"2024-03-21T14:19:14.689789+0000","last_became_peered":"2024-03-21T14:19:14.689789+0000","last_unstale":"2024-03-21T14:26:38.938501+0000","last_undegraded":"2024-03-21T14:26:38.938501+0000","last_fullsized":"2024-03-21T14:26:38.938501+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:11.504383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:12:58.642926+0000","last_clean_scrub_stamp":"2024-03-21T14:26:11.504383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.680629+0000","scrub_duration":174,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"393'13","reported_seq":325,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323107+0000","last_change":"2024-03-21T14:26:03.501335+0000","last_active":"2024-03-21T14:26:39.323107+0000","last_peered":"2024-03-21T14:26:39.323107+0000","last_clean":"2024-03-21T14:26:39.323107+0000","last_became_active":"2024-03-21T14:20:21.784249+0000","last_became_peered":"2024-03-21T14:20:21.784249+0000","last_unstale":"2024-03-21T14:26:39.323107+0000","last_undegraded":"2024-03-21T14:26:39.323107+0000","last_fullsized":"2024-03-21T14:26:39.323107+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"393'13","last_scrub_stamp":"2024-03-21T14:26:03.501200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.501200+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:24.623319+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":446,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938736+0000","last_change":"2024-03-21T14:25:24.874294+0000","last_active":"2024-03-21T14:26:38.938736+0000","last_peered":"2024-03-21T14:26:38.938736+0000","last_clean":"2024-03-21T14:26:38.938736+0000","last_became_active":"2024-03-21T14:25:06.179429+0000","last_became_peered":"2024-03-21T14:25:06.179429+0000","last_unstale":"2024-03-21T14:26:38.938736+0000","last_undegraded":"2024-03-21T14:26:38.938736+0000","last_fullsized":"2024-03-21T14:26:38.938736+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:24.874215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:24.874215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.610200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"441'25","reported_seq":777,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649997+0000","last_change":"2024-03-21T14:26:34.619719+0000","last_active":"2024-03-21T14:26:39.649997+0000","last_peered":"2024-03-21T14:26:39.649997+0000","last_clean":"2024-03-21T14:26:39.649997+0000","last_became_active":"2024-03-21T14:25:33.857978+0000","last_became_peered":"2024-03-21T14:25:33.857978+0000","last_unstale":"2024-03-21T14:26:39.649997+0000","last_undegraded":"2024-03-21T14:26:39.649997+0000","last_fullsized":"2024-03-21T14:26:39.649997+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-21T14:26:34.619601+0000","last_deep_scrub":"338'19","last_deep_scrub_stamp":"2024-03-21T14:17:57.556294+0000","last_clean_scrub_stamp":"2024-03-21T14:26:34.619601+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:44.889422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":789,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002365+0000","last_change":"2024-03-21T14:26:04.629299+0000","last_active":"2024-03-21T14:26:26.002365+0000","last_peered":"2024-03-21T14:26:26.002365+0000","last_clean":"2024-03-21T14:26:26.002365+0000","last_became_active":"2024-03-21T14:19:14.694491+0000","last_became_peered":"2024-03-21T14:19:14.694491+0000","last_unstale":"2024-03-21T14:26:26.002365+0000","last_undegraded":"2024-03-21T14:26:26.002365+0000","last_fullsized":"2024-03-21T14:26:26.002365+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:04.629212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:04.629212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.685194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":237,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323690+0000","last_change":"2024-03-21T14:25:57.481292+0000","last_active":"2024-03-21T14:26:39.323690+0000","last_peered":"2024-03-21T14:26:39.323690+0000","last_clean":"2024-03-21T14:26:39.323690+0000","last_became_active":"2024-03-21T14:20:21.787497+0000","last_became_peered":"2024-03-21T14:20:21.787497+0000","last_unstale":"2024-03-21T14:26:39.323690+0000","last_undegraded":"2024-03-21T14:26:39.323690+0000","last_fullsized":"2024-03-21T14:26:39.323690+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:57.481185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:57.481185+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.481185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.296347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":813,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.547924+0000","last_change":"2024-03-21T14:25:40.954500+0000","last_active":"2024-03-21T14:26:39.547924+0000","last_peered":"2024-03-21T14:26:39.547924+0000","last_clean":"2024-03-21T14:26:39.547924+0000","last_became_active":"2024-03-21T14:25:40.881176+0000","last_became_peered":"2024-03-21T14:25:40.881176+0000","last_unstale":"2024-03-21T14:26:39.547924+0000","last_undegraded":"2024-03-21T14:26:39.547924+0000","last_fullsized":"2024-03-21T14:26:39.547924+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:40.954378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:40.954378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:06.553550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"441'25","reported_seq":612,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938668+0000","last_change":"2024-03-21T14:26:32.270529+0000","last_active":"2024-03-21T14:26:38.938668+0000","last_peered":"2024-03-21T14:26:38.938668+0000","last_clean":"2024-03-21T14:26:38.938668+0000","last_became_active":"2024-03-21T14:25:06.191249+0000","last_became_peered":"2024-03-21T14:25:06.191249+0000","last_unstale":"2024-03-21T14:26:38.938668+0000","last_undegraded":"2024-03-21T14:26:38.938668+0000","last_fullsized":"2024-03-21T14:26:38.938668+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'25","last_scrub_stamp":"2024-03-21T14:26:32.270430+0000","last_deep_scrub":"432'19","last_deep_scrub_stamp":"2024-03-21T14:21:49.128273+0000","last_clean_scrub_stamp":"2024-03-21T14:26:32.270430+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:39.435787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":977,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548311+0000","last_change":"2024-03-21T14:25:44.822031+0000","last_active":"2024-03-21T14:26:39.548311+0000","last_peered":"2024-03-21T14:26:39.548311+0000","last_clean":"2024-03-21T14:26:39.548311+0000","last_became_active":"2024-03-21T14:25:44.821221+0000","last_became_peered":"2024-03-21T14:25:44.821221+0000","last_unstale":"2024-03-21T14:26:39.548311+0000","last_undegraded":"2024-03-21T14:26:39.548311+0000","last_fullsized":"2024-03-21T14:26:39.548311+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.065344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.065344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:59.177723+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":584,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649809+0000","last_change":"2024-03-21T14:26:25.639560+0000","last_active":"2024-03-21T14:26:39.649809+0000","last_peered":"2024-03-21T14:26:39.649809+0000","last_clean":"2024-03-21T14:26:39.649809+0000","last_became_active":"2024-03-21T14:25:33.853341+0000","last_became_peered":"2024-03-21T14:25:33.853341+0000","last_unstale":"2024-03-21T14:26:39.649809+0000","last_undegraded":"2024-03-21T14:26:39.649809+0000","last_fullsized":"2024-03-21T14:26:39.649809+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:25.639429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:07.301711+0000","last_clean_scrub_stamp":"2024-03-21T14:26:25.639429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:46.063037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":187,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002345+0000","last_change":"2024-03-21T14:25:21.880898+0000","last_active":"2024-03-21T14:26:26.002345+0000","last_peered":"2024-03-21T14:26:26.002345+0000","last_clean":"2024-03-21T14:26:26.002345+0000","last_became_active":"2024-03-21T14:21:26.765914+0000","last_became_peered":"2024-03-21T14:21:26.765914+0000","last_unstale":"2024-03-21T14:26:26.002345+0000","last_undegraded":"2024-03-21T14:26:26.002345+0000","last_fullsized":"2024-03-21T14:26:26.002345+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:21.880770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:21.880770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:33.461677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"441'18","reported_seq":254,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323341+0000","last_change":"2024-03-21T14:25:59.486475+0000","last_active":"2024-03-21T14:26:39.323341+0000","last_peered":"2024-03-21T14:26:39.323341+0000","last_clean":"2024-03-21T14:26:39.323341+0000","last_became_active":"2024-03-21T14:20:21.787732+0000","last_became_peered":"2024-03-21T14:20:21.787732+0000","last_unstale":"2024-03-21T14:26:39.323341+0000","last_undegraded":"2024-03-21T14:26:39.323341+0000","last_fullsized":"2024-03-21T14:26:39.323341+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'18","last_scrub_stamp":"2024-03-21T14:25:59.486351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:59.486351+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.064493+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":591,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649599+0000","last_change":"2024-03-21T14:25:44.823215+0000","last_active":"2024-03-21T14:26:39.649599+0000","last_peered":"2024-03-21T14:26:39.649599+0000","last_clean":"2024-03-21T14:26:39.649599+0000","last_became_active":"2024-03-21T14:25:44.822423+0000","last_became_peered":"2024-03-21T14:25:44.822423+0000","last_unstale":"2024-03-21T14:26:39.649599+0000","last_undegraded":"2024-03-21T14:26:39.649599+0000","last_fullsized":"2024-03-21T14:26:39.649599+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:28.864058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:22:52.940702+0000","last_clean_scrub_stamp":"2024-03-21T14:25:28.864058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"372'4","reported_seq":1016,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548671+0000","last_change":"2024-03-21T14:26:36.163345+0000","last_active":"2024-03-21T14:26:39.548671+0000","last_peered":"2024-03-21T14:26:39.548671+0000","last_clean":"2024-03-21T14:26:39.548671+0000","last_became_active":"2024-03-21T14:19:34.689863+0000","last_became_peered":"2024-03-21T14:19:34.689863+0000","last_unstale":"2024-03-21T14:26:39.548671+0000","last_undegraded":"2024-03-21T14:26:39.548671+0000","last_fullsized":"2024-03-21T14:26:39.548671+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'4","last_scrub_stamp":"2024-03-21T14:26:36.163244+0000","last_deep_scrub":"372'4","last_deep_scrub_stamp":"2024-03-21T14:25:14.914516+0000","last_clean_scrub_stamp":"2024-03-21T14:26:36.163244+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:41.864856+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"486'46","reported_seq":859,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649161+0000","last_change":"2024-03-21T14:25:46.098239+0000","last_active":"2024-03-21T14:26:39.649161+0000","last_peered":"2024-03-21T14:26:39.649161+0000","last_clean":"2024-03-21T14:26:39.649161+0000","last_became_active":"2024-03-21T14:25:44.829654+0000","last_became_peered":"2024-03-21T14:25:44.829654+0000","last_unstale":"2024-03-21T14:26:39.649161+0000","last_undegraded":"2024-03-21T14:26:39.649161+0000","last_fullsized":"2024-03-21T14:26:39.649161+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'46","last_scrub_stamp":"2024-03-21T14:25:46.098114+0000","last_deep_scrub":"486'46","last_deep_scrub_stamp":"2024-03-21T14:25:46.098114+0000","last_clean_scrub_stamp":"2024-03-21T14:25:46.098114+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:58.383137+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"488'11","reported_seq":1282,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548184+0000","last_change":"2024-03-21T14:25:44.842237+0000","last_active":"2024-03-21T14:26:39.548184+0000","last_peered":"2024-03-21T14:26:39.548184+0000","last_clean":"2024-03-21T14:26:39.548184+0000","last_became_active":"2024-03-21T14:25:44.824707+0000","last_became_peered":"2024-03-21T14:25:44.824707+0000","last_unstale":"2024-03-21T14:26:39.548184+0000","last_undegraded":"2024-03-21T14:26:39.548184+0000","last_fullsized":"2024-03-21T14:26:39.548184+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'10","last_scrub_stamp":"2024-03-21T14:25:30.037228+0000","last_deep_scrub":"372'10","last_deep_scrub_stamp":"2024-03-21T14:25:30.037228+0000","last_clean_scrub_stamp":"2024-03-21T14:25:30.037228+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:56.824245+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":696,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322984+0000","last_change":"2024-03-21T14:25:55.625669+0000","last_active":"2024-03-21T14:26:39.322984+0000","last_peered":"2024-03-21T14:26:39.322984+0000","last_clean":"2024-03-21T14:26:39.322984+0000","last_became_active":"2024-03-21T14:10:53.017161+0000","last_became_peered":"2024-03-21T14:10:53.017161+0000","last_unstale":"2024-03-21T14:26:39.322984+0000","last_undegraded":"2024-03-21T14:26:39.322984+0000","last_fullsized":"2024-03-21T14:26:39.322984+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:55.625575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:55.625575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:24.833630+0000","scrub_duration":135,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":994,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548497+0000","last_change":"2024-03-21T14:26:12.891340+0000","last_active":"2024-03-21T14:26:39.548497+0000","last_peered":"2024-03-21T14:26:39.548497+0000","last_clean":"2024-03-21T14:26:39.548497+0000","last_became_active":"2024-03-21T14:02:33.447831+0000","last_became_peered":"2024-03-21T14:02:33.447831+0000","last_unstale":"2024-03-21T14:26:39.548497+0000","last_undegraded":"2024-03-21T14:26:39.548497+0000","last_fullsized":"2024-03-21T14:26:39.548497+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:12.891238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:12.891238+0000","last_clean_scrub_stamp":"2024-03-21T14:26:12.891238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:18.616398+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"488'29","reported_seq":1042,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937731+0000","last_change":"2024-03-21T14:25:50.493067+0000","last_active":"2024-03-21T14:26:38.937731+0000","last_peered":"2024-03-21T14:26:38.937731+0000","last_clean":"2024-03-21T14:26:38.937731+0000","last_became_active":"2024-03-21T14:20:21.782208+0000","last_became_peered":"2024-03-21T14:20:21.782208+0000","last_unstale":"2024-03-21T14:26:38.937731+0000","last_undegraded":"2024-03-21T14:26:38.937731+0000","last_fullsized":"2024-03-21T14:26:38.937731+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"488'29","last_scrub_stamp":"2024-03-21T14:25:50.492941+0000","last_deep_scrub":"488'29","last_deep_scrub_stamp":"2024-03-21T14:25:50.492941+0000","last_clean_scrub_stamp":"2024-03-21T14:25:50.492941+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:59.184709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":801,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001408+0000","last_change":"2024-03-21T14:26:08.620478+0000","last_active":"2024-03-21T14:26:26.001408+0000","last_peered":"2024-03-21T14:26:26.001408+0000","last_clean":"2024-03-21T14:26:26.001408+0000","last_became_active":"2024-03-21T14:19:14.698780+0000","last_became_peered":"2024-03-21T14:19:14.698780+0000","last_unstale":"2024-03-21T14:26:26.001408+0000","last_undegraded":"2024-03-21T14:26:26.001408+0000","last_fullsized":"2024-03-21T14:26:26.001408+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:08.620360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:09:28.007714+0000","last_clean_scrub_stamp":"2024-03-21T14:26:08.620360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:13.936929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":575,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.547523+0000","last_change":"2024-03-21T14:25:41.133611+0000","last_active":"2024-03-21T14:26:39.547523+0000","last_peered":"2024-03-21T14:26:39.547523+0000","last_clean":"2024-03-21T14:26:39.547523+0000","last_became_active":"2024-03-21T14:25:41.133266+0000","last_became_peered":"2024-03-21T14:25:41.133266+0000","last_unstale":"2024-03-21T14:26:39.547523+0000","last_undegraded":"2024-03-21T14:26:39.547523+0000","last_fullsized":"2024-03-21T14:26:39.547523+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:15.472742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:15.472742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":754,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001976+0000","last_change":"2024-03-21T14:25:24.473276+0000","last_active":"2024-03-21T14:26:26.001976+0000","last_peered":"2024-03-21T14:26:26.001976+0000","last_clean":"2024-03-21T14:26:26.001976+0000","last_became_active":"2024-03-21T14:19:14.682302+0000","last_became_peered":"2024-03-21T14:19:14.682302+0000","last_unstale":"2024-03-21T14:26:26.001976+0000","last_undegraded":"2024-03-21T14:26:26.001976+0000","last_fullsized":"2024-03-21T14:26:26.001976+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:24.473195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:24.473195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:38.561497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":606,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548497+0000","last_change":"2024-03-21T14:25:45.950388+0000","last_active":"2024-03-21T14:26:39.548497+0000","last_peered":"2024-03-21T14:26:39.548497+0000","last_clean":"2024-03-21T14:26:39.548497+0000","last_became_active":"2024-03-21T14:25:40.881274+0000","last_became_peered":"2024-03-21T14:25:40.881274+0000","last_unstale":"2024-03-21T14:26:39.548497+0000","last_undegraded":"2024-03-21T14:26:39.548497+0000","last_fullsized":"2024-03-21T14:26:39.548497+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:45.950289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:45.950289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:05.031215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"441'60","reported_seq":59,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:37.742237+0000","last_change":"2024-03-21T14:26:37.742292+0000","last_active":"2024-03-21T14:26:37.742237+0000","last_peered":"2024-03-21T14:26:37.742237+0000","last_clean":"2024-03-21T14:26:37.742237+0000","last_became_active":"2024-03-21T14:25:44.828035+0000","last_became_peered":"2024-03-21T14:25:44.828035+0000","last_unstale":"2024-03-21T14:26:37.742237+0000","last_undegraded":"2024-03-21T14:26:37.742237+0000","last_fullsized":"2024-03-21T14:26:37.742237+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'60","last_scrub_stamp":"2024-03-21T14:26:37.742168+0000","last_deep_scrub":"237'44","last_deep_scrub_stamp":"2024-03-21T14:20:49.145792+0000","last_clean_scrub_stamp":"2024-03-21T14:26:37.742168+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:50.334467+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":97,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938902+0000","last_change":"2024-03-21T14:26:34.416111+0000","last_active":"2024-03-21T14:26:38.938902+0000","last_peered":"2024-03-21T14:26:38.938902+0000","last_clean":"2024-03-21T14:26:38.938902+0000","last_became_active":"2024-03-21T14:25:06.182766+0000","last_became_peered":"2024-03-21T14:25:06.182766+0000","last_unstale":"2024-03-21T14:26:38.938902+0000","last_undegraded":"2024-03-21T14:26:38.938902+0000","last_fullsized":"2024-03-21T14:26:38.938902+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:34.416008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:34.416008+0000","last_clean_scrub_stamp":"2024-03-21T14:26:34.416008+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:50.636577+0000","scrub_duration":139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":801,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.001808+0000","last_change":"2024-03-21T14:26:13.569734+0000","last_active":"2024-03-21T14:26:26.001808+0000","last_peered":"2024-03-21T14:26:26.001808+0000","last_clean":"2024-03-21T14:26:26.001808+0000","last_became_active":"2024-03-21T14:19:14.690218+0000","last_became_peered":"2024-03-21T14:19:14.690218+0000","last_unstale":"2024-03-21T14:26:26.001808+0000","last_undegraded":"2024-03-21T14:26:26.001808+0000","last_fullsized":"2024-03-21T14:26:26.001808+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:13.569654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:41.007598+0000","last_clean_scrub_stamp":"2024-03-21T14:26:13.569654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:43.442676+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":72,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938430+0000","last_change":"2024-03-21T14:25:53.649028+0000","last_active":"2024-03-21T14:26:38.938430+0000","last_peered":"2024-03-21T14:26:38.938430+0000","last_clean":"2024-03-21T14:26:38.938430+0000","last_became_active":"2024-03-21T14:25:06.179804+0000","last_became_peered":"2024-03-21T14:25:06.179804+0000","last_unstale":"2024-03-21T14:26:38.938430+0000","last_undegraded":"2024-03-21T14:26:38.938430+0000","last_fullsized":"2024-03-21T14:26:38.938430+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:53.648939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:53.648939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:04.581288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"441'21","reported_seq":407,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.939495+0000","last_change":"2024-03-21T14:25:47.531439+0000","last_active":"2024-03-21T14:26:38.939495+0000","last_peered":"2024-03-21T14:26:38.939495+0000","last_clean":"2024-03-21T14:26:38.939495+0000","last_became_active":"2024-03-21T14:25:06.185191+0000","last_became_peered":"2024-03-21T14:25:06.185191+0000","last_unstale":"2024-03-21T14:26:38.939495+0000","last_undegraded":"2024-03-21T14:26:38.939495+0000","last_fullsized":"2024-03-21T14:26:38.939495+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'21","last_scrub_stamp":"2024-03-21T14:25:47.531335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:47.531335+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:17.385156+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"167'130","reported_seq":758,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649148+0000","last_change":"2024-03-21T14:26:24.687482+0000","last_active":"2024-03-21T14:26:39.649148+0000","last_peered":"2024-03-21T14:26:39.649148+0000","last_clean":"2024-03-21T14:26:39.649148+0000","last_became_active":"2024-03-21T14:25:33.853161+0000","last_became_peered":"2024-03-21T14:25:33.853161+0000","last_unstale":"2024-03-21T14:26:39.649148+0000","last_undegraded":"2024-03-21T14:26:39.649148+0000","last_fullsized":"2024-03-21T14:26:39.649148+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:24.687341+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:24.687341+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:48.649683+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":943,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323473+0000","last_change":"2024-03-21T14:26:15.627527+0000","last_active":"2024-03-21T14:26:39.323473+0000","last_peered":"2024-03-21T14:26:39.323473+0000","last_clean":"2024-03-21T14:26:39.323473+0000","last_became_active":"2024-03-21T14:02:39.472206+0000","last_became_peered":"2024-03-21T14:02:39.472206+0000","last_unstale":"2024-03-21T14:26:39.323473+0000","last_undegraded":"2024-03-21T14:26:39.323473+0000","last_fullsized":"2024-03-21T14:26:39.323473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:15.627438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:15.627438+0000","last_clean_scrub_stamp":"2024-03-21T14:26:15.627438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:45.401433+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":624,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:41.706222+0000","last_change":"2024-03-21T14:26:41.706252+0000","last_active":"2024-03-21T14:26:41.706222+0000","last_peered":"2024-03-21T14:26:41.706222+0000","last_clean":"2024-03-21T14:26:41.706222+0000","last_became_active":"2024-03-21T14:25:44.822898+0000","last_became_peered":"2024-03-21T14:25:44.822898+0000","last_unstale":"2024-03-21T14:26:41.706222+0000","last_undegraded":"2024-03-21T14:26:41.706222+0000","last_fullsized":"2024-03-21T14:26:41.706222+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-03-21T14:26:41.706157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:31.423682+0000","last_clean_scrub_stamp":"2024-03-21T14:26:41.706157+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:55.076023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089371800000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":958,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937615+0000","last_change":"2024-03-21T14:26:36.345770+0000","last_active":"2024-03-21T14:26:38.937615+0000","last_peered":"2024-03-21T14:26:38.937615+0000","last_clean":"2024-03-21T14:26:38.937615+0000","last_became_active":"2024-03-21T14:25:44.819631+0000","last_became_peered":"2024-03-21T14:25:44.819631+0000","last_unstale":"2024-03-21T14:26:38.937615+0000","last_undegraded":"2024-03-21T14:26:38.937615+0000","last_fullsized":"2024-03-21T14:26:38.937615+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:36.345679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:14.635743+0000","last_clean_scrub_stamp":"2024-03-21T14:26:36.345679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:43.092044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"480'67","reported_seq":844,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649612+0000","last_change":"2024-03-21T14:26:37.632036+0000","last_active":"2024-03-21T14:26:39.649612+0000","last_peered":"2024-03-21T14:26:39.649612+0000","last_clean":"2024-03-21T14:26:39.649612+0000","last_became_active":"2024-03-21T14:25:33.853554+0000","last_became_peered":"2024-03-21T14:25:33.853554+0000","last_unstale":"2024-03-21T14:26:39.649612+0000","last_undegraded":"2024-03-21T14:26:39.649612+0000","last_fullsized":"2024-03-21T14:26:39.649612+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'67","last_scrub_stamp":"2024-03-21T14:26:37.631844+0000","last_deep_scrub":"480'67","last_deep_scrub_stamp":"2024-03-21T14:26:37.631844+0000","last_clean_scrub_stamp":"2024-03-21T14:26:37.631844+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:49.852068+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":602,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938214+0000","last_change":"2024-03-21T14:26:03.590862+0000","last_active":"2024-03-21T14:26:38.938214+0000","last_peered":"2024-03-21T14:26:38.938214+0000","last_clean":"2024-03-21T14:26:38.938214+0000","last_became_active":"2024-03-21T14:25:06.179299+0000","last_became_peered":"2024-03-21T14:25:06.179299+0000","last_unstale":"2024-03-21T14:26:38.938214+0000","last_undegraded":"2024-03-21T14:26:38.938214+0000","last_fullsized":"2024-03-21T14:26:38.938214+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:03.590757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:03.590757+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.590757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:28.253491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"167'130","reported_seq":903,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.547796+0000","last_change":"2024-03-21T14:26:34.172768+0000","last_active":"2024-03-21T14:26:39.547796+0000","last_peered":"2024-03-21T14:26:39.547796+0000","last_clean":"2024-03-21T14:26:39.547796+0000","last_became_active":"2024-03-21T14:25:40.881368+0000","last_became_peered":"2024-03-21T14:25:40.881368+0000","last_unstale":"2024-03-21T14:26:39.547796+0000","last_undegraded":"2024-03-21T14:26:39.547796+0000","last_fullsized":"2024-03-21T14:26:39.547796+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:34.172654+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:34.172654+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:49.615631+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":972,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548597+0000","last_change":"2024-03-21T14:25:57.129592+0000","last_active":"2024-03-21T14:26:39.548597+0000","last_peered":"2024-03-21T14:26:39.548597+0000","last_clean":"2024-03-21T14:26:39.548597+0000","last_became_active":"2024-03-21T14:25:44.822674+0000","last_became_peered":"2024-03-21T14:25:44.822674+0000","last_unstale":"2024-03-21T14:26:39.548597+0000","last_undegraded":"2024-03-21T14:26:39.548597+0000","last_fullsized":"2024-03-21T14:26:39.548597+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:57.129466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.129466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:57.817089+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":975,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895222+0000","last_change":"2024-03-21T14:26:14.513221+0000","last_active":"2024-03-21T14:26:25.895222+0000","last_peered":"2024-03-21T14:26:25.895222+0000","last_clean":"2024-03-21T14:26:25.895222+0000","last_became_active":"2024-03-21T14:25:44.823980+0000","last_became_peered":"2024-03-21T14:25:44.823980+0000","last_unstale":"2024-03-21T14:26:25.895222+0000","last_undegraded":"2024-03-21T14:26:25.895222+0000","last_fullsized":"2024-03-21T14:26:25.895222+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:14.513144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:14.513144+0000","last_clean_scrub_stamp":"2024-03-21T14:26:14.513144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.322255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":804,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:32.630490+0000","last_change":"2024-03-21T14:26:32.630513+0000","last_active":"2024-03-21T14:26:32.630490+0000","last_peered":"2024-03-21T14:26:32.630490+0000","last_clean":"2024-03-21T14:26:32.630490+0000","last_became_active":"2024-03-21T14:19:14.696464+0000","last_became_peered":"2024-03-21T14:19:14.696464+0000","last_unstale":"2024-03-21T14:26:32.630490+0000","last_undegraded":"2024-03-21T14:26:32.630490+0000","last_fullsized":"2024-03-21T14:26:32.630490+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:32.630425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:29.414864+0000","last_clean_scrub_stamp":"2024-03-21T14:26:32.630425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:42.132480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"441'26","reported_seq":1053,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897156+0000","last_change":"2024-03-21T14:25:51.672227+0000","last_active":"2024-03-21T14:26:25.897156+0000","last_peered":"2024-03-21T14:26:25.897156+0000","last_clean":"2024-03-21T14:26:25.897156+0000","last_became_active":"2024-03-21T14:19:34.695528+0000","last_became_peered":"2024-03-21T14:19:34.695528+0000","last_unstale":"2024-03-21T14:26:25.897156+0000","last_undegraded":"2024-03-21T14:26:25.897156+0000","last_fullsized":"2024-03-21T14:26:25.897156+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'26","last_scrub_stamp":"2024-03-21T14:25:51.672111+0000","last_deep_scrub":"236'22","last_deep_scrub_stamp":"2024-03-21T14:21:47.628002+0000","last_clean_scrub_stamp":"2024-03-21T14:25:51.672111+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:18.811171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":779,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002344+0000","last_change":"2024-03-21T14:26:06.660489+0000","last_active":"2024-03-21T14:26:26.002344+0000","last_peered":"2024-03-21T14:26:26.002344+0000","last_clean":"2024-03-21T14:26:26.002344+0000","last_became_active":"2024-03-21T14:19:14.683146+0000","last_became_peered":"2024-03-21T14:19:14.683146+0000","last_unstale":"2024-03-21T14:26:26.002344+0000","last_undegraded":"2024-03-21T14:26:26.002344+0000","last_fullsized":"2024-03-21T14:26:26.002344+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:06.660395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:49.614290+0000","last_clean_scrub_stamp":"2024-03-21T14:26:06.660395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:15.205994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":573,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.650072+0000","last_change":"2024-03-21T14:26:29.779112+0000","last_active":"2024-03-21T14:26:39.650072+0000","last_peered":"2024-03-21T14:26:39.650072+0000","last_clean":"2024-03-21T14:26:39.650072+0000","last_became_active":"2024-03-21T14:25:33.853793+0000","last_became_peered":"2024-03-21T14:25:33.853793+0000","last_unstale":"2024-03-21T14:26:39.650072+0000","last_undegraded":"2024-03-21T14:26:39.650072+0000","last_fullsized":"2024-03-21T14:26:39.650072+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:29.778995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:26:29.778995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:42.549764+0000","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"167'130","reported_seq":1287,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.547654+0000","last_change":"2024-03-21T14:25:47.821305+0000","last_active":"2024-03-21T14:26:39.547654+0000","last_peered":"2024-03-21T14:26:39.547654+0000","last_clean":"2024-03-21T14:26:39.547654+0000","last_became_active":"2024-03-21T14:25:47.820896+0000","last_became_peered":"2024-03-21T14:25:47.820896+0000","last_unstale":"2024-03-21T14:26:39.547654+0000","last_undegraded":"2024-03-21T14:26:39.547654+0000","last_fullsized":"2024-03-21T14:26:39.547654+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":0,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:22.065873+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:22.065873+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.877713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":975,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322687+0000","last_change":"2024-03-21T14:25:44.831295+0000","last_active":"2024-03-21T14:26:39.322687+0000","last_peered":"2024-03-21T14:26:39.322687+0000","last_clean":"2024-03-21T14:26:39.322687+0000","last_became_active":"2024-03-21T14:25:44.830942+0000","last_became_peered":"2024-03-21T14:25:44.830942+0000","last_unstale":"2024-03-21T14:26:39.322687+0000","last_undegraded":"2024-03-21T14:26:39.322687+0000","last_fullsized":"2024-03-21T14:26:39.322687+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.320710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:16:20.564321+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.320710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:55.779155+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"488'42","reported_seq":60,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:33.777073+0000","last_change":"2024-03-21T14:25:44.850008+0000","last_active":"2024-03-21T14:26:33.777073+0000","last_peered":"2024-03-21T14:26:33.777073+0000","last_clean":"2024-03-21T14:26:33.777073+0000","last_became_active":"2024-03-21T14:25:44.829253+0000","last_became_peered":"2024-03-21T14:25:44.829253+0000","last_unstale":"2024-03-21T14:26:33.777073+0000","last_undegraded":"2024-03-21T14:26:33.777073+0000","last_fullsized":"2024-03-21T14:26:33.777073+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'41","last_scrub_stamp":"2024-03-21T14:25:33.551480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:33.551480+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:53.122097+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":673,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548265+0000","last_change":"2024-03-21T14:25:40.882008+0000","last_active":"2024-03-21T14:26:39.548265+0000","last_peered":"2024-03-21T14:26:39.548265+0000","last_clean":"2024-03-21T14:26:39.548265+0000","last_became_active":"2024-03-21T14:25:40.881519+0000","last_became_peered":"2024-03-21T14:25:40.881519+0000","last_unstale":"2024-03-21T14:26:39.548265+0000","last_undegraded":"2024-03-21T14:26:39.548265+0000","last_fullsized":"2024-03-21T14:26:39.548265+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:19.476852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:19.476852+0000","last_clean_scrub_stamp":"2024-03-21T14:25:19.476852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:40.161115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"167'130","reported_seq":1040,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895876+0000","last_change":"2024-03-21T14:26:09.521260+0000","last_active":"2024-03-21T14:26:25.895876+0000","last_peered":"2024-03-21T14:26:25.895876+0000","last_clean":"2024-03-21T14:26:25.895876+0000","last_became_active":"2024-03-21T14:25:47.816308+0000","last_became_peered":"2024-03-21T14:25:47.816308+0000","last_unstale":"2024-03-21T14:26:25.895876+0000","last_undegraded":"2024-03-21T14:26:25.895876+0000","last_fullsized":"2024-03-21T14:26:25.895876+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:09.521162+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:09.521162+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:33.929416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":967,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896246+0000","last_change":"2024-03-21T14:25:35.587679+0000","last_active":"2024-03-21T14:26:25.896246+0000","last_peered":"2024-03-21T14:26:25.896246+0000","last_clean":"2024-03-21T14:26:25.896246+0000","last_became_active":"2024-03-21T14:15:59.602322+0000","last_became_peered":"2024-03-21T14:15:59.602322+0000","last_unstale":"2024-03-21T14:26:25.896246+0000","last_undegraded":"2024-03-21T14:26:25.896246+0000","last_fullsized":"2024-03-21T14:26:25.896246+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:35.587577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:13:04.716819+0000","last_clean_scrub_stamp":"2024-03-21T14:25:35.587577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:52.412718+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":596,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.939040+0000","last_change":"2024-03-21T14:26:33.235630+0000","last_active":"2024-03-21T14:26:38.939040+0000","last_peered":"2024-03-21T14:26:38.939040+0000","last_clean":"2024-03-21T14:26:38.939040+0000","last_became_active":"2024-03-21T14:25:06.179126+0000","last_became_peered":"2024-03-21T14:25:06.179126+0000","last_unstale":"2024-03-21T14:26:38.939040+0000","last_undegraded":"2024-03-21T14:26:38.939040+0000","last_fullsized":"2024-03-21T14:26:38.939040+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:33.235531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:33.235531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:43.277062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"290'6","reported_seq":1508,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:40.975451+0000","last_change":"2024-03-21T14:26:07.020323+0000","last_active":"2024-03-21T14:26:40.975451+0000","last_peered":"2024-03-21T14:26:40.975451+0000","last_clean":"2024-03-21T14:26:40.975451+0000","last_became_active":"2024-03-21T14:25:40.879713+0000","last_became_peered":"2024-03-21T14:25:40.879713+0000","last_unstale":"2024-03-21T14:26:40.975451+0000","last_undegraded":"2024-03-21T14:26:40.975451+0000","last_fullsized":"2024-03-21T14:26:40.975451+0000","mapping_epoch":482,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'6","last_scrub_stamp":"2024-03-21T14:26:07.020164+0000","last_deep_scrub":"290'6","last_deep_scrub_stamp":"2024-03-21T14:15:37.009777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:07.020164+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:25.549813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00044230099999999998,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":893,"num_read_kb":888,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"441'36","reported_seq":856,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938263+0000","last_change":"2024-03-21T14:26:16.500849+0000","last_active":"2024-03-21T14:26:38.938263+0000","last_peered":"2024-03-21T14:26:38.938263+0000","last_clean":"2024-03-21T14:26:38.938263+0000","last_became_active":"2024-03-21T14:25:06.185735+0000","last_became_peered":"2024-03-21T14:25:06.185735+0000","last_unstale":"2024-03-21T14:26:38.938263+0000","last_undegraded":"2024-03-21T14:26:38.938263+0000","last_fullsized":"2024-03-21T14:26:38.938263+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'36","last_scrub_stamp":"2024-03-21T14:26:16.500753+0000","last_deep_scrub":"441'36","last_deep_scrub_stamp":"2024-03-21T14:22:51.893561+0000","last_clean_scrub_stamp":"2024-03-21T14:26:16.500753+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.770255+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":46,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":585,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.650264+0000","last_change":"2024-03-21T14:25:40.151087+0000","last_active":"2024-03-21T14:26:39.650264+0000","last_peered":"2024-03-21T14:26:39.650264+0000","last_clean":"2024-03-21T14:26:39.650264+0000","last_became_active":"2024-03-21T14:25:33.852775+0000","last_became_peered":"2024-03-21T14:25:33.852775+0000","last_unstale":"2024-03-21T14:26:39.650264+0000","last_undegraded":"2024-03-21T14:26:39.650264+0000","last_fullsized":"2024-03-21T14:26:39.650264+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:40.150980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:11:06.946833+0000","last_clean_scrub_stamp":"2024-03-21T14:25:40.150980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:47.587288+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"167'130","reported_seq":604,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003935+0000","last_change":"2024-03-21T14:26:21.630400+0000","last_active":"2024-03-21T14:26:26.003935+0000","last_peered":"2024-03-21T14:26:26.003935+0000","last_clean":"2024-03-21T14:26:26.003935+0000","last_became_active":"2024-03-21T14:25:47.819834+0000","last_became_peered":"2024-03-21T14:25:47.819834+0000","last_unstale":"2024-03-21T14:26:26.003935+0000","last_undegraded":"2024-03-21T14:26:26.003935+0000","last_fullsized":"2024-03-21T14:26:26.003935+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:21.630293+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:26:21.630293+0000","last_clean_scrub_stamp":"2024-03-21T14:26:21.630293+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.490949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.895520+0000","last_change":"2024-03-21T14:25:44.820606+0000","last_active":"2024-03-21T14:26:25.895520+0000","last_peered":"2024-03-21T14:26:25.895520+0000","last_clean":"2024-03-21T14:26:25.895520+0000","last_became_active":"2024-03-21T14:25:44.820252+0000","last_became_peered":"2024-03-21T14:25:44.820252+0000","last_unstale":"2024-03-21T14:26:25.895520+0000","last_undegraded":"2024-03-21T14:26:25.895520+0000","last_fullsized":"2024-03-21T14:26:25.895520+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.610372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:24:05.738523+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.610372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:51.054393+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":970,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896470+0000","last_change":"2024-03-21T14:26:23.540870+0000","last_active":"2024-03-21T14:26:25.896470+0000","last_peered":"2024-03-21T14:26:25.896470+0000","last_clean":"2024-03-21T14:26:25.896470+0000","last_became_active":"2024-03-21T14:19:14.694193+0000","last_became_peered":"2024-03-21T14:19:14.694193+0000","last_unstale":"2024-03-21T14:26:25.896470+0000","last_undegraded":"2024-03-21T14:26:25.896470+0000","last_fullsized":"2024-03-21T14:26:25.896470+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:23.540724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:26:23.540724+0000","last_clean_scrub_stamp":"2024-03-21T14:26:23.540724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:40.249226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"485'47","reported_seq":656,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.939472+0000","last_change":"2024-03-21T14:25:45.611105+0000","last_active":"2024-03-21T14:26:38.939472+0000","last_peered":"2024-03-21T14:26:38.939472+0000","last_clean":"2024-03-21T14:26:38.939472+0000","last_became_active":"2024-03-21T14:25:06.190350+0000","last_became_peered":"2024-03-21T14:25:06.190350+0000","last_unstale":"2024-03-21T14:26:38.939472+0000","last_undegraded":"2024-03-21T14:26:38.939472+0000","last_fullsized":"2024-03-21T14:26:38.939472+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'47","last_scrub_stamp":"2024-03-21T14:25:45.611005+0000","last_deep_scrub":"485'47","last_deep_scrub_stamp":"2024-03-21T14:25:45.611005+0000","last_clean_scrub_stamp":"2024-03-21T14:25:45.611005+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:58.728839+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":67,"num_read_kb":53,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":31,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003076+0000","last_change":"2024-03-21T14:25:44.833182+0000","last_active":"2024-03-21T14:26:26.003076+0000","last_peered":"2024-03-21T14:26:26.003076+0000","last_clean":"2024-03-21T14:26:26.003076+0000","last_became_active":"2024-03-21T14:25:44.832497+0000","last_became_peered":"2024-03-21T14:25:44.832497+0000","last_unstale":"2024-03-21T14:26:26.003076+0000","last_undegraded":"2024-03-21T14:26:26.003076+0000","last_fullsized":"2024-03-21T14:26:26.003076+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:34.759171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:08:32.024362+0000","last_clean_scrub_stamp":"2024-03-21T14:25:34.759171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:41.617355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"167'130","reported_seq":936,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.003837+0000","last_change":"2024-03-21T14:25:49.680443+0000","last_active":"2024-03-21T14:26:26.003837+0000","last_peered":"2024-03-21T14:26:26.003837+0000","last_clean":"2024-03-21T14:26:26.003837+0000","last_became_active":"2024-03-21T14:25:47.819406+0000","last_became_peered":"2024-03-21T14:25:47.819406+0000","last_unstale":"2024-03-21T14:26:26.003837+0000","last_undegraded":"2024-03-21T14:26:26.003837+0000","last_fullsized":"2024-03-21T14:26:26.003837+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:49.680292+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:49.680292+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:08.330411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":434,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938397+0000","last_change":"2024-03-21T14:25:38.717994+0000","last_active":"2024-03-21T14:26:38.938397+0000","last_peered":"2024-03-21T14:26:38.938397+0000","last_clean":"2024-03-21T14:26:38.938397+0000","last_became_active":"2024-03-21T14:25:06.181080+0000","last_became_peered":"2024-03-21T14:25:06.181080+0000","last_unstale":"2024-03-21T14:26:38.938397+0000","last_undegraded":"2024-03-21T14:26:38.938397+0000","last_fullsized":"2024-03-21T14:26:38.938397+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:38.717867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:38.717867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:08.476649+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":636,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649904+0000","last_change":"2024-03-21T14:25:43.156304+0000","last_active":"2024-03-21T14:26:39.649904+0000","last_peered":"2024-03-21T14:26:39.649904+0000","last_clean":"2024-03-21T14:26:39.649904+0000","last_became_active":"2024-03-21T14:25:33.853651+0000","last_became_peered":"2024-03-21T14:25:33.853651+0000","last_unstale":"2024-03-21T14:26:39.649904+0000","last_undegraded":"2024-03-21T14:26:39.649904+0000","last_fullsized":"2024-03-21T14:26:39.649904+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:43.156192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:43.156192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:07.984041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"441'35","reported_seq":1015,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896011+0000","last_change":"2024-03-21T14:26:00.607520+0000","last_active":"2024-03-21T14:26:25.896011+0000","last_peered":"2024-03-21T14:26:25.896011+0000","last_clean":"2024-03-21T14:26:25.896011+0000","last_became_active":"2024-03-21T14:02:35.453177+0000","last_became_peered":"2024-03-21T14:02:35.453177+0000","last_unstale":"2024-03-21T14:26:25.896011+0000","last_undegraded":"2024-03-21T14:26:25.896011+0000","last_fullsized":"2024-03-21T14:26:25.896011+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:26:00.607405+0000","last_deep_scrub":"152'15","last_deep_scrub_stamp":"2024-03-21T14:08:25.164008+0000","last_clean_scrub_stamp":"2024-03-21T14:26:00.607405+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:25.304382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"488'14","reported_seq":598,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.939610+0000","last_change":"2024-03-21T14:25:44.845004+0000","last_active":"2024-03-21T14:26:38.939610+0000","last_peered":"2024-03-21T14:26:38.939610+0000","last_clean":"2024-03-21T14:26:38.939610+0000","last_became_active":"2024-03-21T14:25:44.828780+0000","last_became_peered":"2024-03-21T14:25:44.828780+0000","last_unstale":"2024-03-21T14:26:38.939610+0000","last_undegraded":"2024-03-21T14:26:38.939610+0000","last_fullsized":"2024-03-21T14:26:38.939610+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'13","last_scrub_stamp":"2024-03-21T14:25:33.790676+0000","last_deep_scrub":"479'13","last_deep_scrub_stamp":"2024-03-21T14:25:33.790676+0000","last_clean_scrub_stamp":"2024-03-21T14:25:33.790676+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.957223+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"167'130","reported_seq":1017,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322799+0000","last_change":"2024-03-21T14:26:05.476597+0000","last_active":"2024-03-21T14:26:39.322799+0000","last_peered":"2024-03-21T14:26:39.322799+0000","last_clean":"2024-03-21T14:26:39.322799+0000","last_became_active":"2024-03-21T14:14:35.588439+0000","last_became_peered":"2024-03-21T14:14:35.588439+0000","last_unstale":"2024-03-21T14:26:39.322799+0000","last_undegraded":"2024-03-21T14:26:39.322799+0000","last_fullsized":"2024-03-21T14:26:39.322799+0000","mapping_epoch":273,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":274,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:05.476458+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:05.476458+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.292381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":972,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.548132+0000","last_change":"2024-03-21T14:25:55.109936+0000","last_active":"2024-03-21T14:26:39.548132+0000","last_peered":"2024-03-21T14:26:39.548132+0000","last_clean":"2024-03-21T14:26:39.548132+0000","last_became_active":"2024-03-21T14:25:44.820219+0000","last_became_peered":"2024-03-21T14:25:44.820219+0000","last_unstale":"2024-03-21T14:26:39.548132+0000","last_undegraded":"2024-03-21T14:26:39.548132+0000","last_fullsized":"2024-03-21T14:26:39.548132+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:55.109799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:25:55.109799+0000","last_clean_scrub_stamp":"2024-03-21T14:25:55.109799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:15.368434+0000","scrub_duration":126,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"243'2","reported_seq":600,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938145+0000","last_change":"2024-03-21T14:25:57.609618+0000","last_active":"2024-03-21T14:26:38.938145+0000","last_peered":"2024-03-21T14:26:38.938145+0000","last_clean":"2024-03-21T14:26:38.938145+0000","last_became_active":"2024-03-21T14:25:06.185624+0000","last_became_peered":"2024-03-21T14:25:06.185624+0000","last_unstale":"2024-03-21T14:26:38.938145+0000","last_undegraded":"2024-03-21T14:26:38.938145+0000","last_fullsized":"2024-03-21T14:26:38.938145+0000","mapping_epoch":470,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"243'2","last_scrub_stamp":"2024-03-21T14:25:57.609485+0000","last_deep_scrub":"243'2","last_deep_scrub_stamp":"2024-03-21T14:13:02.667511+0000","last_clean_scrub_stamp":"2024-03-21T14:25:57.609485+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:27.358450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"441'42","reported_seq":1049,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937502+0000","last_change":"2024-03-21T14:25:18.618565+0000","last_active":"2024-03-21T14:26:38.937502+0000","last_peered":"2024-03-21T14:26:38.937502+0000","last_clean":"2024-03-21T14:26:38.937502+0000","last_became_active":"2024-03-21T14:21:26.768622+0000","last_became_peered":"2024-03-21T14:21:26.768622+0000","last_unstale":"2024-03-21T14:26:38.937502+0000","last_undegraded":"2024-03-21T14:26:38.937502+0000","last_fullsized":"2024-03-21T14:26:38.937502+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'42","last_scrub_stamp":"2024-03-21T14:25:18.618422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:18.618422+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"72'8","reported_seq":1246,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323065+0000","last_change":"2024-03-21T14:25:32.205688+0000","last_active":"2024-03-21T14:26:39.323065+0000","last_peered":"2024-03-21T14:26:39.323065+0000","last_clean":"2024-03-21T14:26:39.323065+0000","last_became_active":"2024-03-21T14:02:37.457478+0000","last_became_peered":"2024-03-21T14:02:37.457478+0000","last_unstale":"2024-03-21T14:26:39.323065+0000","last_undegraded":"2024-03-21T14:26:39.323065+0000","last_fullsized":"2024-03-21T14:26:39.323065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:25:32.205594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:32.205594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:45.179180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"167'130","reported_seq":1028,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938070+0000","last_change":"2024-03-21T14:26:03.442579+0000","last_active":"2024-03-21T14:26:38.938070+0000","last_peered":"2024-03-21T14:26:38.938070+0000","last_clean":"2024-03-21T14:26:38.938070+0000","last_became_active":"2024-03-21T14:25:47.815743+0000","last_became_peered":"2024-03-21T14:25:47.815743+0000","last_unstale":"2024-03-21T14:26:38.938070+0000","last_undegraded":"2024-03-21T14:26:38.938070+0000","last_fullsized":"2024-03-21T14:26:38.938070+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":3,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:03.442461+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:03.442461+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.094457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":958,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937500+0000","last_change":"2024-03-21T14:25:10.802241+0000","last_active":"2024-03-21T14:26:38.937500+0000","last_peered":"2024-03-21T14:26:38.937500+0000","last_clean":"2024-03-21T14:26:38.937500+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:26:38.937500+0000","last_undegraded":"2024-03-21T14:26:38.937500+0000","last_fullsized":"2024-03-21T14:26:38.937500+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:10.802145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:15:04.115222+0000","last_clean_scrub_stamp":"2024-03-21T14:25:10.802145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":955,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323642+0000","last_change":"2024-03-21T14:25:17.236560+0000","last_active":"2024-03-21T14:26:39.323642+0000","last_peered":"2024-03-21T14:26:39.323642+0000","last_clean":"2024-03-21T14:26:39.323642+0000","last_became_active":"2024-03-21T14:02:33.440848+0000","last_became_peered":"2024-03-21T14:02:33.440848+0000","last_unstale":"2024-03-21T14:26:39.323642+0000","last_undegraded":"2024-03-21T14:26:39.323642+0000","last_fullsized":"2024-03-21T14:26:39.323642+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:17.236474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:10:10.780543+0000","last_clean_scrub_stamp":"2024-03-21T14:25:17.236474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:44.448612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"441'50","reported_seq":274,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322467+0000","last_change":"2024-03-21T14:25:44.320136+0000","last_active":"2024-03-21T14:26:39.322467+0000","last_peered":"2024-03-21T14:26:39.322467+0000","last_clean":"2024-03-21T14:26:39.322467+0000","last_became_active":"2024-03-21T14:20:21.781972+0000","last_became_peered":"2024-03-21T14:20:21.781972+0000","last_unstale":"2024-03-21T14:26:39.322467+0000","last_undegraded":"2024-03-21T14:26:39.322467+0000","last_fullsized":"2024-03-21T14:26:39.322467+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'50","last_scrub_stamp":"2024-03-21T14:25:44.320004+0000","last_deep_scrub":"262'38","last_deep_scrub_stamp":"2024-03-21T14:13:56.515093+0000","last_clean_scrub_stamp":"2024-03-21T14:25:44.320004+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:46.047233+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":930,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937350+0000","last_change":"2024-03-21T14:25:27.525688+0000","last_active":"2024-03-21T14:26:38.937350+0000","last_peered":"2024-03-21T14:26:38.937350+0000","last_clean":"2024-03-21T14:26:38.937350+0000","last_became_active":"2024-03-21T14:02:37.456149+0000","last_became_peered":"2024-03-21T14:02:37.456149+0000","last_unstale":"2024-03-21T14:26:38.937350+0000","last_undegraded":"2024-03-21T14:26:38.937350+0000","last_fullsized":"2024-03-21T14:26:38.937350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:27.525551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:27.525551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:54.393229+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"167'130","reported_seq":652,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649237+0000","last_change":"2024-03-21T14:25:47.816918+0000","last_active":"2024-03-21T14:26:39.649237+0000","last_peered":"2024-03-21T14:26:39.649237+0000","last_clean":"2024-03-21T14:26:39.649237+0000","last_became_active":"2024-03-21T14:25:47.815317+0000","last_became_peered":"2024-03-21T14:25:47.815317+0000","last_unstale":"2024-03-21T14:26:39.649237+0000","last_undegraded":"2024-03-21T14:26:39.649237+0000","last_fullsized":"2024-03-21T14:26:39.649237+0000","mapping_epoch":487,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":488,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:32.486240+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:32.486240+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:55.152806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":719,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649687+0000","last_change":"2024-03-21T14:25:33.856569+0000","last_active":"2024-03-21T14:26:39.649687+0000","last_peered":"2024-03-21T14:26:39.649687+0000","last_clean":"2024-03-21T14:26:39.649687+0000","last_became_active":"2024-03-21T14:25:33.854431+0000","last_became_peered":"2024-03-21T14:25:33.854431+0000","last_unstale":"2024-03-21T14:26:39.649687+0000","last_undegraded":"2024-03-21T14:26:39.649687+0000","last_fullsized":"2024-03-21T14:26:39.649687+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:27.839318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:03:54.381485+0000","last_clean_scrub_stamp":"2024-03-21T14:25:27.839318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:47.601257+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"72'8","reported_seq":1235,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938362+0000","last_change":"2024-03-21T14:25:12.729483+0000","last_active":"2024-03-21T14:26:38.938362+0000","last_peered":"2024-03-21T14:26:38.938362+0000","last_clean":"2024-03-21T14:26:38.938362+0000","last_became_active":"2024-03-21T14:02:37.455245+0000","last_became_peered":"2024-03-21T14:02:37.455245+0000","last_unstale":"2024-03-21T14:26:38.938362+0000","last_undegraded":"2024-03-21T14:26:38.938362+0000","last_fullsized":"2024-03-21T14:26:38.938362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"72'8","last_scrub_stamp":"2024-03-21T14:25:12.729381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:25:12.729381+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":237,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322509+0000","last_change":"2024-03-21T14:25:25.214748+0000","last_active":"2024-03-21T14:26:39.322509+0000","last_peered":"2024-03-21T14:26:39.322509+0000","last_clean":"2024-03-21T14:26:39.322509+0000","last_became_active":"2024-03-21T14:20:21.782708+0000","last_became_peered":"2024-03-21T14:20:21.782708+0000","last_unstale":"2024-03-21T14:26:39.322509+0000","last_undegraded":"2024-03-21T14:26:39.322509+0000","last_fullsized":"2024-03-21T14:26:39.322509+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:25.214650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:25.214650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:48.181660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"441'39","reported_seq":1039,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.897217+0000","last_change":"2024-03-21T14:25:20.682193+0000","last_active":"2024-03-21T14:26:25.897217+0000","last_peered":"2024-03-21T14:26:25.897217+0000","last_clean":"2024-03-21T14:26:25.897217+0000","last_became_active":"2024-03-21T14:21:19.713483+0000","last_became_peered":"2024-03-21T14:21:19.713483+0000","last_unstale":"2024-03-21T14:26:25.897217+0000","last_undegraded":"2024-03-21T14:26:25.897217+0000","last_fullsized":"2024-03-21T14:26:25.897217+0000","mapping_epoch":419,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":420,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'39","last_scrub_stamp":"2024-03-21T14:25:20.682101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:20.682101+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:39.338545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":954,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.323005+0000","last_change":"2024-03-21T14:25:48.350718+0000","last_active":"2024-03-21T14:26:39.323005+0000","last_peered":"2024-03-21T14:26:39.323005+0000","last_clean":"2024-03-21T14:26:39.323005+0000","last_became_active":"2024-03-21T14:25:44.835088+0000","last_became_peered":"2024-03-21T14:25:44.835088+0000","last_unstale":"2024-03-21T14:26:39.323005+0000","last_undegraded":"2024-03-21T14:26:39.323005+0000","last_fullsized":"2024-03-21T14:26:39.323005+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:48.350617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:48.350617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.456467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":819,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.650224+0000","last_change":"2024-03-21T14:26:02.856790+0000","last_active":"2024-03-21T14:26:39.650224+0000","last_peered":"2024-03-21T14:26:39.650224+0000","last_clean":"2024-03-21T14:26:39.650224+0000","last_became_active":"2024-03-21T14:25:33.849755+0000","last_became_peered":"2024-03-21T14:25:33.849755+0000","last_unstale":"2024-03-21T14:26:39.650224+0000","last_undegraded":"2024-03-21T14:26:39.650224+0000","last_fullsized":"2024-03-21T14:26:39.650224+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:02.856647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:36.433777+0000","last_clean_scrub_stamp":"2024-03-21T14:26:02.856647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:32.498375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":955,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937355+0000","last_change":"2024-03-21T14:25:05.783808+0000","last_active":"2024-03-21T14:26:38.937355+0000","last_peered":"2024-03-21T14:26:38.937355+0000","last_clean":"2024-03-21T14:26:38.937355+0000","last_became_active":"2024-03-21T14:19:14.681876+0000","last_became_peered":"2024-03-21T14:19:14.681876+0000","last_unstale":"2024-03-21T14:26:38.937355+0000","last_undegraded":"2024-03-21T14:26:38.937355+0000","last_fullsized":"2024-03-21T14:26:38.937355+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:05.783708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:05.783708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"441'27","reported_seq":1020,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938225+0000","last_change":"2024-03-21T14:25:44.840396+0000","last_active":"2024-03-21T14:26:38.938225+0000","last_peered":"2024-03-21T14:26:38.938225+0000","last_clean":"2024-03-21T14:26:38.938225+0000","last_became_active":"2024-03-21T14:25:44.820435+0000","last_became_peered":"2024-03-21T14:25:44.820435+0000","last_unstale":"2024-03-21T14:26:38.938225+0000","last_undegraded":"2024-03-21T14:26:38.938225+0000","last_fullsized":"2024-03-21T14:26:38.938225+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'27","last_scrub_stamp":"2024-03-21T14:25:41.505523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:25:41.505523+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:57.713745+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"167'130","reported_seq":643,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.649146+0000","last_change":"2024-03-21T14:25:33.853837+0000","last_active":"2024-03-21T14:26:39.649146+0000","last_peered":"2024-03-21T14:26:39.649146+0000","last_clean":"2024-03-21T14:26:39.649146+0000","last_became_active":"2024-03-21T14:25:33.852988+0000","last_became_peered":"2024-03-21T14:25:33.852988+0000","last_unstale":"2024-03-21T14:26:39.649146+0000","last_undegraded":"2024-03-21T14:26:39.649146+0000","last_fullsized":"2024-03-21T14:26:39.649146+0000","mapping_epoch":479,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":480,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:25:21.473145+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:25:21.473145+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.653602+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"405'2","reported_seq":748,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002115+0000","last_change":"2024-03-21T14:25:30.604982+0000","last_active":"2024-03-21T14:26:26.002115+0000","last_peered":"2024-03-21T14:26:26.002115+0000","last_clean":"2024-03-21T14:26:26.002115+0000","last_became_active":"2024-03-21T14:19:14.693855+0000","last_became_peered":"2024-03-21T14:19:14.693855+0000","last_unstale":"2024-03-21T14:26:26.002115+0000","last_undegraded":"2024-03-21T14:26:26.002115+0000","last_fullsized":"2024-03-21T14:26:26.002115+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"405'2","last_scrub_stamp":"2024-03-21T14:25:30.604798+0000","last_deep_scrub":"405'2","last_deep_scrub_stamp":"2024-03-21T14:20:45.611844+0000","last_clean_scrub_stamp":"2024-03-21T14:25:30.604798+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:43.361050+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":960,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:25.896627+0000","last_change":"2024-03-21T14:25:59.574316+0000","last_active":"2024-03-21T14:26:25.896627+0000","last_peered":"2024-03-21T14:26:25.896627+0000","last_clean":"2024-03-21T14:26:25.896627+0000","last_became_active":"2024-03-21T14:02:37.456608+0000","last_became_peered":"2024-03-21T14:02:37.456608+0000","last_unstale":"2024-03-21T14:26:25.896627+0000","last_undegraded":"2024-03-21T14:26:25.896627+0000","last_fullsized":"2024-03-21T14:26:25.896627+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:59.574237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:21:54.617817+0000","last_clean_scrub_stamp":"2024-03-21T14:25:59.574237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:14.411374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":39,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:26.002865+0000","last_change":"2024-03-21T14:26:20.645093+0000","last_active":"2024-03-21T14:26:26.002865+0000","last_peered":"2024-03-21T14:26:26.002865+0000","last_clean":"2024-03-21T14:26:26.002865+0000","last_became_active":"2024-03-21T14:25:44.824548+0000","last_became_peered":"2024-03-21T14:25:44.824548+0000","last_unstale":"2024-03-21T14:26:26.002865+0000","last_undegraded":"2024-03-21T14:26:26.002865+0000","last_fullsized":"2024-03-21T14:26:26.002865+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:26:20.644966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:26:20.644966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:31.247950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"441'45","reported_seq":1012,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938349+0000","last_change":"2024-03-21T14:26:15.296122+0000","last_active":"2024-03-21T14:26:38.938349+0000","last_peered":"2024-03-21T14:26:38.938349+0000","last_clean":"2024-03-21T14:26:38.938349+0000","last_became_active":"2024-03-21T14:19:14.683303+0000","last_became_peered":"2024-03-21T14:19:14.683303+0000","last_unstale":"2024-03-21T14:26:38.938349+0000","last_undegraded":"2024-03-21T14:26:38.938349+0000","last_fullsized":"2024-03-21T14:26:38.938349+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'45","last_scrub_stamp":"2024-03-21T14:26:15.295959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:34.430360+0000","last_clean_scrub_stamp":"2024-03-21T14:26:15.295959+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:20.430574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"167'130","reported_seq":1284,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.547594+0000","last_change":"2024-03-21T14:26:09.982844+0000","last_active":"2024-03-21T14:26:39.547594+0000","last_peered":"2024-03-21T14:26:39.547594+0000","last_clean":"2024-03-21T14:26:39.547594+0000","last_became_active":"2024-03-21T14:19:14.690762+0000","last_became_peered":"2024-03-21T14:19:14.690762+0000","last_unstale":"2024-03-21T14:26:39.547594+0000","last_undegraded":"2024-03-21T14:26:39.547594+0000","last_fullsized":"2024-03-21T14:26:39.547594+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":4,"last_scrub":"167'130","last_scrub_stamp":"2024-03-21T14:26:09.982734+0000","last_deep_scrub":"167'130","last_deep_scrub_stamp":"2024-03-21T14:09:26.512762+0000","last_clean_scrub_stamp":"2024-03-21T14:26:09.982734+0000","objects_scrubbed":0,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:19.580279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":958,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.938439+0000","last_change":"2024-03-21T14:25:48.526680+0000","last_active":"2024-03-21T14:26:38.938439+0000","last_peered":"2024-03-21T14:26:38.938439+0000","last_clean":"2024-03-21T14:26:38.938439+0000","last_became_active":"2024-03-21T14:19:15.689055+0000","last_became_peered":"2024-03-21T14:19:15.689055+0000","last_unstale":"2024-03-21T14:26:38.938439+0000","last_undegraded":"2024-03-21T14:26:38.938439+0000","last_fullsized":"2024-03-21T14:26:38.938439+0000","mapping_epoch":359,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":360,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:48.526574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:38.441204+0000","last_clean_scrub_stamp":"2024-03-21T14:25:48.526574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:27:05.170583+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"360'11","reported_seq":1015,"reported_epoch":495,"state":"active+clean","last_fresh":"2024-03-21T14:26:37.566263+0000","last_change":"2024-03-21T14:26:37.566307+0000","last_active":"2024-03-21T14:26:37.566263+0000","last_peered":"2024-03-21T14:26:37.566263+0000","last_clean":"2024-03-21T14:26:37.566263+0000","last_became_active":"2024-03-21T14:19:14.682365+0000","last_became_peered":"2024-03-21T14:19:14.682365+0000","last_unstale":"2024-03-21T14:26:37.566263+0000","last_undegraded":"2024-03-21T14:26:37.566263+0000","last_fullsized":"2024-03-21T14:26:37.566263+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"360'11","last_scrub_stamp":"2024-03-21T14:26:37.566179+0000","last_deep_scrub":"360'11","last_deep_scrub_stamp":"2024-03-21T14:26:37.566179+0000","last_clean_scrub_stamp":"2024-03-21T14:26:37.566179+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:28:03.908575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":966,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:39.322365+0000","last_change":"2024-03-21T14:25:21.216238+0000","last_active":"2024-03-21T14:26:39.322365+0000","last_peered":"2024-03-21T14:26:39.322365+0000","last_clean":"2024-03-21T14:26:39.322365+0000","last_became_active":"2024-03-21T14:19:34.692775+0000","last_became_peered":"2024-03-21T14:19:34.692775+0000","last_unstale":"2024-03-21T14:26:39.322365+0000","last_undegraded":"2024-03-21T14:26:39.322365+0000","last_fullsized":"2024-03-21T14:26:39.322365+0000","mapping_epoch":365,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":366,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-21T14:25:21.216147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-21T14:02:32.427364+0000","last_clean_scrub_stamp":"2024-03-21T14:25:21.216147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:48.704283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"441'35","reported_seq":1027,"reported_epoch":496,"state":"active+clean","last_fresh":"2024-03-21T14:26:38.937932+0000","last_change":"2024-03-21T14:25:44.839269+0000","last_active":"2024-03-21T14:26:38.937932+0000","last_peered":"2024-03-21T14:26:38.937932+0000","last_clean":"2024-03-21T14:26:38.937932+0000","last_became_active":"2024-03-21T14:25:44.820487+0000","last_became_peered":"2024-03-21T14:25:44.820487+0000","last_unstale":"2024-03-21T14:26:38.937932+0000","last_undegraded":"2024-03-21T14:26:38.937932+0000","last_fullsized":"2024-03-21T14:26:38.937932+0000","mapping_epoch":484,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"441'35","last_scrub_stamp":"2024-03-21T14:25:39.590137+0000","last_deep_scrub":"167'15","last_deep_scrub_stamp":"2024-03-21T14:09:22.119861+0000","last_clean_scrub_stamp":"2024-03-21T14:25:39.590137+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-21T14:26:49.801270+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":893,"num_read_kb":888,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},"log_size":1430,"ondisk_log_size":1430,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1532,"num_read_kb":1289,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":230,"num_bytes_recovered":4736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1083,"ondisk_log_size":1083,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":58,"seq":249108103463,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":898580,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":897470,"kb_avail":92846572,"statfs":{"total":95995035648,"available":95074889728,"internally_reserved":0,"allocated":1122304,"data_stored":729484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":919009738},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":351,"seq":1507533520990,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83820,"kb_used_data":740,"kb_used_omap":4,"kb_used_meta":83067,"kb_avail":93661332,"statfs":{"total":95995035648,"available":95909203968,"internally_reserved":0,"allocated":757760,"data_stored":393089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":85061284},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575174,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":916316,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92828836,"statfs":{"total":95995035648,"available":95056728064,"internally_reserved":0,"allocated":806912,"data_stored":395643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738700,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880212,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864940,"statfs":{"total":95995035648,"available":95093698560,"internally_reserved":0,"allocated":794624,"data_stored":400635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":358,"seq":1537598292059,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":404220,"kb_used_data":1012,"kb_used_omap":7,"kb_used_meta":403192,"kb_avail":93340932,"statfs":{"total":95995035648,"available":95581114368,"internally_reserved":0,"allocated":1036288,"data_stored":656235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8060,"internal_metadata":412868740},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182715,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822500,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92922652,"statfs":{"total":95995035648,"available":95152795648,"internally_reserved":0,"allocated":1404928,"data_stored":991439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":840825287},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849019191,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":924640,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":923838,"kb_avail":92820512,"statfs":{"total":95995035648,"available":95048204288,"internally_reserved":0,"allocated":806912,"data_stored":400895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":946010565},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692146,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":907156,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":905790,"kb_avail":92837996,"statfs":{"total":95995035648,"available":95066107904,"internally_reserved":0,"allocated":1388544,"data_stored":991804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":927529413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1185,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-21T14:26:43.698 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-21T14:26:43.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988 shell --fsid c60e082a-e78a-11ee-95cd-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-21T14:26:43.949 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/mon.a/config 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[29576]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[29576]: from='client.26243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[29576]: 6.1e scrub starts 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[29576]: 6.1e scrub ok 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[29576]: 3.12 scrub starts 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[29576]: 3.12 scrub ok 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[40510]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:44.212 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[40510]: from='client.26243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:26:44.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[40510]: 6.1e scrub starts 2024-03-21T14:26:44.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[40510]: 6.1e scrub ok 2024-03-21T14:26:44.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[40510]: 3.12 scrub starts 2024-03-21T14:26:44.213 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:43 smithi003 ceph-mon[40510]: 3.12 scrub ok 2024-03-21T14:26:44.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:43 smithi118 ceph-mon[33862]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:43 smithi118 ceph-mon[33862]: from='client.26243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-21T14:26:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:43 smithi118 ceph-mon[33862]: 6.1e scrub starts 2024-03-21T14:26:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:43 smithi118 ceph-mon[33862]: 6.1e scrub ok 2024-03-21T14:26:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:43 smithi118 ceph-mon[33862]: 3.12 scrub starts 2024-03-21T14:26:44.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:43 smithi118 ceph-mon[33862]: 3.12 scrub ok 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: 6.8 scrub starts 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: 6.8 scrub ok 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: pgmap v1135: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='client.? 172.21.15.3:0/2609148700' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:26:46.183 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[29576]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:45] ENGINE Bus STOPPING 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:45] ENGINE Bus STOPPED 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:45] ENGINE Bus STARTING 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:45] ENGINE Serving on http://:::9283 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:45] ENGINE Bus STARTED 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: 6.8 scrub starts 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: 6.8 scrub ok 2024-03-21T14:26:46.184 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: pgmap v1135: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='client.? 172.21.15.3:0/2609148700' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:26:46.185 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:45 smithi003 ceph-mon[40510]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: 6.8 scrub starts 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: 6.8 scrub ok 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: pgmap v1135: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='client.? 172.21.15.3:0/2609148700' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='mgr.14589 172.21.15.3:0/65769005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-21T14:26:46.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:45 smithi118 ceph-mon[33862]: from='mgr.14589 ' entity='mgr.y' 2024-03-21T14:26:46.573 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-21T14:26:46.584 INFO:tasks.cephadm:Teardown begin 2024-03-21T14:26:46.584 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T14:26:46.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T14:26:46.673 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-21T14:26:46.673 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T14:26:46.717 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-21T14:26:46.743 INFO:tasks.cephadm:Stopping all daemons... 2024-03-21T14:26:46.745 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-21T14:26:46.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a 2024-03-21T14:26:46.753 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: ::ffff:172.21.15.118 - - [21/Mar/2024:14:26:46] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.51.0" 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 systemd[1]: Stopping Ceph mon.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 5.4 scrub starts 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 5.4 scrub ok 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 3.7 scrub starts 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 3.7 scrub ok 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-21T14:26:47.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 3.9 scrub starts 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 3.9 scrub ok 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 3.1f scrub starts 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 3.1f scrub ok 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[29576]: 6.19 scrub starts 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 5.4 scrub starts 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 5.4 scrub ok 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 3.7 scrub starts 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 3.7 scrub ok 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 3.9 scrub starts 2024-03-21T14:26:47.131 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 3.9 scrub ok 2024-03-21T14:26:47.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 3.1f scrub starts 2024-03-21T14:26:47.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 3.1f scrub ok 2024-03-21T14:26:47.132 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:46 smithi003 ceph-mon[40510]: 6.19 scrub starts 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 5.4 scrub starts 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 5.4 scrub ok 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 3.7 scrub starts 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 3.7 scrub ok 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 730 B/s rd, 0 op/s 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 3.9 scrub starts 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 3.9 scrub ok 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 3.1f scrub starts 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 3.1f scrub ok 2024-03-21T14:26:47.431 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:46 smithi118 ceph-mon[33862]: 6.19 scrub starts 2024-03-21T14:26:47.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:47 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a[29572]: 2024-03-21T14:26:47.129+0000 7f885689b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:26:47.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:47 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a[29572]: 2024-03-21T14:26:47.129+0000 7f885689b700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-21T14:26:47.489 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:47 smithi003 podman[216871]: 2024-03-21 14:26:47.202080846 +0000 UTC m=+0.201234017 container died 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T14:26:47.753 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:47 smithi003 podman[216871]: 2024-03-21 14:26:47.588894093 +0000 UTC m=+0.588047254 container cleanup 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-21T14:26:47.753 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:47 smithi003 bash[216871]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a 2024-03-21T14:26:48.180 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:26:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug there is no tcmu-runner data available 2024-03-21T14:26:48.263 INFO:journalctl@ceph.mon.a.smithi003.stdout:Mar 21 14:26:48 smithi003 podman[216883]: 2024-03-21 14:26:48.006912447 +0000 UTC m=+0.802806494 container remove 53ed71f0a2c3221ceab8e7fe82da8d1eb2ff28e5796686391ea356a486f0ec33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:26:48.269 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.a.service' 2024-03-21T14:26:48.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:26:48.317 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-21T14:26:48.317 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-21T14:26:48.317 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.c 2024-03-21T14:26:48.635 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:48 smithi003 systemd[1]: Stopping Ceph mon.c for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:26:48.909 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:48] ENGINE Bus STOPPING 2024-03-21T14:26:48.909 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:48] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-21T14:26:48.910 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:48] ENGINE Bus STOPPED 2024-03-21T14:26:48.910 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:48] ENGINE Bus STARTING 2024-03-21T14:26:48.910 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:48] ENGINE Serving on http://:::9283 2024-03-21T14:26:48.910 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: [21/Mar/2024:14:26:48] ENGINE Bus STARTED 2024-03-21T14:26:48.910 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c[40506]: 2024-03-21T14:26:48.634+0000 7fa4f1bc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:26:48.910 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:48 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c[40506]: 2024-03-21T14:26:48.634+0000 7fa4f1bc4700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-21T14:26:49.202 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:48 smithi003 podman[216992]: 2024-03-21 14:26:48.910258368 +0000 UTC m=+0.409291399 container died ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-21T14:26:49.503 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:49 smithi003 podman[216992]: 2024-03-21 14:26:49.385600887 +0000 UTC m=+0.884633978 container cleanup ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:26:49.504 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:49 smithi003 bash[216992]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c 2024-03-21T14:26:49.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.c.service' 2024-03-21T14:26:49.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:49 smithi003 podman[217017]: 2024-03-21 14:26:49.553483043 +0000 UTC m=+0.644681519 container remove ea8a4ec34da7e6b88df35691ff747a9e21adeac8358ec18393c2691a5db70240 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-21T14:26:49.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:49 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-21T14:26:49.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:49 smithi003 systemd[1]: Stopped Ceph mon.c for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:26:49.803 INFO:journalctl@ceph.mon.c.smithi003.stdout:Mar 21 14:26:49 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.c.service: Consumed 26.227s CPU time. 2024-03-21T14:26:49.814 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:26:49.814 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-21T14:26:49.814 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-21T14:26:49.814 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.b 2024-03-21T14:26:50.179 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:49 smithi118 systemd[1]: Stopping Ceph mon.b for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:26:50.430 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:50 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b[33858]: 2024-03-21T14:26:50.178+0000 7ff787adc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:26:50.433 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:50 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b[33858]: 2024-03-21T14:26:50.178+0000 7ff787adc700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-21T14:26:50.763 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:50 smithi118 podman[161349]: 2024-03-21 14:26:50.444300477 +0000 UTC m=+0.389364118 container died ef71a106405ebb82651f494ade417813a017236a394315676da42399342aee89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-21T14:26:51.015 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:51 smithi118 podman[161349]: 2024-03-21 14:26:51.013971075 +0000 UTC m=+0.959034745 container cleanup ef71a106405ebb82651f494ade417813a017236a394315676da42399342aee89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-21T14:26:51.015 INFO:journalctl@ceph.mon.b.smithi118.stdout:Mar 21 14:26:51 smithi118 bash[161349]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mon-b 2024-03-21T14:26:51.419 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mon.b.service' 2024-03-21T14:26:51.461 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:26:51.462 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-21T14:26:51.462 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-21T14:26:51.462 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y 2024-03-21T14:26:51.753 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:51 smithi003 systemd[1]: Stopping Ceph mgr.y for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:26:52.107 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:51 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y[29806]: 2024-03-21T14:26:51.789+0000 7fa1f9ebd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:26:52.107 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:51 smithi003 podman[217124]: 2024-03-21 14:26:51.821350603 +0000 UTC m=+0.165192818 container died a24079a1e6eb3aa9bf34027805da31f6b05e8eda3dc338c4719e93e63113d4c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-21T14:26:52.503 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:52 smithi003 podman[217124]: 2024-03-21 14:26:52.322357873 +0000 UTC m=+0.666200127 container cleanup a24079a1e6eb3aa9bf34027805da31f6b05e8eda3dc338c4719e93e63113d4c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:ba760091cd7bd2b0d23f4825ac856ba66450e988, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-21T14:26:52.503 INFO:journalctl@ceph.mgr.y.smithi003.stdout:Mar 21 14:26:52 smithi003 bash[217124]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-y 2024-03-21T14:26:52.744 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-21T14:26:52.782 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:26:52.782 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-21T14:26:52.782 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-21T14:26:52.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.x 2024-03-21T14:26:53.142 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:26:52 smithi118 systemd[1]: Stopping Ceph mgr.x for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:26:53.425 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:26:53 smithi118 podman[161473]: 2024-03-21 14:26:53.143553958 +0000 UTC m=+0.157887829 container died 8b7dde2a0e5ead6dff5bb9b34e96b355b356c2e7fc712fa25896ac14a83d0f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-21T14:26:53.680 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:26:53 smithi118 podman[161473]: 2024-03-21 14:26:53.535356349 +0000 UTC m=+0.549690220 container cleanup 8b7dde2a0e5ead6dff5bb9b34e96b355b356c2e7fc712fa25896ac14a83d0f82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-21T14:26:53.680 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 21 14:26:53 smithi118 bash[161473]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-mgr-x 2024-03-21T14:26:53.898 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-21T14:26:53.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:26:53.936 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-21T14:26:53.936 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-21T14:26:53.937 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.0 2024-03-21T14:26:54.236 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:26:53 smithi003 systemd[1]: Stopping Ceph osd.0 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:26:54.504 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:26:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0[44524]: 2024-03-21T14:26:54.235+0000 7fa430517700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:26:54.504 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:26:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0[44524]: 2024-03-21T14:26:54.235+0000 7fa430517700 -1 osd.0 496 *** Got signal Terminated *** 2024-03-21T14:26:54.504 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:26:54 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0[44524]: 2024-03-21T14:26:54.235+0000 7fa430517700 -1 osd.0 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:00.253 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:26:59 smithi003 podman[217245]: 2024-03-21 14:26:59.875842896 +0000 UTC m=+5.741224646 container died 128661086e0e7aeaecebb6abaff2985655c59c670a63bb570dcbcf7d5130acde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-21T14:27:00.512 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:00 smithi003 podman[217245]: 2024-03-21 14:27:00.261752646 +0000 UTC m=+6.127134438 container cleanup 128661086e0e7aeaecebb6abaff2985655c59c670a63bb570dcbcf7d5130acde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:00.512 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:00 smithi003 bash[217245]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0 2024-03-21T14:27:00.811 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:00 smithi003 podman[217262]: 2024-03-21 14:27:00.513275098 +0000 UTC m=+0.640309242 container remove 128661086e0e7aeaecebb6abaff2985655c59c670a63bb570dcbcf7d5130acde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-21T14:27:01.132 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:00 smithi003 podman[217353]: 2024-03-21 14:27:00.715849301 +0000 UTC m=+0.018735144 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:01.132 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:00 smithi003 podman[217353]: 2024-03-21 14:27:00.814046279 +0000 UTC m=+0.116932112 container create 3dc799fcfb304bed3b10d6f5faa82c9445144cf4751e20ef02ff86789081c058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:01.132 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:01 smithi003 podman[217353]: 2024-03-21 14:27:01.050025306 +0000 UTC m=+0.352911145 container init 3dc799fcfb304bed3b10d6f5faa82c9445144cf4751e20ef02ff86789081c058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-21T14:27:01.132 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:01 smithi003 podman[217353]: 2024-03-21 14:27:01.053319901 +0000 UTC m=+0.356205733 container start 3dc799fcfb304bed3b10d6f5faa82c9445144cf4751e20ef02ff86789081c058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:01.504 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:01 smithi003 podman[217353]: 2024-03-21 14:27:01.133177726 +0000 UTC m=+0.436063577 container attach 3dc799fcfb304bed3b10d6f5faa82c9445144cf4751e20ef02ff86789081c058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:01.894 INFO:journalctl@ceph.osd.0.smithi003.stdout:Mar 21 14:27:01 smithi003 podman[217353]: 2024-03-21 14:27:01.579464849 +0000 UTC m=+0.882350702 container died 3dc799fcfb304bed3b10d6f5faa82c9445144cf4751e20ef02ff86789081c058 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5) 2024-03-21T14:27:02.318 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.0.service' 2024-03-21T14:27:02.362 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:02.362 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-21T14:27:02.362 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-21T14:27:02.362 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.1 2024-03-21T14:27:02.706 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:02 smithi003 systemd[1]: Stopping Ceph osd.1 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:03.004 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1[50259]: 2024-03-21T14:27:02.706+0000 7f88ea199700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:03.004 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1[50259]: 2024-03-21T14:27:02.706+0000 7f88ea199700 -1 osd.1 496 *** Got signal Terminated *** 2024-03-21T14:27:03.004 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:02 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1[50259]: 2024-03-21T14:27:02.706+0000 7f88ea199700 -1 osd.1 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:08.524 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:08 smithi003 podman[217462]: 2024-03-21 14:27:08.260472146 +0000 UTC m=+5.676332388 container died 41379aad93989e6d9c1cabef5c4b6e6f9c0b6961f62f5bf0c8e68362d6f476e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319) 2024-03-21T14:27:08.899 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:08 smithi003 podman[217462]: 2024-03-21 14:27:08.707123694 +0000 UTC m=+6.122983984 container cleanup 41379aad93989e6d9c1cabef5c4b6e6f9c0b6961f62f5bf0c8e68362d6f476e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:27:08.899 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:08 smithi003 bash[217462]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1 2024-03-21T14:27:09.253 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:08 smithi003 podman[217479]: 2024-03-21 14:27:08.900107168 +0000 UTC m=+0.636551948 container remove 41379aad93989e6d9c1cabef5c4b6e6f9c0b6961f62f5bf0c8e68362d6f476e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:09.544 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:09 smithi003 podman[217567]: 2024-03-21 14:27:09.196829669 +0000 UTC m=+0.017563139 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:09.544 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:09 smithi003 podman[217567]: 2024-03-21 14:27:09.30103335 +0000 UTC m=+0.121766833 container create 5b8fa7f881ff9488384db19c1a227930b57658e7006b2d6cd2d875a00839f2d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-21T14:27:09.846 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:09 smithi003 podman[217567]: 2024-03-21 14:27:09.545135504 +0000 UTC m=+0.365868956 container init 5b8fa7f881ff9488384db19c1a227930b57658e7006b2d6cd2d875a00839f2d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-21T14:27:09.846 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:09 smithi003 podman[217567]: 2024-03-21 14:27:09.548546715 +0000 UTC m=+0.369280157 container start 5b8fa7f881ff9488384db19c1a227930b57658e7006b2d6cd2d875a00839f2d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:09.846 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:09 smithi003 podman[217567]: 2024-03-21 14:27:09.678279255 +0000 UTC m=+0.499012698 container attach 5b8fa7f881ff9488384db19c1a227930b57658e7006b2d6cd2d875a00839f2d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-03-21T14:27:10.138 INFO:journalctl@ceph.osd.1.smithi003.stdout:Mar 21 14:27:09 smithi003 podman[217567]: 2024-03-21 14:27:09.847119824 +0000 UTC m=+0.667853264 container died 5b8fa7f881ff9488384db19c1a227930b57658e7006b2d6cd2d875a00839f2d1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319) 2024-03-21T14:27:10.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.1.service' 2024-03-21T14:27:10.483 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:10.484 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-21T14:27:10.484 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-21T14:27:10.484 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2 2024-03-21T14:27:10.755 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:10 smithi003 systemd[1]: Stopping Ceph osd.2 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:11.254 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[175187]: 2024-03-21T14:27:10.842+0000 7f7e75e78700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:11.254 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[175187]: 2024-03-21T14:27:10.842+0000 7f7e75e78700 -1 osd.2 496 *** Got signal Terminated *** 2024-03-21T14:27:11.254 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:10 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2[175187]: 2024-03-21T14:27:10.842+0000 7f7e75e78700 -1 osd.2 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:16.717 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:16 smithi003 podman[217672]: 2024-03-21 14:27:16.354125405 +0000 UTC m=+5.640460584 container died 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-03-21T14:27:17.003 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:16 smithi003 podman[217672]: 2024-03-21 14:27:16.884119542 +0000 UTC m=+6.170454711 container cleanup 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-21T14:27:17.004 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:16 smithi003 bash[217672]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2 2024-03-21T14:27:17.503 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:17 smithi003 podman[217689]: 2024-03-21 14:27:17.110553292 +0000 UTC m=+0.752753718 container remove 548341a5051a8b3930d81f166866e6956aeb700599d107ff468e1bab4cc537a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:17.838 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:17 smithi003 podman[217783]: 2024-03-21 14:27:17.428826857 +0000 UTC m=+0.017479941 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:17.838 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:17 smithi003 podman[217783]: 2024-03-21 14:27:17.569757934 +0000 UTC m=+0.158411084 container create de16b26641507a7d46d5b5c9db98e7f17d6eebdfb1c0786a20ff0f4a7663aee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-21T14:27:17.838 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:17 smithi003 podman[217783]: 2024-03-21 14:27:17.763739434 +0000 UTC m=+0.352392525 container init de16b26641507a7d46d5b5c9db98e7f17d6eebdfb1c0786a20ff0f4a7663aee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:27:17.838 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:17 smithi003 podman[217783]: 2024-03-21 14:27:17.7667975 +0000 UTC m=+0.355450593 container start de16b26641507a7d46d5b5c9db98e7f17d6eebdfb1c0786a20ff0f4a7663aee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:18.254 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:17 smithi003 podman[217783]: 2024-03-21 14:27:17.838739415 +0000 UTC m=+0.427392508 container attach de16b26641507a7d46d5b5c9db98e7f17d6eebdfb1c0786a20ff0f4a7663aee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-21T14:27:18.254 INFO:journalctl@ceph.osd.2.smithi003.stdout:Mar 21 14:27:18 smithi003 podman[217783]: 2024-03-21 14:27:18.061810301 +0000 UTC m=+0.650463383 container died de16b26641507a7d46d5b5c9db98e7f17d6eebdfb1c0786a20ff0f4a7663aee2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:18.780 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.2.service' 2024-03-21T14:27:18.823 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:18.823 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-21T14:27:18.823 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-21T14:27:18.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.3 2024-03-21T14:27:19.145 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:18 smithi003 systemd[1]: Stopping Ceph osd.3 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:19.503 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:19.144+0000 7f5c6fe12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:19.503 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:19.144+0000 7f5c6fe12700 -1 osd.3 496 *** Got signal Terminated *** 2024-03-21T14:27:19.503 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:19 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:19.144+0000 7f5c6fe12700 -1 osd.3 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:21.004 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:20 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:20.613+0000 7f5c683fe700 -1 osd.3 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:54.739223+0000 front 2024-03-21T14:26:54.739371+0000 (oldest deadline 2024-03-21T14:27:20.039417+0000) 2024-03-21T14:27:21.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:20 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:20.843+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:21.930 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:21 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:21.809+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:21.931 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:21 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:21.617+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:22.003 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:21 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:21.656+0000 7f5c683fe700 -1 osd.3 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:54.739223+0000 front 2024-03-21T14:26:54.739371+0000 (oldest deadline 2024-03-21T14:27:20.039417+0000) 2024-03-21T14:27:22.430 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:22.062+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:22.856 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:22.606+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:23.003 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:22 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:22.613+0000 7f5c683fe700 -1 osd.3 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:54.739223+0000 front 2024-03-21T14:26:54.739371+0000 (oldest deadline 2024-03-21T14:27:20.039417+0000) 2024-03-21T14:27:23.180 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:23.025+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:23.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:22.857+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:23.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:22.984+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:23.181 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:27:22.980Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-21T14:27:23.181 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:27:22.984Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-21T14:27:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:27:22.984Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-21T14:27:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:27:22.987Z caller=refresh.go:90 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-21T14:27:23.182 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:27:22 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:27:22.988Z caller=refresh.go:90 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.3:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.3:8765: connect: connection refused" 2024-03-21T14:27:23.930 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:23.592+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:23.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:23.837+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:24.004 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:23 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3[62233]: 2024-03-21T14:27:23.616+0000 7f5c683fe700 -1 osd.3 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:54.739223+0000 front 2024-03-21T14:26:54.739371+0000 (oldest deadline 2024-03-21T14:27:20.039417+0000) 2024-03-21T14:27:24.430 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:24 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:24.049+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:24.431 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:23 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:23.950+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:24.827 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:24 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:24.626+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:24.963 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:24 smithi003 podman[217888]: 2024-03-21 14:27:24.694292509 +0000 UTC m=+5.647659070 container died 077e13c28798696e543696020bf067b245bec4d24cf1c275dfa28c178af6983f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:25.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:24 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:24.956+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:25.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:24 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:24.827+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:25.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:25 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:25.014+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:25.253 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 podman[217888]: 2024-03-21 14:27:25.153186518 +0000 UTC m=+6.106553137 container cleanup 077e13c28798696e543696020bf067b245bec4d24cf1c275dfa28c178af6983f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:25.253 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 bash[217888]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3 2024-03-21T14:27:25.754 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 podman[217905]: 2024-03-21 14:27:25.42138529 +0000 UTC m=+0.714494596 container remove 077e13c28798696e543696020bf067b245bec4d24cf1c275dfa28c178af6983f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-21T14:27:25.930 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:25 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:25.649+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:25.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:25 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:25.821+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:26.056 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 podman[217997]: 2024-03-21 14:27:25.675885357 +0000 UTC m=+0.018220150 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:26.056 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 podman[217997]: 2024-03-21 14:27:25.780571551 +0000 UTC m=+0.122906380 container create d9d0afcb564e30277895ea9e357372f974c616405232115a51e46186ae8276c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-21T14:27:26.057 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 podman[217997]: 2024-03-21 14:27:25.974508832 +0000 UTC m=+0.316843623 container init d9d0afcb564e30277895ea9e357372f974c616405232115a51e46186ae8276c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-21T14:27:26.057 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:25 smithi003 podman[217997]: 2024-03-21 14:27:25.977477585 +0000 UTC m=+0.319812374 container start d9d0afcb564e30277895ea9e357372f974c616405232115a51e46186ae8276c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-21T14:27:26.057 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:26 smithi003 podman[217997]: 2024-03-21 14:27:26.057677627 +0000 UTC m=+0.400012420 container attach d9d0afcb564e30277895ea9e357372f974c616405232115a51e46186ae8276c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-21T14:27:26.430 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:25 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:25.942+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:26.431 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:25 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:25.994+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:26.504 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:26 smithi003 podman[217997]: 2024-03-21 14:27:26.27465782 +0000 UTC m=+0.616992617 container died d9d0afcb564e30277895ea9e357372f974c616405232115a51e46186ae8276c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:27:26.833 INFO:journalctl@ceph.osd.3.smithi003.stdout:Mar 21 14:27:26 smithi003 podman[218024]: 2024-03-21 14:27:26.810789658 +0000 UTC m=+0.531115116 container remove d9d0afcb564e30277895ea9e357372f974c616405232115a51e46186ae8276c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T14:27:26.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.3.service' 2024-03-21T14:27:26.879 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:26.879 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-21T14:27:26.880 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-21T14:27:26.880 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.4 2024-03-21T14:27:26.910 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:26.638+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:26.910 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:26.852+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:27.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:26.916+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:27.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:26 smithi118 systemd[1]: Stopping Ceph osd.4 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:27.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:26 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:26.980+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:27.604 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:27.186+0000 7fdd05176700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:27.605 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:27.186+0000 7fdd05176700 -1 osd.4 496 *** Got signal Terminated *** 2024-03-21T14:27:27.605 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:27.186+0000 7fdd05176700 -1 osd.4 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:27.930 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:27.604+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:27.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:27.854+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:28.430 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:27 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:27.934+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:28.431 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:28 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:28.010+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:28.910 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:28 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:28.647+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:28.911 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:28 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:28.808+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:29.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:28 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:28.910+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:29.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:29 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:29.038+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:29.909 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:29 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:29.631+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:29.910 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:29 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:29.808+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:29.910 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:29 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:29.808+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:30.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:29 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:29.909+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:30.180 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:30 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:30.010+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:30.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:30 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:30.010+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:05.947519+0000 front 2024-03-21T14:27:05.947566+0000 (oldest deadline 2024-03-21T14:27:29.447393+0000) 2024-03-21T14:27:30.930 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:30 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:30.650+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:30.931 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:30 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:30.867+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:30.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:30 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:30.759+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:30.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:30 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:30.759+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:31.430 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:31 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:31.057+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:31.430 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:31 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:31.057+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:05.947519+0000 front 2024-03-21T14:27:05.947566+0000 (oldest deadline 2024-03-21T14:27:29.447393+0000) 2024-03-21T14:27:31.930 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:31 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:31.659+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:31.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:31 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:31.778+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:31.931 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:31 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:31.778+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:31.931 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:31 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:31.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:32.431 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:32.091+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:57.345856+0000 front 2024-03-21T14:26:57.345630+0000 (oldest deadline 2024-03-21T14:27:21.445666+0000) 2024-03-21T14:27:32.431 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4[38763]: 2024-03-21T14:27:32.091+0000 7fdcfd762700 -1 osd.4 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:05.947519+0000 front 2024-03-21T14:27:05.947566+0000 (oldest deadline 2024-03-21T14:27:29.447393+0000) 2024-03-21T14:27:32.893 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:32.633+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:32.893 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:32.633+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:32.894 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:32 smithi118 podman[161596]: 2024-03-21 14:27:32.776504138 +0000 UTC m=+5.687367833 container died 249efb3076ae9abba3d05818fa815517c09d8ac456f6cd3a0dade62aa0ab01ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-21T14:27:32.894 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:32.810+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:32.894 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:32.810+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:33.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:32.893+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:33.597 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:33 smithi118 podman[161596]: 2024-03-21 14:27:33.204510414 +0000 UTC m=+6.115374097 container cleanup 249efb3076ae9abba3d05818fa815517c09d8ac456f6cd3a0dade62aa0ab01ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:27:33.597 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:33 smithi118 bash[161596]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4 2024-03-21T14:27:33.598 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:33 smithi118 podman[161613]: 2024-03-21 14:27:33.40601223 +0000 UTC m=+0.624157568 container remove 249efb3076ae9abba3d05818fa815517c09d8ac456f6cd3a0dade62aa0ab01ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:33.919 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:33 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:33.597+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:33.919 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:33 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:33.597+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:33.919 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:33 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:33.821+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:33.919 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:33 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:33.821+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:33.920 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:33 smithi118 podman[161706]: 2024-03-21 14:27:33.725891021 +0000 UTC m=+0.018046584 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:33.920 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:33 smithi118 podman[161706]: 2024-03-21 14:27:33.823544315 +0000 UTC m=+0.115699880 container create 19f86bc9774b732946cea0f8f1667bac3c7e003c6252d013b1d54a8ba8eafc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-21T14:27:34.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:33 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:33.937+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:34.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:34 smithi118 podman[161706]: 2024-03-21 14:27:34.026163591 +0000 UTC m=+0.318319158 container init 19f86bc9774b732946cea0f8f1667bac3c7e003c6252d013b1d54a8ba8eafc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T14:27:34.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:34 smithi118 podman[161706]: 2024-03-21 14:27:34.029292608 +0000 UTC m=+0.321448178 container start 19f86bc9774b732946cea0f8f1667bac3c7e003c6252d013b1d54a8ba8eafc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-21T14:27:34.181 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:34 smithi118 podman[161706]: 2024-03-21 14:27:34.134571221 +0000 UTC m=+0.426726798 container attach 19f86bc9774b732946cea0f8f1667bac3c7e003c6252d013b1d54a8ba8eafc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-21T14:27:34.846 INFO:journalctl@ceph.osd.4.smithi118.stdout:Mar 21 14:27:34 smithi118 podman[161706]: 2024-03-21 14:27:34.556080388 +0000 UTC m=+0.848235956 container died 19f86bc9774b732946cea0f8f1667bac3c7e003c6252d013b1d54a8ba8eafc72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:34.846 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:34 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:34.598+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:34.847 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:34 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:34.598+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:35.137 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:34 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:34.846+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:35.138 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:34 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:34.846+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:35.138 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:34 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:34.961+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:35.138 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:34 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:34.961+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:35.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.4.service' 2024-03-21T14:27:35.235 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:35.235 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-21T14:27:35.235 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-21T14:27:35.236 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.5 2024-03-21T14:27:35.419 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:35 smithi118 systemd[1]: Stopping Ceph osd.5 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:35.680 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:35.575+0000 7f3bf290f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:35.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:35.575+0000 7f3bf290f700 -1 osd.5 496 *** Got signal Terminated *** 2024-03-21T14:27:35.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:35.575+0000 7f3bf290f700 -1 osd.5 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:35.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:35.588+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:35.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:35.588+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:36.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:35.862+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:36.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:35.862+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:36.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:35.951+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:36.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:35 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:35.951+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:36.680 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:36.575+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:36.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:36.575+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:37.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:36.829+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:37.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:36.829+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:37.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:36.951+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:37.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:36 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:36.951+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:37.850 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:37.579+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:37.850 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:37.579+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:38.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:37.948+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:38.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:37.948+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:38.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:37.948+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:38.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:37.849+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:38.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:37.849+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:38.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:37 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:37.850+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:38.828 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:38.577+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:38.828 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:38.577+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:38.828 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:38.577+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.933005+0000 front 2024-03-21T14:27:13.933037+0000 (oldest deadline 2024-03-21T14:27:38.032854+0000) 2024-03-21T14:27:39.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:38.828+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:39.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:38.828+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:39.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:38.828+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:39.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:38.910+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:39.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:38.910+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:39.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:38 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:38.910+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:39.866 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:39.562+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:39.866 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:39.562+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:39.866 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:39.562+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.933005+0000 front 2024-03-21T14:27:13.933037+0000 (oldest deadline 2024-03-21T14:27:38.032854+0000) 2024-03-21T14:27:39.866 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:39.796+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:39.866 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:39.796+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:39.866 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:39.796+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:40.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:39.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:40.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:39.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:40.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:39.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:40.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:39 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:39.866+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:40.912 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:40.573+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:55.831031+0000 front 2024-03-21T14:26:55.831171+0000 (oldest deadline 2024-03-21T14:27:21.130884+0000) 2024-03-21T14:27:40.913 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:40.573+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.532137+0000 front 2024-03-21T14:27:07.532234+0000 (oldest deadline 2024-03-21T14:27:32.231981+0000) 2024-03-21T14:27:40.913 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5[44831]: 2024-03-21T14:27:40.573+0000 7f3beaefb700 -1 osd.5 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.933005+0000 front 2024-03-21T14:27:13.933037+0000 (oldest deadline 2024-03-21T14:27:38.032854+0000) 2024-03-21T14:27:40.913 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:40.774+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:40.913 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:40.774+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:40.913 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:40.774+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:41.180 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:41 smithi118 podman[161814]: 2024-03-21 14:27:41.138801193 +0000 UTC m=+5.674279401 container died 2e113a52b7f14ecb3aad7d5d767301b4e273dddb3c27879ce7e786a05f62315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-21T14:27:41.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:40.912+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:41.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:40.912+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:41.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:40.912+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:41.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:40 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:40.912+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:41.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:41 smithi118 podman[161814]: 2024-03-21 14:27:41.49216402 +0000 UTC m=+6.027642227 container cleanup 2e113a52b7f14ecb3aad7d5d767301b4e273dddb3c27879ce7e786a05f62315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-21T14:27:41.681 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:41 smithi118 bash[161814]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5 2024-03-21T14:27:42.150 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:41 smithi118 podman[161842]: 2024-03-21 14:27:41.843676972 +0000 UTC m=+0.707823635 container remove 2e113a52b7f14ecb3aad7d5d767301b4e273dddb3c27879ce7e786a05f62315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:41.801+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:41.801+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:41.801+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:41.944+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:41.944+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:41.944+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:42.151 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:41 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:41.944+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:42.430 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:42 smithi118 podman[161928]: 2024-03-21 14:27:42.054559452 +0000 UTC m=+0.018620976 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:42.431 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:42 smithi118 podman[161928]: 2024-03-21 14:27:42.169903524 +0000 UTC m=+0.133965075 container create 9cf9868bc93e25a98c7a7796ad77dec92e8bd00449d594ddb0de0f5f2a26aa91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-21T14:27:42.431 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:42 smithi118 podman[161928]: 2024-03-21 14:27:42.389149683 +0000 UTC m=+0.353211192 container init 9cf9868bc93e25a98c7a7796ad77dec92e8bd00449d594ddb0de0f5f2a26aa91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319) 2024-03-21T14:27:42.431 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:42 smithi118 podman[161928]: 2024-03-21 14:27:42.392848418 +0000 UTC m=+0.356909928 container start 9cf9868bc93e25a98c7a7796ad77dec92e8bd00449d594ddb0de0f5f2a26aa91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-21T14:27:42.780 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:42 smithi118 podman[161928]: 2024-03-21 14:27:42.456478114 +0000 UTC m=+0.420539638 container attach 9cf9868bc93e25a98c7a7796ad77dec92e8bd00449d594ddb0de0f5f2a26aa91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-21T14:27:42.780 INFO:journalctl@ceph.osd.5.smithi118.stdout:Mar 21 14:27:42 smithi118 podman[161928]: 2024-03-21 14:27:42.678589433 +0000 UTC m=+0.642650942 container died 9cf9868bc93e25a98c7a7796ad77dec92e8bd00449d594ddb0de0f5f2a26aa91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-21T14:27:43.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:42.779+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:43.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:42.779+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:43.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:42.779+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:43.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:42.922+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:43.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:42.922+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:43.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:42.922+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:43.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:42 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:42.922+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:43.258 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.5.service' 2024-03-21T14:27:43.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:43.301 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-21T14:27:43.301 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-21T14:27:43.301 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6 2024-03-21T14:27:43.628 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 systemd[1]: Stopping Ceph osd.6 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:43.916 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:43.628+0000 7efc0e030700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:43.916 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:43.628+0000 7efc0e030700 -1 osd.6 496 *** Got signal Terminated *** 2024-03-21T14:27:43.916 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:43.628+0000 7efc0e030700 -1 osd.6 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:43.916 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:43.807+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:43.916 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:43.807+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:43.916 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:43.807+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:44.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:43.916+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:44.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:43.916+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:44.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:43.916+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:44.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:43 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:43.916+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:45.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:44.789+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:45.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:44.789+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:45.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:44.789+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:45.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:44.893+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:45.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:44.893+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:45.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:44.893+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:45.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:44 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:44.893+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:46.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:45.824+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:46.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:45.824+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:46.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:45.824+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:46.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:45.874+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:46.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:45.874+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:46.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:45.874+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:46.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:45 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:45.874+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:47.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:46.850+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:47.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:46.850+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:47.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:46.850+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:47.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:46.886+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:47.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:46.886+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:47.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:46.886+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:47.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:46 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:46.886+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:48.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:47.835+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.395693+0000 front 2024-03-21T14:26:56.395610+0000 (oldest deadline 2024-03-21T14:27:19.895446+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:47.835+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:07.397503+0000 front 2024-03-21T14:27:07.397599+0000 (oldest deadline 2024-03-21T14:27:29.097339+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6[134100]: 2024-03-21T14:27:47.835+0000 7efc0661c700 -1 osd.6 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.198299+0000 front 2024-03-21T14:27:13.198380+0000 (oldest deadline 2024-03-21T14:27:37.298201+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:47.903+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:47.903+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:47.903+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:47.903+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:48.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:47 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:47.903+0000 7fb68b3b4700 -1 osd.7 496 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:1460 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e496) 2024-03-21T14:27:49.180 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:49 smithi118 podman[162033]: 2024-03-21 14:27:49.062259531 +0000 UTC m=+5.526892228 container died fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True) 2024-03-21T14:27:49.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:48 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:48.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:49.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:48 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:48.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:49.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:48 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:48.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:49.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:48 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:48.866+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:49.680 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:49 smithi118 podman[162033]: 2024-03-21 14:27:49.420654768 +0000 UTC m=+5.885287475 container cleanup fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True) 2024-03-21T14:27:49.681 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:49 smithi118 bash[162033]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6 2024-03-21T14:27:50.015 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:49 smithi118 podman[162053]: 2024-03-21 14:27:49.713722665 +0000 UTC m=+0.651162691 container remove fc48ba45aecc897857e14344226cfdc095b3ae9ddb3576e7708e78872cf14e53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-21T14:27:50.015 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:49 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:49.861+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:50.016 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:49 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:49.861+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:50.016 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:49 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:49.861+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:50.016 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:49 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:49.861+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:50.283 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:50 smithi118 podman[162145]: 2024-03-21 14:27:49.919220607 +0000 UTC m=+0.018598273 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:50.283 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:50 smithi118 podman[162145]: 2024-03-21 14:27:50.073168245 +0000 UTC m=+0.172545911 container create 92bf2794e77c60ebe094b03c5470f40ba72b690b8f09f951828bd042a2f97375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-21T14:27:50.577 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:50 smithi118 podman[162145]: 2024-03-21 14:27:50.284019263 +0000 UTC m=+0.383396942 container init 92bf2794e77c60ebe094b03c5470f40ba72b690b8f09f951828bd042a2f97375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:50.577 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:50 smithi118 podman[162145]: 2024-03-21 14:27:50.287744891 +0000 UTC m=+0.387122566 container start 92bf2794e77c60ebe094b03c5470f40ba72b690b8f09f951828bd042a2f97375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-21T14:27:50.577 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:50 smithi118 podman[162145]: 2024-03-21 14:27:50.358891271 +0000 UTC m=+0.458268951 container attach 92bf2794e77c60ebe094b03c5470f40ba72b690b8f09f951828bd042a2f97375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:50.835 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 21 14:27:50 smithi118 podman[162145]: 2024-03-21 14:27:50.578468082 +0000 UTC m=+0.677845759 container died 92bf2794e77c60ebe094b03c5470f40ba72b690b8f09f951828bd042a2f97375 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-6-deactivate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319) 2024-03-21T14:27:51.145 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:50 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:50.835+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:51.145 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:50 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:50.835+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:51.145 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:50 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:50.835+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:51.145 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:50 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:50.835+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:51.163 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.6.service' 2024-03-21T14:27:51.203 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:51.203 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-21T14:27:51.204 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-21T14:27:51.204 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.7 2024-03-21T14:27:51.431 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 systemd[1]: Stopping Ceph osd.7 for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:51.878 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.532+0000 7fb692dc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:51.878 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.532+0000 7fb692dc8700 -1 osd.7 496 *** Got signal Terminated *** 2024-03-21T14:27:51.878 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.532+0000 7fb692dc8700 -1 osd.7 496 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-21T14:27:51.878 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.878+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:51.878 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.878+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:51.878 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.878+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:51.879 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:51 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:51.878+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:53.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:52 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:52.907+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:53.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:52 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:52.907+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:53.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:52 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:52.907+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:53.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:52 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:52.907+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:54.431 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:53.954+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:54.431 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:53.954+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:54.431 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:53.954+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:54.431 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:53 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:53.954+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:55.180 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:54 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:54.927+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:55.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:54 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:54.927+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:55.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:54 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:54.927+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:55.181 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:54 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:54.927+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:56.299 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:55 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:55.945+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6803 osd.0 since back 2024-03-21T14:26:56.948689+0000 front 2024-03-21T14:26:56.948804+0000 (oldest deadline 2024-03-21T14:27:22.248800+0000) 2024-03-21T14:27:56.299 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:55 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:55.945+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6807 osd.1 since back 2024-03-21T14:27:06.349688+0000 front 2024-03-21T14:27:06.349813+0000 (oldest deadline 2024-03-21T14:27:33.950067+0000) 2024-03-21T14:27:56.299 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:55 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:55.945+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6811 osd.2 since back 2024-03-21T14:27:13.950694+0000 front 2024-03-21T14:27:13.950721+0000 (oldest deadline 2024-03-21T14:27:37.450488+0000) 2024-03-21T14:27:56.299 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:55 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7[56147]: 2024-03-21T14:27:55.945+0000 7fb68b3b4700 -1 osd.7 496 heartbeat_check: no reply from 172.21.15.3:6815 osd.3 since back 2024-03-21T14:27:22.751443+0000 front 2024-03-21T14:27:22.751487+0000 (oldest deadline 2024-03-21T14:27:47.451399+0000) 2024-03-21T14:27:57.400 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:57 smithi118 podman[162251]: 2024-03-21 14:27:57.112500382 +0000 UTC m=+5.678954263 container died 492937b5df713a0e21c3d96e3f4e074cb2e003074f9de303cc60818504437272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319) 2024-03-21T14:27:57.680 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:57 smithi118 podman[162251]: 2024-03-21 14:27:57.498799404 +0000 UTC m=+6.065253294 container cleanup 492937b5df713a0e21c3d96e3f4e074cb2e003074f9de303cc60818504437272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-21T14:27:57.681 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:57 smithi118 bash[162251]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7 2024-03-21T14:27:58.076 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:57 smithi118 podman[162281]: 2024-03-21 14:27:57.766670348 +0000 UTC m=+0.649474470 container remove 492937b5df713a0e21c3d96e3f4e074cb2e003074f9de303cc60818504437272 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-21T14:27:58.352 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:58 smithi118 podman[162378]: 2024-03-21 14:27:57.981872706 +0000 UTC m=+0.019244982 image pull 07a0e2f85eccbc661c4d74d2d7069b657c3df680b0555f207e4d40ae3737b84c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7 2024-03-21T14:27:58.353 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:58 smithi118 podman[162378]: 2024-03-21 14:27:58.084471345 +0000 UTC m=+0.121843655 container create 192096ea583747c3961f440aabd7e3c96e377902ae935adb6ebf37ba1445d407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-21T14:27:58.353 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:58 smithi118 podman[162378]: 2024-03-21 14:27:58.286891331 +0000 UTC m=+0.324263603 container init 192096ea583747c3961f440aabd7e3c96e377902ae935adb6ebf37ba1445d407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-21T14:27:58.353 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:58 smithi118 podman[162378]: 2024-03-21 14:27:58.289987376 +0000 UTC m=+0.327359644 container start 192096ea583747c3961f440aabd7e3c96e377902ae935adb6ebf37ba1445d407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, GIT_BRANCH=HEAD) 2024-03-21T14:27:58.353 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:58 smithi118 podman[162378]: 2024-03-21 14:27:58.353551815 +0000 UTC m=+0.390924096 container attach 192096ea583747c3961f440aabd7e3c96e377902ae935adb6ebf37ba1445d407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-21T14:27:58.680 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:58 smithi118 podman[162378]: 2024-03-21 14:27:58.583928637 +0000 UTC m=+0.621300905 container died 192096ea583747c3961f440aabd7e3c96e377902ae935adb6ebf37ba1445d407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-21T14:27:59.126 INFO:journalctl@ceph.osd.7.smithi118.stdout:Mar 21 14:27:59 smithi118 podman[162404]: 2024-03-21 14:27:59.115360924 +0000 UTC m=+0.524335695 container remove 192096ea583747c3961f440aabd7e3c96e377902ae935adb6ebf37ba1445d407 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-21T14:27:59.132 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@osd.7.service' 2024-03-21T14:27:59.175 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:27:59.175 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-21T14:27:59.175 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-21T14:27:59.175 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-21T14:27:59.475 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:27:59 smithi003 systemd[1]: Stopping Ceph rgw.foo.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:27:59.753 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:27:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a[67220]: 2024-03-21T14:27:59.475+0000 7f1d5b031700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-21T14:27:59.754 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:27:59 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a[67220]: 2024-03-21T14:27:59.475+0000 7f1d698b6ac0 -1 shutting down 2024-03-21T14:28:09.741 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:28:09 smithi003 bash[218102]: time="2024-03-21T14:28:09Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-21T14:28:09.742 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:28:09 smithi003 podman[218102]: 2024-03-21 14:28:09.497008734 +0000 UTC m=+10.124862332 container died afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-21T14:28:10.003 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:28:09 smithi003 podman[218102]: 2024-03-21 14:28:09.908058299 +0000 UTC m=+10.535911896 container cleanup afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, GIT_CLEAN=True) 2024-03-21T14:28:10.004 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:28:09 smithi003 bash[218102]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-21T14:28:10.465 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:28:10 smithi003 podman[218117]: 2024-03-21 14:28:10.192632394 +0000 UTC m=+0.688942717 container remove afc0cffc49e0f4e3128cb0a63b51d5ba4e972d88837013410e46355c37dc1e86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-21T14:28:10.466 INFO:journalctl@ceph.rgw.foo.a.smithi003.stdout:Mar 21 14:28:10 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-21T14:28:10.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-21T14:28:10.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:28:10.520 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-21T14:28:10.520 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-21T14:28:10.520 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-c60e082a-e78a-11ee-95cd-87774f69a715@prometheus.a 2024-03-21T14:28:10.808 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 systemd[1]: Stopping Ceph prometheus.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:28:11.084 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.808Z caller=main.go:964 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-21T14:28:11.084 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=main.go:988 level=info msg="Stopping scrape discovery manager..." 2024-03-21T14:28:11.084 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=main.go:1002 level=info msg="Stopping notify discovery manager..." 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=manager.go:177 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=main.go:984 level=info msg="Scrape discovery manager stopped" 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=main.go:998 level=info msg="Notify discovery manager stopped" 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=manager.go:187 level=info component="rule manager" msg="Rule manager stopped" 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=main.go:1039 level=info msg="Stopping scrape manager..." 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.809Z caller=main.go:1031 level=info msg="Scrape manager stopped" 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.810Z caller=notifier.go:618 level=info component=notifier msg="Stopping notification manager..." 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.810Z caller=main.go:1261 level=info msg="Notifier manager stopped" 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a[70112]: ts=2024-03-21T14:28:10.810Z caller=main.go:1273 level=info msg="See you next time!" 2024-03-21T14:28:11.085 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:10 smithi118 podman[162484]: 2024-03-21 14:28:10.819167686 +0000 UTC m=+0.102369525 container died e24138720969ffa80f9d4030b8922678b7a904857108845f0b9dacd0b05f7d53 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:11.430 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:11 smithi118 podman[162484]: 2024-03-21 14:28:11.259082789 +0000 UTC m=+0.542284626 container cleanup e24138720969ffa80f9d4030b8922678b7a904857108845f0b9dacd0b05f7d53 (image=quay.io/prometheus/prometheus:v2.51.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:11.430 INFO:journalctl@ceph.prometheus.a.smithi118.stdout:Mar 21 14:28:11 smithi118 bash[162484]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-prometheus-a 2024-03-21T14:28:11.650 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c60e082a-e78a-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-21T14:28:11.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-21T14:28:11.722 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-21T14:28:11.723 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c60e082a-e78a-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T14:28:11.921 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T14:28:14.254 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:14 smithi003 systemd[1]: Stopping Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:28:14.713 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:14 smithi003 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a[83808]: ts=2024-03-21T14:28:14.404Z caller=main.go:542 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-21T14:28:14.713 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:14 smithi003 podman[218596]: 2024-03-21 14:28:14.4457915 +0000 UTC m=+0.141570421 container died da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:15.003 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:14 smithi003 podman[218596]: 2024-03-21 14:28:14.855100443 +0000 UTC m=+0.550879421 container cleanup da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:15.004 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:14 smithi003 bash[218596]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a 2024-03-21T14:28:15.315 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:15 smithi003 podman[218610]: 2024-03-21 14:28:15.022832473 +0000 UTC m=+0.581111665 container remove da16e39eac86825730c536f0d1eed3f12bba60b301044e51f7540520792beb06 (image=quay.io/prometheus/alertmanager:v0.27.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:15.316 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:15 smithi003 podman[218610]: 2024-03-21 14:28:15.090034511 +0000 UTC m=+0.648313786 volume remove fa10807a53d89fae3f8570e351cf8ad500f1c4d5f1cc1de085603a0636b966c1 2024-03-21T14:28:15.753 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:15 smithi003 systemd[1]: Stopping Ceph node-exporter.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:28:15.754 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:15 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-21T14:28:15.754 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:15 smithi003 systemd[1]: Stopped Ceph alertmanager.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:28:15.754 INFO:journalctl@ceph.alertmanager.a.smithi003.stdout:Mar 21 14:28:15 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.705s CPU time. 2024-03-21T14:28:16.050 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:15 smithi003 podman[218722]: 2024-03-21 14:28:15.786413431 +0000 UTC m=+0.115291957 container died d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:16.442 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 podman[218722]: 2024-03-21 14:28:16.217698704 +0000 UTC m=+0.546577269 container cleanup d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:16.443 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 bash[218722]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a 2024-03-21T14:28:16.694 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 podman[218736]: 2024-03-21 14:28:16.443792456 +0000 UTC m=+0.658202935 container remove d16f044ca2cd77112fddd73853a016c564e6d4da0622663dcaac4d684ac023d1 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-21T14:28:16.695 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-21T14:28:16.695 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-21T14:28:16.695 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 systemd[1]: Stopped Ceph node-exporter.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:28:16.695 INFO:journalctl@ceph.node-exporter.a.smithi003.stdout:Mar 21 14:28:16 smithi003 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 4.936s CPU time. 2024-03-21T14:28:17.351 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c60e082a-e78a-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-21T14:28:17.543 INFO:teuthology.orchestra.run.smithi118.stdout:Deleting cluster with fsid: c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T14:28:19.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:19 smithi118 systemd[1]: Stopping Ceph iscsi.iscsi.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:28:19.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:19 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a[61359]: debug Shutdown received 2024-03-21T14:28:29.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:29 smithi118 bash[162895]: time="2024-03-21T14:28:29Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-21T14:28:29.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:29 smithi118 podman[162895]: 2024-03-21 14:28:29.68796394 +0000 UTC m=+10.122384522 container died 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-21T14:28:30.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 podman[162895]: 2024-03-21 14:28:30.062128212 +0000 UTC m=+10.496548792 container cleanup 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-21T14:28:30.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 bash[162895]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-21T14:28:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 podman[162910]: 2024-03-21 14:28:30.322021801 +0000 UTC m=+0.635235553 container remove 2c43d2885112980d4043e339cf85c524364ad57059ce287ccabbb1bf03de3854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c36dfee38a9e30f94631724b80f0cc56436755c5ff8c0ab73d50c8801d644bd7, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-20-2326-ba76009, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-03-21T14:28:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-21T14:28:30.680 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-21T14:28:30.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 systemd[1]: Stopped Ceph iscsi.iscsi.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:28:30.681 INFO:journalctl@ceph.iscsi.iscsi.a.smithi118.stdout:Mar 21 14:28:30 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 3.556s CPU time. 2024-03-21T14:28:32.212 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:31 smithi118 systemd[1]: Stopping Ceph grafana.a for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:28:32.505 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=server t=2024-03-21T14:28:32.213120635Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-21T14:28:32.506 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=tracing t=2024-03-21T14:28:32.213518851Z level=info msg="Closing tracing" 2024-03-21T14:28:32.506 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=ticker t=2024-03-21T14:28:32.21356303Z level=info msg=stopped last_tick=2024-03-21T14:28:30Z 2024-03-21T14:28:32.506 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a[64859]: logger=grafana-apiserver t=2024-03-21T14:28:32.213591459Z level=info msg="StorageObjectCountTracker pruner is exiting" 2024-03-21T14:28:32.506 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 podman[163183]: 2024-03-21 14:28:32.233673315 +0000 UTC m=+0.126984038 container died 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 (image=quay.io/ceph/grafana:10.4.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T14:28:32.881 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 podman[163183]: 2024-03-21 14:28:32.597034648 +0000 UTC m=+0.490345370 container cleanup 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 (image=quay.io/ceph/grafana:10.4.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T14:28:32.881 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 bash[163183]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a 2024-03-21T14:28:33.180 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 podman[163197]: 2024-03-21 14:28:32.881365724 +0000 UTC m=+0.649489820 container remove 2e5d5643041b41593bdefb82fefc8d6282c58bb26f07023c0eb594bbc415fbd9 (image=quay.io/ceph/grafana:10.4.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-21T14:28:33.180 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-21T14:28:33.180 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 systemd[1]: Stopped Ceph grafana.a for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:28:33.180 INFO:journalctl@ceph.grafana.a.smithi118.stdout:Mar 21 14:28:32 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@grafana.a.service: Consumed 15.757s CPU time. 2024-03-21T14:28:33.460 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:33 smithi118 systemd[1]: Stopping Ceph node-exporter.b for c60e082a-e78a-11ee-95cd-87774f69a715... 2024-03-21T14:28:33.746 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:33 smithi118 podman[163310]: 2024-03-21 14:28:33.461507215 +0000 UTC m=+0.110389764 container died a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T14:28:34.117 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:33 smithi118 podman[163310]: 2024-03-21 14:28:33.842739801 +0000 UTC m=+0.491622391 container cleanup a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T14:28:34.117 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:33 smithi118 bash[163310]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b 2024-03-21T14:28:34.396 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:34 smithi118 podman[163324]: 2024-03-21 14:28:34.118596745 +0000 UTC m=+0.648709670 container remove a8970f22b4d92ef64dbc81f4f48bf9b8056bcec8a0ec89af8a4d234e89c8e2b8 (image=quay.io/prometheus/node-exporter:v1.7.0, name=ceph-c60e082a-e78a-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-21T14:28:34.396 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:34 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-21T14:28:34.396 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:34 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-21T14:28:34.396 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:34 smithi118 systemd[1]: Stopped Ceph node-exporter.b for c60e082a-e78a-11ee-95cd-87774f69a715. 2024-03-21T14:28:34.396 INFO:journalctl@ceph.node-exporter.b.smithi118.stdout:Mar 21 14:28:34 smithi118 systemd[1]: ceph-c60e082a-e78a-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 4.419s CPU time. 2024-03-21T14:28:34.862 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T14:28:34.897 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-21T14:28:34.926 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-21T14:28:34.928 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/remote/smithi003/crash 2024-03-21T14:28:34.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/crash -- . 2024-03-21T14:28:34.997 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T14:28:34.997 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2024-03-21T14:28:34.999 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/remote/smithi118/crash 2024-03-21T14:28:35.000 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/crash -- . 2024-03-21T14:28:35.081 INFO:teuthology.orchestra.run.smithi118.stderr:tar: /var/lib/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-21T14:28:35.081 INFO:teuthology.orchestra.run.smithi118.stderr:tar: Error is not recoverable: exiting now 2024-03-21T14:28:35.082 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-21T14:28:35.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T14:28:35.163 INFO:teuthology.orchestra.run.smithi003.stdout:2024-03-21T14:07:27.429357+0000 mon.a (mon.0) 957 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:28:35.164 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-21T14:28:35.164 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[SEC\]' /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T14:28:35.198 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]' /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T14:28:35.273 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[WRN\]' /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-21T14:28:35.345 INFO:teuthology.orchestra.run.smithi003.stdout:2024-03-21T14:07:27.429357+0000 mon.a (mon.0) 957 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-21T14:28:35.345 INFO:tasks.cephadm:Compressing logs... 2024-03-21T14:28:35.346 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T14:28:35.388 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-21T14:28:35.451 INFO:teuthology.orchestra.run.smithi118.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T14:28:35.452 INFO:teuthology.orchestra.run.smithi118.stderr:: No such file or directory 2024-03-21T14:28:35.463 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-21T14:28:35.464 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T14:28:35.465 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-21T14:28:35.465 INFO:teuthology.orchestra.run.smithi003.stderr:: No such file or directory 2024-03-21T14:28:35.466 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-21T14:28:35.466 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-21T14:28:35.466 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-21T14:28:35.469 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/objectstore_tool.2.log 2024-03-21T14:28:35.473 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-21T14:28:35.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-21T14:28:35.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-21T14:28:35.473 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr: 57.7% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/objectstore_tool.2.log 2024-03-21T14:28:35.474 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-21T14:28:35.481 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/objectstore_tool.2.log: /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.8% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T14:28:35.534 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.b.log: /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/objectstore_tool.2.log: 93.5% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/objectstore_tool.2.log.gz 2024-03-21T14:28:35.537 INFO:teuthology.orchestra.run.smithi003.stderr: 95.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T14:28:35.658 INFO:teuthology.orchestra.run.smithi118.stderr: 81.6% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T14:28:35.688 INFO:teuthology.orchestra.run.smithi003.stderr: 89.0% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-21T14:28:35.751 INFO:teuthology.orchestra.run.smithi118.stderr: 67.8% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-21T14:28:35.823 INFO:teuthology.orchestra.run.smithi118.stderr: 92.2% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-21T14:28:35.854 INFO:teuthology.orchestra.run.smithi118.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-21T14:28:35.862 INFO:teuthology.orchestra.run.smithi118.stderr: 91.1% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T14:28:35.914 INFO:teuthology.orchestra.run.smithi118.stderr: 90.1% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T14:28:35.919 INFO:teuthology.orchestra.run.smithi118.stderr: 85.6% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-21T14:28:36.180 INFO:teuthology.orchestra.run.smithi003.stderr: 91.1% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-21T14:28:36.243 INFO:teuthology.orchestra.run.smithi003.stderr: 94.0% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-21T14:28:37.240 INFO:teuthology.orchestra.run.smithi003.stderr: 89.7% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-21T14:28:37.961 INFO:teuthology.orchestra.run.smithi003.stderr: 94.3% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/objectstore_tool.2.log.gz 2024-03-21T14:28:38.663 INFO:teuthology.orchestra.run.smithi118.stderr: 91.2% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-21T14:28:51.729 INFO:teuthology.orchestra.run.smithi003.stderr: 91.1% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-21T14:28:55.701 INFO:teuthology.orchestra.run.smithi003.stderr: 90.4% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-21T14:30:06.295 INFO:teuthology.orchestra.run.smithi118.stderr: 93.7% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-21T14:30:47.971 INFO:teuthology.orchestra.run.smithi003.stderr: 93.6% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-21T14:33:47.974 INFO:teuthology.orchestra.run.smithi118.stderr: 93.3% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-21T14:33:53.592 INFO:teuthology.orchestra.run.smithi118.stderr: 93.8% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-21T14:34:54.663 INFO:teuthology.orchestra.run.smithi118.stderr: 93.6% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-21T14:34:54.665 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-21T14:34:54.665 INFO:teuthology.orchestra.run.smithi118.stderr:real 6m19.262s 2024-03-21T14:34:54.665 INFO:teuthology.orchestra.run.smithi118.stderr:user 5m8.819s 2024-03-21T14:34:54.665 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m14.032s 2024-03-21T14:35:05.187 INFO:teuthology.orchestra.run.smithi003.stderr: 93.6% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-21T14:35:05.253 INFO:teuthology.orchestra.run.smithi003.stderr: 93.7% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-21T14:35:58.746 INFO:teuthology.orchestra.run.smithi003.stderr: 93.5% -- replaced with /var/log/ceph/c60e082a-e78a-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-21T14:35:58.748 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-03-21T14:35:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:real 7m23.348s 2024-03-21T14:35:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:user 5m52.015s 2024-03-21T14:35:58.748 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m14.886s 2024-03-21T14:35:58.748 INFO:tasks.cephadm:Archiving logs... 2024-03-21T14:35:58.749 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/remote/smithi003/log 2024-03-21T14:35:58.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T14:36:48.631 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/remote/smithi118/log 2024-03-21T14:36:48.632 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-21T14:37:37.514 INFO:tasks.cephadm:Removing cluster... 2024-03-21T14:37:37.514 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c60e082a-e78a-11ee-95cd-87774f69a715 --force 2024-03-21T14:37:37.715 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T14:37:38.361 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c60e082a-e78a-11ee-95cd-87774f69a715 --force 2024-03-21T14:37:38.565 INFO:teuthology.orchestra.run.smithi118.stdout:Deleting cluster with fsid: c60e082a-e78a-11ee-95cd-87774f69a715 2024-03-21T14:37:39.227 INFO:tasks.cephadm:Removing cephadm ... 2024-03-21T14:37:39.228 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T14:37:39.255 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-21T14:37:39.270 INFO:tasks.cephadm:Teardown complete 2024-03-21T14:37:39.271 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-21T14:37:39.280 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-21T14:37:39.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T14:37:39.298 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-21T14:37:39.408 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-21T14:37:39.417 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-21T14:37:39.417 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T14:37:39.420 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-21T14:37:39.433 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-03-21T14:37:39.433 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-21T14:37:39.455 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T14:37:39.455 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-03-21T14:37:39.455 INFO:teuthology.orchestra.run.smithi003.stdout:^- hv01.front.sepia.ceph.com 2 6 377 44 +4612us[+4612us] +/- 151ms 2024-03-21T14:37:39.456 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 6 377 42 -732us[ -731us] +/- 61ms 2024-03-21T14:37:39.456 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +1026us[+1026us] +/- 78ms 2024-03-21T14:37:39.456 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T14:37:39.459 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-21T14:37:39.459 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-21T14:37:39.459 INFO:teuthology.orchestra.run.smithi118.stdout:^- hv01.front.sepia.ceph.com 2 6 377 42 +4555us[+4555us] +/- 151ms 2024-03-21T14:37:39.459 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -780us[ -776us] +/- 61ms 2024-03-21T14:37:39.459 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 42 +982us[ +982us] +/- 78ms 2024-03-21T14:37:39.460 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-21T14:37:39.460 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-21T14:37:39.471 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-21T14:37:39.472 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-21T14:37:39.480 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T14:37:39.594 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-21T14:37:39.666 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T14:37:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:37:39.701 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-21T14:37:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:37:39.730 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-21T14:37:39.740 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-21T14:37:39.749 INFO:teuthology.task.internal:Duration was 2900.522701 seconds 2024-03-21T14:37:39.750 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-21T14:37:39.759 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-21T14:37:39.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T14:37:39.762 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-21T14:37:39.833 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T14:37:39.844 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-21T14:37:40.121 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-21T14:37:40.121 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-03-21T14:37:40.121 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T14:37:40.142 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-03-21T14:37:40.142 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-21T14:37:40.181 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-21T14:37:40.181 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T14:37:40.184 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-21T14:37:40.249 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-21T14:37:40.249 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T14:37:40.252 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-21T14:37:41.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-21T14:37:41.914 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-21T14:37:41.915 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T14:37:41.951 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-21T14:37:41.990 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-21T14:37:42.005 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T14:37:42.007 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-21T14:37:42.034 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-03-21T14:37:42.065 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-03-21T14:37:42.091 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T14:37:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:37:42.109 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-21T14:37:42.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-21T14:37:42.147 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-21T14:37:42.157 INFO:teuthology.task.internal:Transferring archived files... 2024-03-21T14:37:42.157 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/remote/smithi003 2024-03-21T14:37:42.158 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T14:37:42.225 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898/remote/smithi118 2024-03-21T14:37:42.226 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-21T14:37:42.268 INFO:teuthology.task.internal:Removing archive directory... 2024-03-21T14:37:42.268 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T14:37:42.270 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-21T14:37:42.317 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-21T14:37:42.330 INFO:teuthology.task.internal:Not uploading archives. 2024-03-21T14:37:42.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-21T14:37:42.345 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-21T14:37:42.345 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T14:37:42.348 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-21T14:37:42.360 INFO:teuthology.orchestra.run.smithi003.stdout: 265422 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 14:37 /home/ubuntu/cephtest 2024-03-21T14:37:42.373 INFO:teuthology.orchestra.run.smithi118.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 21 14:37 /home/ubuntu/cephtest 2024-03-21T14:37:42.374 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-21T14:37:42.384 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-21T14:37:42.406 INFO:teuthology.nuke:Checking targets against current locks 2024-03-21T14:37:42.433 DEBUG:teuthology.nuke:shortname: smithi003 2024-03-21T14:37:42.434 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T14:37:42.460 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 13:39:44.895663', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCWhCkuReM9xbtZKpVQKs+kZrE6aoEb7ZNCcv5kMELOyG62o4S0DlDfqRDFSemtZec7rJY0D/ZjFMj6ZWPvdD64='} 2024-03-21T14:37:42.465 DEBUG:teuthology.nuke:shortname: smithi118 2024-03-21T14:37:42.465 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-21T14:37:42.491 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-21_11:21:25-orch:cephadm-wip-adk-testing-2024-03-20-2326-distro-default-smithi/7614898', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-21 13:39:44.898204', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIZuKDaELm0Kia4bvPb7Em1xTwNKRVDWQkiRD09GTgPkuopYjIIVVlLoyVxsUp9NPlJzCxiINkKKvNYRB/Vh6u8='} 2024-03-21T14:37:42.517 INFO:teuthology.orchestra.console.smithi003:Power off 2024-03-21T14:37:42.518 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T14:37:42.549 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-03-21T14:37:42.549 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:42.569 INFO:teuthology.orchestra.console.smithi118:Power off 2024-03-21T14:37:42.569 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-21T14:37:42.579 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-21T14:37:42.593 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2024-03-21T14:37:42.593 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:42.618 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-21T14:37:46.580 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:46.619 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:46.690 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-21T14:37:46.729 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-21T14:37:50.694 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:50.730 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:50.805 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-03-21T14:37:50.843 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2024-03-21T14:37:50.943 INFO:teuthology.orchestra.console.smithi118:Power off completed 2024-03-21T14:37:54.808 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-21T14:37:54.919 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-03-21T14:37:55.019 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-03-21T14:37:55.125 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 2900.5227007865906 failure_reason: '"2024-03-21T14:07:27.429357+0000 mon.a (mon.0) 957 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn''t resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL)" in cluster log' flavor: default owner: scheduled_adking@teuthology success: false 2024-03-21T14:37:55.125 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-21T14:37:55.203 INFO:teuthology.run:FAIL